[protobuf] C++ json decoding is too slow in 3.1.0

[Sanchay H]

We have a fairly big protobuf (~37MB). Here is the breakup of times:

Serializing proto to binary std::string ~= 110ms
Serializing proto to json std::string ~= 4200ms (size ~ 115MB)


Parsing from binary std::string ~= 700ms

Parsing from json std::string ~= 35 mins !!!!!


In comparison,

Parsing same json using rapidjson ~= 800ms


Code I used to serialize and parse:


std::string json;

google::protobuf::util::JsonStringToMessage(json, &proto);


std::string json;

google::protobuf::util::MessageToJsonString(proto, &json);


Surely this is a not as expected right?

['Feng Xiao' via Protocol Buffers]

Post by Sanchay H
Serializing proto to binary std::string ~= 110ms
Serializing proto to json std::string ~= 4200ms (size ~ 115MB)
Parsing from binary std::string ~= 700ms
Parsing from json std::string ~= 35 mins !!!!!
In comparison,
Parsing same json using rapidjson ~= 800ms
std::string json;
google::protobuf::util::JsonStringToMessage(json, &proto);
std::string json;
google::protobuf::util::MessageToJsonString(proto, &json);
Surely this is a not as expected right?

Can you file a github issue? If it takes that long, I suspect the actual
problem is that the parser uses up all your machine's physical memory, and
causes the system to swap in/out memory pages from/to disk the whole time.

Post by Sanchay H
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[Sanchay H]

https://github.com/google/protobuf/issues/2305

I don't think it is a swap issue. The process takes 1.45GB and remains
there. My system has memory left. Plus there is constant 100% CPU usage.

Post by 'Feng Xiao' via Protocol Buffers

Post by Sanchay H
Serializing proto to binary std::string ~= 110ms
Serializing proto to json std::string ~= 4200ms (size ~ 115MB)
Parsing from binary std::string ~= 700ms
Parsing from json std::string ~= 35 mins !!!!!
In comparison,
Parsing same json using rapidjson ~= 800ms
std::string json;
google::protobuf::util::JsonStringToMessage(json, &proto);
std::string json;
google::protobuf::util::MessageToJsonString(proto, &json);
Surely this is a not as expected right?

Can you file a github issue? If it takes that long, I suspect the actual
problem is that the parser uses up all your machine's physical memory, and
causes the system to swap in/out memory pages from/to disk the whole time.

Post by Sanchay H
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.