[protobuf] NoSuchMethodError exception when read the protobuf file from HDFS

[David Yu]

Hi,
I am working on the project to count the record in the protobuf file
but I get the exception

Exception in thread "main" java.lang.NoSuchMethodError:
com.google.protobuf.CodedInputStream.shouldDiscardUnknownFields()Z

at
com.google.protobuf.GeneratedMessageV3.parseUnknownField(GeneratedMessageV3.java:290)


This only happens on some of the protobuf files, not all of them has the
issue.

Same code is working for files with different schema.


My protobuf file is gzipped.


//Here is the code

Configuration conf = new Configuration();

FileSystem fs = FileSystem.get(conf);

Path path = new Path(<HDFS path to file>);

InputStream input = new GZIPInputStream(fs.open(path));

Message m;

while ((m = defaultMsg.getParserForType().parseDelimitedFrom(input)) !=
null) {

recordCount++;

}


Does anyone have idea?


Thanks

David

[David Yu]

Forgot to mention, I execute the jar file with "hadoop -jar countLine.jar"

If I put the file in local and execute with "java -jar countLine.jar"

InputStream input = new GZIPInputStream(new File(path_to_local_file));

Message m;

while ((m = defaultMsg.getParserForType().parseDelimitedFrom(input)) !=
null) {

recordCount++;

}


It works fine. =(

Post by David Yu
Hi,
I am working on the project to count the record in the protobuf file
but I get the exception
com.google.protobuf.CodedInputStream.shouldDiscardUnknownFields()Z
at
com.google.protobuf.GeneratedMessageV3.parseUnknownField(GeneratedMessageV3.java:290)
This only happens on some of the protobuf files, not all of them has the
issue.
Same code is working for files with different schema.
My protobuf file is gzipped.
//Here is the code
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
Path path = new Path(<HDFS path to file>);
InputStream input = new GZIPInputStream(fs.open(path));
Message m;
while ((m = defaultMsg.getParserForType().parseDelimitedFrom(input)) !=
null) {
recordCount++;
}
Does anyone have idea?
Thanks
David

[David Yu]

Figured out this is because the protobuf version is different between
hadoop (2.5.0) and the one I used (3.6.1) to generate the java files.
Once I downgrade the protoc to re-generate the java files, the issue
resolved.

Thanks
David

Post by David Yu
Hi,
I am working on the project to count the record in the protobuf file
but I get the exception
com.google.protobuf.CodedInputStream.shouldDiscardUnknownFields()Z
at
com.google.protobuf.GeneratedMessageV3.parseUnknownField(GeneratedMessageV3.java:290)
This only happens on some of the protobuf files, not all of them has the
issue.
Same code is working for files with different schema.
My protobuf file is gzipped.
//Here is the code
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
Path path = new Path(<HDFS path to file>);
InputStream input = new GZIPInputStream(fs.open(path));
Message m;
while ((m = defaultMsg.getParserForType().parseDelimitedFrom(input)) !=
null) {
recordCount++;
}
Does anyone have idea?
Thanks
David