Creating a FileDescriptorSet of all .proto files used in a certain executable

[Oren Shemesh]

Hello,

My executable is composed of multiple .proto files, some of them
importing others.
I would like the executable to communicate with a generic program
(Which is compiled once and does not change), using protobuf messages.
So, I need the generic program to get the descriptors of messages used
in the communication at run-time, from my executable.
I understand that to do this, I need the executable to create a
FileDescriptorSet containing all the files needed to describe a
certain message, serialize this set, pass it to my generic program,
which will then create a descriptor pool and feed it with the
deserialized FileDescriptorSet. Now that I have this populated pool,
the generic program can find any descriptor by it's name, as
communicated from the executable.

My question is: Is there a method to automatically create the file
descriptor set of all files used inside a given application, in proper
topological order (i.e. imported files appearing before files
depending on them) ?
I know that protoc can create such a set, but this set reflects the
set of files used in a single protoc run, and my application has
multiple files.
I thought about iterating all descriptors found in the generated_pool,
gathering the set of files, sorting them and creating this set, but
there is no way to iterate all descriptors in the pool.
It seems to me that in order to gather all file descriptor protos used
in my executable, I need to manually define a single .proto file
importing all other files, but I do not like this idea because it
means taht every time I add a new .proto file to my application, this
'master proto' file needs to be updated.
(Yes, the build system could do it automatically).

Is there a better way ?

Thanks, Oren.

[Kenton Varda]

As you note, you can't iterate over all compiled-in protos. However, if you
can collect a set of "root" protos that you care about, it's easy to
traverse down the tree to collect all their dependencies.

In general, trying to enumerate "everything that is linked in" tends to lead
to trouble. The linker may drop things that it thinks are unused.
Technically the C++ standard does not even require initializers to be run
until the first time their translation unit is accessed, although most
compilers run all initializers at startup. It can also be confusing if
linking in a new library causes behavioral changes in unrelated parts of
your program. I actually wish that we had never introduced the singleton
generated_pool in the C++ library since it creates so much confusion -- the
Java protobuf library has no central registry, and this avoids a lot of
problems.

Post by Oren Shemesh
Hello,
My executable is composed of multiple .proto files, some of them
importing others.
I would like the executable to communicate with a generic program
(Which is compiled once and does not change), using protobuf messages.
So, I need the generic program to get the descriptors of messages used
in the communication at run-time, from my executable.
I understand that to do this, I need the executable to create a
FileDescriptorSet containing all the files needed to describe a
certain message, serialize this set, pass it to my generic program,
which will then create a descriptor pool and feed it with the
deserialized FileDescriptorSet. Now that I have this populated pool,
the generic program can find any descriptor by it's name, as
communicated from the executable.
My question is: Is there a method to automatically create the file
descriptor set of all files used inside a given application, in proper
topological order (i.e. imported files appearing before files
depending on them) ?
I know that protoc can create such a set, but this set reflects the
set of files used in a single protoc run, and my application has
multiple files.
I thought about iterating all descriptors found in the generated_pool,
gathering the set of files, sorting them and creating this set, but
there is no way to iterate all descriptors in the pool.
It seems to me that in order to gather all file descriptor protos used
in my executable, I need to manually define a single .proto file
importing all other files, but I do not like this idea because it
means taht every time I add a new .proto file to my application, this
'master proto' file needs to be updated.
(Yes, the build system could do it automatically).
Is there a better way ?
Thanks, Oren.
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.

[Oren Shemesh]

Thanks Kenton, I appreciate the explanation.

Regards, Oren.

Post by Kenton Varda
As you note, you can't iterate over all compiled-in protos. However, if
you can collect a set of "root" protos that you care about, it's easy to
traverse down the tree to collect all their dependencies.
In general, trying to enumerate "everything that is linked in" tends to
lead to trouble. The linker may drop things that it thinks are unused.
Technically the C++ standard does not even require initializers to be run
until the first time their translation unit is accessed, although most
compilers run all initializers at startup. It can also be confusing if
linking in a new library causes behavioral changes in unrelated parts of
your program. I actually wish that we had never introduced the singleton
generated_pool in the C++ library since it creates so much confusion -- the
Java protobuf library has no central registry, and this avoids a lot of
problems.

Post by Oren Shemesh
Hello,
My executable is composed of multiple .proto files, some of them
importing others.
I would like the executable to communicate with a generic program
(Which is compiled once and does not change), using protobuf messages.
So, I need the generic program to get the descriptors of messages used
in the communication at run-time, from my executable.
I understand that to do this, I need the executable to create a
FileDescriptorSet containing all the files needed to describe a
certain message, serialize this set, pass it to my generic program,
which will then create a descriptor pool and feed it with the
deserialized FileDescriptorSet. Now that I have this populated pool,
the generic program can find any descriptor by it's name, as
communicated from the executable.
My question is: Is there a method to automatically create the file
descriptor set of all files used inside a given application, in proper
topological order (i.e. imported files appearing before files
depending on them) ?
I know that protoc can create such a set, but this set reflects the
set of files used in a single protoc run, and my application has
multiple files.
I thought about iterating all descriptors found in the generated_pool,
gathering the set of files, sorting them and creating this set, but
there is no way to iterate all descriptors in the pool.
It seems to me that in order to gather all file descriptor protos used
in my executable, I need to manually define a single .proto file
importing all other files, but I do not like this idea because it
means taht every time I add a new .proto file to my application, this
'master proto' file needs to be updated.
(Yes, the build system could do it automatically).
Is there a better way ?
Thanks, Oren.
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/protobuf?hl=en.