Discussion:
[protobuf] Getting started developing a protoc plugin
Michael Powell
2018-11-02 01:14:33 UTC
Permalink
Hello,

I'm a bit confused. I thought I was grasping it earlier, then I
thought I wasn't ... What is the starting point developing a protoc
plugin?

What I want to accomplish is processing a .proto (v2) and generating
some C# boilerplate code for it. I'm not generating a proto adapter
itself, per se, but C# DSL that would be used to support it.

The starting point for me would seem to be receiving the compiler
request, but that itself is a .proto, which leaves me a bit confused.
Where do you obtain that from? Or are you starting by compiling those
.protos from the bits bundled with the protoc.exe compiler?

What I want is to just have the descriptors drawn from the target
.proto(s) themselves. That's it. A full-on plugin seems like a little
bit of overkill.

Suggestions?

Thanks so much!

Best regards,

Michael Powell
--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Nadav Samet
2018-11-10 22:33:11 UTC
Permalink
Michael,

A protoc plugin is a program that receives a serialized
CodeGeneratorRequest message via its stdin, and writes a
CodeGeneratorResponse to its stdout.

You asked where the CodeGeneratorRequest comes from. The answer is that the
message is defined here
<https://github.com/protocolbuffers/protobuf/blob/446e34ffc71ad458c21c60529cbcdc1de74e4c3d/src/google/protobuf/compiler/plugin.proto#L67>.
Assuming your plugin is named protoc-gen-boilerplate, then when you invoke
protoc like this:
protoc --boilerplate_out=outdir file1.proto [...]

Then protoc would parse the proto files, prepare a CodeGeneratorRequest and
pipe it into your plugin. That request object contains a representation of
all the files you passed in the command line as well as the files their
transitively import. Your plugin should parse the request and print out a
serialized CodeGeneratorResponse. Unless you set an error message, protoc
would create the files you specified in your response. The plugin can be
written in any language, but if you use some of the supported languages (at
least C++, Python and Java, but probably others), you'll find a method
named something like DescriptorPool.BuildFile which would convert a
FileDescriptorProto to an higher-level object (FileDescriptor) which is
easier to work with.

Plugins are pretty simple to write (look online for examples). I would
advise not writing a parser for proto files manually, since you'll be
chasing a moving target - over time language features get added albeit not
very often. It would also be hard to match the language spec perfectly.
Post by Michael Powell
Hello,
I'm a bit confused. I thought I was grasping it earlier, then I
thought I wasn't ... What is the starting point developing a protoc
plugin?
What I want to accomplish is processing a .proto (v2) and generating
some C# boilerplate code for it. I'm not generating a proto adapter
itself, per se, but C# DSL that would be used to support it.
The starting point for me would seem to be receiving the compiler
request, but that itself is a .proto, which leaves me a bit confused.
Where do you obtain that from? Or are you starting by compiling those
.protos from the bits bundled with the protoc.exe compiler?
What I want is to just have the descriptors drawn from the target
.proto(s) themselves. That's it. A full-on plugin seems like a little
bit of overkill.
Suggestions?
Thanks so much!
Best regards,
Michael Powell
--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Rahul Shrivastava
2018-11-26 22:37:25 UTC
Permalink
Nadav,

I have been looking online for a concrete example to protobuf plugin
development in Java. Do you have a concrete example to share ?
Post by Nadav Samet
Michael,
A protoc plugin is a program that receives a serialized
CodeGeneratorRequest message via its stdin, and writes a
CodeGeneratorResponse to its stdout.
You asked where the CodeGeneratorRequest comes from. The answer is that
the message is defined here
<https://github.com/protocolbuffers/protobuf/blob/446e34ffc71ad458c21c60529cbcdc1de74e4c3d/src/google/protobuf/compiler/plugin.proto#L67>.
Assuming your plugin is named protoc-gen-boilerplate, then when you invoke
protoc --boilerplate_out=outdir file1.proto [...]
Then protoc would parse the proto files, prepare a CodeGeneratorRequest
and pipe it into your plugin. That request object contains a representation
of all the files you passed in the command line as well as the files their
transitively import. Your plugin should parse the request and print out a
serialized CodeGeneratorResponse. Unless you set an error message, protoc
would create the files you specified in your response. The plugin can be
written in any language, but if you use some of the supported languages (at
least C++, Python and Java, but probably others), you'll find a method
named something like DescriptorPool.BuildFile which would convert a
FileDescriptorProto to an higher-level object (FileDescriptor) which is
easier to work with.
Plugins are pretty simple to write (look online for examples). I would
advise not writing a parser for proto files manually, since you'll be
chasing a moving target - over time language features get added albeit not
very often. It would also be hard to match the language spec perfectly.
Post by Michael Powell
Hello,
I'm a bit confused. I thought I was grasping it earlier, then I
thought I wasn't ... What is the starting point developing a protoc
plugin?
What I want to accomplish is processing a .proto (v2) and generating
some C# boilerplate code for it. I'm not generating a proto adapter
itself, per se, but C# DSL that would be used to support it.
The starting point for me would seem to be receiving the compiler
request, but that itself is a .proto, which leaves me a bit confused.
Where do you obtain that from? Or are you starting by compiling those
.protos from the bits bundled with the protoc.exe compiler?
What I want is to just have the descriptors drawn from the target
.proto(s) themselves. That's it. A full-on plugin seems like a little
bit of overkill.
Suggestions?
Thanks so much!
Best regards,
Michael Powell
--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Michael Powell
2018-11-27 14:01:26 UTC
Permalink
Post by Nadav Samet
Michael,
A protoc plugin is a program that receives a serialized
CodeGeneratorRequest message via its stdin, and writes a
CodeGeneratorResponse to its stdout.
You asked where the CodeGeneratorRequest comes from. The answer is that
the message is defined here
<https://github.com/protocolbuffers/protobuf/blob/446e34ffc71ad458c21c60529cbcdc1de74e4c3d/src/google/protobuf/compiler/plugin.proto#L67>.
Assuming your plugin is named protoc-gen-boilerplate, then when you invoke
protoc --boilerplate_out=outdir file1.proto [...]
Then protoc would parse the proto files, prepare a CodeGeneratorRequest
and pipe it into your plugin. That request object contains a representation
of all the files you passed in the command line as well as the files their
transitively import. Your plugin should parse the request and print out a
serialized CodeGeneratorResponse. Unless you set an error message, protoc
would create the files you specified in your response. The plugin can be
written in any language, but if you use some of the supported languages (at
least C++, Python and Java, but probably others), you'll find a method
named something like DescriptorPool.BuildFile which would convert a
FileDescriptorProto to an higher-level object (FileDescriptor) which is
easier to work with.
Plugins are pretty simple to write (look online for examples). I would
advise not writing a parser for proto files manually, since you'll be
chasing a moving target - over time language features get added albeit not
very often. It would also be hard to match the language spec perfectly.
I guess I am not sure quite where to begin. I've downloaded the protoc
compiler, which itself has a set of proto files, but no target language
level generated code.

Am I accurate in assuming I run those through protoc itself to my target
language, in this case I suppose C++?

On Windows, what form does the plugin need to take? DLL? Executable file?

These are the sort of nibblets I think that would make it more evident
where to pick up and run with it.

Examples of this appear to be few and far between, if at all.

Absent examples, I am working on a Boost Spirit Qi parser in the meantime,
which the learning curve is not a plateau, I can agree there. It's not
terrible to work through, and I end up with an AST that I know forwards and
backwards without too much extra effort once the rules have done their
synthetic magic.

Cheers, thank you!
Post by Nadav Samet
Post by Michael Powell
Hello,
I'm a bit confused. I thought I was grasping it earlier, then I
thought I wasn't ... What is the starting point developing a protoc
plugin?
What I want to accomplish is processing a .proto (v2) and generating
some C# boilerplate code for it. I'm not generating a proto adapter
itself, per se, but C# DSL that would be used to support it.
The starting point for me would seem to be receiving the compiler
request, but that itself is a .proto, which leaves me a bit confused.
Where do you obtain that from? Or are you starting by compiling those
.protos from the bits bundled with the protoc.exe compiler?
What I want is to just have the descriptors drawn from the target
.proto(s) themselves. That's it. A full-on plugin seems like a little
bit of overkill.
Suggestions?
Thanks so much!
Best regards,
Michael Powell
--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Nadav Samet
2018-11-28 05:31:25 UTC
Permalink
Hi Rahul and Michael,

I put together a basic example of a protoc plugin in Java that generates
text files: https://github.com/thesamet/protoc-plugin-in-java

The plugin itself is an executable that gets executed by protoc, it can be
written in any language. In this instance, the plugin is written in Java,
so we use gradle to generate a shell script (or bat file for windows) that
protoc will execute to run our plugin inside a JVM. The exact instructions
to run it are in the provided README.md
Post by Michael Powell
Post by Nadav Samet
Michael,
A protoc plugin is a program that receives a serialized
CodeGeneratorRequest message via its stdin, and writes a
CodeGeneratorResponse to its stdout.
You asked where the CodeGeneratorRequest comes from. The answer is that
the message is defined here
<https://github.com/protocolbuffers/protobuf/blob/446e34ffc71ad458c21c60529cbcdc1de74e4c3d/src/google/protobuf/compiler/plugin.proto#L67>.
Assuming your plugin is named protoc-gen-boilerplate, then when you invoke
protoc --boilerplate_out=outdir file1.proto [...]
Then protoc would parse the proto files, prepare a CodeGeneratorRequest
and pipe it into your plugin. That request object contains a representation
of all the files you passed in the command line as well as the files their
transitively import. Your plugin should parse the request and print out a
serialized CodeGeneratorResponse. Unless you set an error message, protoc
would create the files you specified in your response. The plugin can be
written in any language, but if you use some of the supported languages (at
least C++, Python and Java, but probably others), you'll find a method
named something like DescriptorPool.BuildFile which would convert a
FileDescriptorProto to an higher-level object (FileDescriptor) which is
easier to work with.
Plugins are pretty simple to write (look online for examples). I would
advise not writing a parser for proto files manually, since you'll be
chasing a moving target - over time language features get added albeit not
very often. It would also be hard to match the language spec perfectly.
I guess I am not sure quite where to begin. I've downloaded the protoc
compiler, which itself has a set of proto files, but no target language
level generated code.
Am I accurate in assuming I run those through protoc itself to my target
language, in this case I suppose C++?
On Windows, what form does the plugin need to take? DLL? Executable file?
These are the sort of nibblets I think that would make it more evident
where to pick up and run with it.
Examples of this appear to be few and far between, if at all.
Absent examples, I am working on a Boost Spirit Qi parser in the meantime,
which the learning curve is not a plateau, I can agree there. It's not
terrible to work through, and I end up with an AST that I know forwards and
backwards without too much extra effort once the rules have done their
synthetic magic.
Cheers, thank you!
Post by Nadav Samet
Post by Michael Powell
Hello,
I'm a bit confused. I thought I was grasping it earlier, then I
thought I wasn't ... What is the starting point developing a protoc
plugin?
What I want to accomplish is processing a .proto (v2) and generating
some C# boilerplate code for it. I'm not generating a proto adapter
itself, per se, but C# DSL that would be used to support it.
The starting point for me would seem to be receiving the compiler
request, but that itself is a .proto, which leaves me a bit confused.
Where do you obtain that from? Or are you starting by compiling those
.protos from the bits bundled with the protoc.exe compiler?
What I want is to just have the descriptors drawn from the target
.proto(s) themselves. That's it. A full-on plugin seems like a little
bit of overkill.
Suggestions?
Thanks so much!
Best regards,
Michael Powell
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
--
-Nadav
--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Michael Powell
2018-11-28 05:46:15 UTC
Permalink
So...

You've got a magic set of imports (in Java).

import com.google.protobuf.DescriptorProtos;
import com.google.protobuf.Descriptors;
import com.google.protobuf.compiler.PluginProtos.CodeGeneratorResponse;
import com.google.protobuf.compiler.PluginProtos.CodeGeneratorRequest;

Where do you get these from?

Or Assemblies or NuGet Package References in .NET / C#, or
source/libraries, ostensibly, for C/C++.

This is the question I have in getting started.

The nearest assumption I could derive is that I start with protoc.exe
(Windows) and the bundled .proto files, "compile" those to target
plugin language, and build them with the project?

Again, as a what? .DLL? .EXE?
Post by Nadav Samet
Hi Rahul and Michael,
I put together a basic example of a protoc plugin in Java that generates text files: https://github.com/thesamet/protoc-plugin-in-java
The plugin itself is an executable that gets executed by protoc, it can be written in any language. In this instance, the plugin is written in Java, so we use gradle to generate a shell script (or bat file for windows) that protoc will execute to run our plugin inside a JVM. The exact instructions to run it are in the provided README.md
Post by Nadav Samet
Michael,
A protoc plugin is a program that receives a serialized CodeGeneratorRequest message via its stdin, and writes a CodeGeneratorResponse to its stdout.
protoc --boilerplate_out=outdir file1.proto [...]
Then protoc would parse the proto files, prepare a CodeGeneratorRequest and pipe it into your plugin. That request object contains a representation of all the files you passed in the command line as well as the files their transitively import. Your plugin should parse the request and print out a serialized CodeGeneratorResponse. Unless you set an error message, protoc would create the files you specified in your response. The plugin can be written in any language, but if you use some of the supported languages (at least C++, Python and Java, but probably others), you'll find a method named something like DescriptorPool.BuildFile which would convert a FileDescriptorProto to an higher-level object (FileDescriptor) which is easier to work with.
Plugins are pretty simple to write (look online for examples). I would advise not writing a parser for proto files manually, since you'll be chasing a moving target - over time language features get added albeit not very often. It would also be hard to match the language spec perfectly.
I guess I am not sure quite where to begin. I've downloaded the protoc compiler, which itself has a set of proto files, but no target language level generated code.
Am I accurate in assuming I run those through protoc itself to my target language, in this case I suppose C++?
On Windows, what form does the plugin need to take? DLL? Executable file?
These are the sort of nibblets I think that would make it more evident where to pick up and run with it.
Examples of this appear to be few and far between, if at all.
Absent examples, I am working on a Boost Spirit Qi parser in the meantime, which the learning curve is not a plateau, I can agree there. It's not terrible to work through, and I end up with an AST that I know forwards and backwards without too much extra effort once the rules have done their synthetic magic.
Cheers, thank you!
Post by Nadav Samet
Post by Michael Powell
Hello,
I'm a bit confused. I thought I was grasping it earlier, then I
thought I wasn't ... What is the starting point developing a protoc
plugin?
What I want to accomplish is processing a .proto (v2) and generating
some C# boilerplate code for it. I'm not generating a proto adapter
itself, per se, but C# DSL that would be used to support it.
The starting point for me would seem to be receiving the compiler
request, but that itself is a .proto, which leaves me a bit confused.
Where do you obtain that from? Or are you starting by compiling those
.protos from the bits bundled with the protoc.exe compiler?
What I want is to just have the descriptors drawn from the target
.proto(s) themselves. That's it. A full-on plugin seems like a little
bit of overkill.
Suggestions?
Thanks so much!
Best regards,
Michael Powell
--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
--
-Nadav
--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Nadav Samet
2018-11-28 06:02:56 UTC
Permalink
In Java, the dependencies you mentioned are provided by Gradle. We
pull protobuf-java in here
<https://github.com/thesamet/protoc-plugin-in-java/blob/d4e6da3cb7782aefa4d71dc93af7854079f1e07a/build.gradle#L15>
.

In C++, the equivalent files would come from descriptor.h
<https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.descriptor>
which
is shipped with protobuf sources, and the implementation is provided
by libprotobuf, which you can build yourself of the sources.

The plugin is going to be an executable (.EXE on Windows) file.
Post by Michael Powell
So...
You've got a magic set of imports (in Java).
import com.google.protobuf.DescriptorProtos;
import com.google.protobuf.Descriptors;
import com.google.protobuf.compiler.PluginProtos.CodeGeneratorResponse;
import com.google.protobuf.compiler.PluginProtos.CodeGeneratorRequest;
Where do you get these from?
Or Assemblies or NuGet Package References in .NET / C#, or
source/libraries, ostensibly, for C/C++.
This is the question I have in getting started.
The nearest assumption I could derive is that I start with protoc.exe
(Windows) and the bundled .proto files, "compile" those to target
plugin language, and build them with the project?
Again, as a what? .DLL? .EXE?
Post by Nadav Samet
Hi Rahul and Michael,
I put together a basic example of a protoc plugin in Java that generates
text files: https://github.com/thesamet/protoc-plugin-in-java
Post by Nadav Samet
The plugin itself is an executable that gets executed by protoc, it can
be written in any language. In this instance, the plugin is written in
Java, so we use gradle to generate a shell script (or bat file for windows)
that protoc will execute to run our plugin inside a JVM. The exact
instructions to run it are in the provided README.md
Post by Nadav Samet
Post by Michael Powell
Post by Nadav Samet
Michael,
A protoc plugin is a program that receives a serialized
CodeGeneratorRequest message via its stdin, and writes a
CodeGeneratorResponse to its stdout.
Post by Nadav Samet
Post by Michael Powell
Post by Nadav Samet
You asked where the CodeGeneratorRequest comes from. The answer is
that the message is defined here. Assuming your plugin is named
Post by Nadav Samet
Post by Michael Powell
Post by Nadav Samet
protoc --boilerplate_out=outdir file1.proto [...]
Then protoc would parse the proto files, prepare a
CodeGeneratorRequest and pipe it into your plugin. That request object
contains a representation of all the files you passed in the command line
as well as the files their transitively import. Your plugin should parse
the request and print out a serialized CodeGeneratorResponse. Unless you
set an error message, protoc would create the files you specified in your
response. The plugin can be written in any language, but if you use some of
the supported languages (at least C++, Python and Java, but probably
others), you'll find a method named something like DescriptorPool.BuildFile
which would convert a FileDescriptorProto to an higher-level object
(FileDescriptor) which is easier to work with.
Post by Nadav Samet
Post by Michael Powell
Post by Nadav Samet
Plugins are pretty simple to write (look online for examples). I would
advise not writing a parser for proto files manually, since you'll be
chasing a moving target - over time language features get added albeit not
very often. It would also be hard to match the language spec perfectly.
Post by Nadav Samet
Post by Michael Powell
I guess I am not sure quite where to begin. I've downloaded the protoc
compiler, which itself has a set of proto files, but no target language
level generated code.
Post by Nadav Samet
Post by Michael Powell
Am I accurate in assuming I run those through protoc itself to my
target language, in this case I suppose C++?
Post by Nadav Samet
Post by Michael Powell
On Windows, what form does the plugin need to take? DLL? Executable
file?
Post by Nadav Samet
Post by Michael Powell
These are the sort of nibblets I think that would make it more evident
where to pick up and run with it.
Post by Nadav Samet
Post by Michael Powell
Examples of this appear to be few and far between, if at all.
Absent examples, I am working on a Boost Spirit Qi parser in the
meantime, which the learning curve is not a plateau, I can agree there.
It's not terrible to work through, and I end up with an AST that I know
forwards and backwards without too much extra effort once the rules have
done their synthetic magic.
Post by Nadav Samet
Post by Michael Powell
Cheers, thank you!
Post by Nadav Samet
On Thursday, November 1, 2018 at 6:14:54 PM UTC-7, Michael Powell
Post by Michael Powell
Hello,
I'm a bit confused. I thought I was grasping it earlier, then I
thought I wasn't ... What is the starting point developing a protoc
plugin?
What I want to accomplish is processing a .proto (v2) and generating
some C# boilerplate code for it. I'm not generating a proto adapter
itself, per se, but C# DSL that would be used to support it.
The starting point for me would seem to be receiving the compiler
request, but that itself is a .proto, which leaves me a bit confused.
Where do you obtain that from? Or are you starting by compiling those
.protos from the bits bundled with the protoc.exe compiler?
What I want is to just have the descriptors drawn from the target
.proto(s) themselves. That's it. A full-on plugin seems like a little
bit of overkill.
Suggestions?
Thanks so much!
Best regards,
Michael Powell
--
You received this message because you are subscribed to the Google
Groups "Protocol Buffers" group.
Post by Nadav Samet
Post by Michael Powell
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
--
-Nadav
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
--
-Nadav
--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Michael Powell
2018-11-28 06:42:07 UTC
Permalink
In Java, the dependencies you mentioned are provided by Gradle. We pull protobuf-java in here.
I see... Been a little while since I dove that deep in any Java
stacks, so that's a new one to me, but good to know.
In C++, the equivalent files would come from descriptor.h which is shipped with protobuf sources, and the implementation is provided by libprotobuf, which you can build yourself of the sources.
The plugin is going to be an executable (.EXE on Windows) file.
Thanks, I appreciate the clarification. I'm not sure which download I
saw, but it did not include sources, or I downloaded the minimal one.
If memory serves there was a full source one, so I'll have a look
there. Thanks!
Post by Michael Powell
So...
You've got a magic set of imports (in Java).
import com.google.protobuf.DescriptorProtos;
import com.google.protobuf.Descriptors;
import com.google.protobuf.compiler.PluginProtos.CodeGeneratorResponse;
import com.google.protobuf.compiler.PluginProtos.CodeGeneratorRequest;
Where do you get these from?
Or Assemblies or NuGet Package References in .NET / C#, or
source/libraries, ostensibly, for C/C++.
This is the question I have in getting started.
The nearest assumption I could derive is that I start with protoc.exe
(Windows) and the bundled .proto files, "compile" those to target
plugin language, and build them with the project?
Again, as a what? .DLL? .EXE?
Post by Nadav Samet
Hi Rahul and Michael,
I put together a basic example of a protoc plugin in Java that generates text files: https://github.com/thesamet/protoc-plugin-in-java
The plugin itself is an executable that gets executed by protoc, it can be written in any language. In this instance, the plugin is written in Java, so we use gradle to generate a shell script (or bat file for windows) that protoc will execute to run our plugin inside a JVM. The exact instructions to run it are in the provided README.md
Post by Nadav Samet
Michael,
A protoc plugin is a program that receives a serialized CodeGeneratorRequest message via its stdin, and writes a CodeGeneratorResponse to its stdout.
protoc --boilerplate_out=outdir file1.proto [...]
Then protoc would parse the proto files, prepare a CodeGeneratorRequest and pipe it into your plugin. That request object contains a representation of all the files you passed in the command line as well as the files their transitively import. Your plugin should parse the request and print out a serialized CodeGeneratorResponse. Unless you set an error message, protoc would create the files you specified in your response. The plugin can be written in any language, but if you use some of the supported languages (at least C++, Python and Java, but probably others), you'll find a method named something like DescriptorPool.BuildFile which would convert a FileDescriptorProto to an higher-level object (FileDescriptor) which is easier to work with.
Plugins are pretty simple to write (look online for examples). I would advise not writing a parser for proto files manually, since you'll be chasing a moving target - over time language features get added albeit not very often. It would also be hard to match the language spec perfectly.
I guess I am not sure quite where to begin. I've downloaded the protoc compiler, which itself has a set of proto files, but no target language level generated code.
Am I accurate in assuming I run those through protoc itself to my target language, in this case I suppose C++?
On Windows, what form does the plugin need to take? DLL? Executable file?
These are the sort of nibblets I think that would make it more evident where to pick up and run with it.
Examples of this appear to be few and far between, if at all.
Absent examples, I am working on a Boost Spirit Qi parser in the meantime, which the learning curve is not a plateau, I can agree there. It's not terrible to work through, and I end up with an AST that I know forwards and backwards without too much extra effort once the rules have done their synthetic magic.
Cheers, thank you!
Post by Nadav Samet
Post by Michael Powell
Hello,
I'm a bit confused. I thought I was grasping it earlier, then I
thought I wasn't ... What is the starting point developing a protoc
plugin?
What I want to accomplish is processing a .proto (v2) and generating
some C# boilerplate code for it. I'm not generating a proto adapter
itself, per se, but C# DSL that would be used to support it.
The starting point for me would seem to be receiving the compiler
request, but that itself is a .proto, which leaves me a bit confused.
Where do you obtain that from? Or are you starting by compiling those
.protos from the bits bundled with the protoc.exe compiler?
What I want is to just have the descriptors drawn from the target
.proto(s) themselves. That's it. A full-on plugin seems like a little
bit of overkill.
Suggestions?
Thanks so much!
Best regards,
Michael Powell
--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
--
-Nadav
--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
--
-Nadav
--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Nadav Samet
2018-11-28 06:47:41 UTC
Permalink
You'd need to download the source code zip and build from there, to get the
necessary include files, as well as building libprotoc and libprotobuf. As
a quick exercise, I ported my example plugin to C++
https://github.com/thesamet/protoc-plugin-in-cpp hopefully it's going to
be useful to anyone struggling with this!
In Java, the dependencies you mentioned are provided by Gradle. We pull
protobuf-java in here.
I see... Been a little while since I dove that deep in any Java
stacks, so that's a new one to me, but good to know.
In C++, the equivalent files would come from descriptor.h which is
shipped with protobuf sources, and the implementation is provided by
libprotobuf, which you can build yourself of the sources.
The plugin is going to be an executable (.EXE on Windows) file.
Thanks, I appreciate the clarification. I'm not sure which download I
saw, but it did not include sources, or I downloaded the minimal one.
If memory serves there was a full source one, so I'll have a look
there. Thanks!
Post by Michael Powell
So...
You've got a magic set of imports (in Java).
import com.google.protobuf.DescriptorProtos;
import com.google.protobuf.Descriptors;
import com.google.protobuf.compiler.PluginProtos.CodeGeneratorResponse;
import com.google.protobuf.compiler.PluginProtos.CodeGeneratorRequest;
Where do you get these from?
Or Assemblies or NuGet Package References in .NET / C#, or
source/libraries, ostensibly, for C/C++.
This is the question I have in getting started.
The nearest assumption I could derive is that I start with protoc.exe
(Windows) and the bundled .proto files, "compile" those to target
plugin language, and build them with the project?
Again, as a what? .DLL? .EXE?
Post by Nadav Samet
Hi Rahul and Michael,
I put together a basic example of a protoc plugin in Java that
generates text files: https://github.com/thesamet/protoc-plugin-in-java
Post by Michael Powell
Post by Nadav Samet
The plugin itself is an executable that gets executed by protoc, it
can be written in any language. In this instance, the plugin is written in
Java, so we use gradle to generate a shell script (or bat file for windows)
that protoc will execute to run our plugin inside a JVM. The exact
instructions to run it are in the provided README.md
Post by Michael Powell
Post by Nadav Samet
On Saturday, November 10, 2018 at 5:33:11 PM UTC-5, Nadav Samet
Post by Nadav Samet
Michael,
A protoc plugin is a program that receives a serialized
CodeGeneratorRequest message via its stdin, and writes a
CodeGeneratorResponse to its stdout.
Post by Michael Powell
Post by Nadav Samet
Post by Nadav Samet
You asked where the CodeGeneratorRequest comes from. The answer is
that the message is defined here. Assuming your plugin is named
Post by Michael Powell
Post by Nadav Samet
Post by Nadav Samet
protoc --boilerplate_out=outdir file1.proto [...]
Then protoc would parse the proto files, prepare a
CodeGeneratorRequest and pipe it into your plugin. That request object
contains a representation of all the files you passed in the command line
as well as the files their transitively import. Your plugin should parse
the request and print out a serialized CodeGeneratorResponse. Unless you
set an error message, protoc would create the files you specified in your
response. The plugin can be written in any language, but if you use some of
the supported languages (at least C++, Python and Java, but probably
others), you'll find a method named something like DescriptorPool.BuildFile
which would convert a FileDescriptorProto to an higher-level object
(FileDescriptor) which is easier to work with.
Post by Michael Powell
Post by Nadav Samet
Post by Nadav Samet
Plugins are pretty simple to write (look online for examples). I
would advise not writing a parser for proto files manually, since you'll be
chasing a moving target - over time language features get added albeit not
very often. It would also be hard to match the language spec perfectly.
Post by Michael Powell
Post by Nadav Samet
I guess I am not sure quite where to begin. I've downloaded the
protoc compiler, which itself has a set of proto files, but no target
language level generated code.
Post by Michael Powell
Post by Nadav Samet
Am I accurate in assuming I run those through protoc itself to my
target language, in this case I suppose C++?
Post by Michael Powell
Post by Nadav Samet
On Windows, what form does the plugin need to take? DLL? Executable
file?
Post by Michael Powell
Post by Nadav Samet
These are the sort of nibblets I think that would make it more
evident where to pick up and run with it.
Post by Michael Powell
Post by Nadav Samet
Examples of this appear to be few and far between, if at all.
Absent examples, I am working on a Boost Spirit Qi parser in the
meantime, which the learning curve is not a plateau, I can agree there.
It's not terrible to work through, and I end up with an AST that I know
forwards and backwards without too much extra effort once the rules have
done their synthetic magic.
Post by Michael Powell
Post by Nadav Samet
Cheers, thank you!
Post by Nadav Samet
On Thursday, November 1, 2018 at 6:14:54 PM UTC-7, Michael Powell
Post by Michael Powell
Hello,
I'm a bit confused. I thought I was grasping it earlier, then I
thought I wasn't ... What is the starting point developing a protoc
plugin?
What I want to accomplish is processing a .proto (v2) and
generating
Post by Michael Powell
Post by Nadav Samet
Post by Nadav Samet
Post by Michael Powell
some C# boilerplate code for it. I'm not generating a proto adapter
itself, per se, but C# DSL that would be used to support it.
The starting point for me would seem to be receiving the compiler
request, but that itself is a .proto, which leaves me a bit
confused.
Post by Michael Powell
Post by Nadav Samet
Post by Nadav Samet
Post by Michael Powell
Where do you obtain that from? Or are you starting by compiling
those
Post by Michael Powell
Post by Nadav Samet
Post by Nadav Samet
Post by Michael Powell
.protos from the bits bundled with the protoc.exe compiler?
What I want is to just have the descriptors drawn from the target
.proto(s) themselves. That's it. A full-on plugin seems like a
little
Post by Michael Powell
Post by Nadav Samet
Post by Nadav Samet
Post by Michael Powell
bit of overkill.
Suggestions?
Thanks so much!
Best regards,
Michael Powell
--
You received this message because you are subscribed to the Google
Groups "Protocol Buffers" group.
Post by Michael Powell
Post by Nadav Samet
To unsubscribe from this group and stop receiving emails from it,
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
--
-Nadav
--
You received this message because you are subscribed to the Google
Groups "Protocol Buffers" group.
Post by Michael Powell
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
--
-Nadav
--
You received this message because you are subscribed to the Google Groups
"Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
--
-Nadav
--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+***@googlegroups.com.
To post to this group, send email to ***@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.
Loading...