[protobuf] Where to get well-known-types from?

[Jean-Philippe Ouellet]

Hello,

tl;dr - What is the correct way to make well-known types available to
protoc when targeting Go?

In the protobuf documentation [1], well-known types are imported with proto
package path "google/protobuf/[...].proto".

In order for protoc to find such files, it is my understanding that they
must be available within a "google/protobuf" subdir of a provided import
path (protoc -I vendor/[...]).

However, github.com/golang/protobuf provides these files only under a
"ptypes/protobuf" directory, which can not be successfully pointed to by
protoc ("ptypes" != "google", import fails with not found).

They are available under google/protobuf within
github.com/protocolbuffers/protobuf/src and
github.com/googleapis/googleapis, among other places, but these are full of
other extraneous files.

I could also play games with protoc arguments like
`Mgoogle/protobuf/timestamp.proto=github.com/golang/protobuf/ptypes/protobuf/timestamp`
to remap every path from google to ptypes, but this most definitely seems
wrong...

Any hints appreciated.

Thanks,
Jean-Philippe

[Chao-Hung Sun]

You were trying to access those .proto from raw source code, what you need
is downloading/installing protobuf compiler (protoc), Please see official
github installation instruction
(https://github.com/protocolbuffers/protobuf/blob/master/src/README.md). If
you are on Visual Studio on Windows, you can also just download the
Google.Protobuf.Tools nuget pacakge. The package is then bootstrapp as the
following structures:

For this example, ".../package" is where the protobuf compiler and its
resources are bootstrapped.

.../package/protoc
.../package/google/protobuf/*.proto <---these are all the well known types

in your protoc generation argument, you will then add "-I=.../package" to
instruct protoc where to search for .proto

if you have a test.proto file that has *import google/protobuf/any.proto
in it. and test.proto resides in .../src, *the protoc generation will look
something like this

*protoc test.proto -I=<absolute path of .../src> -I=<absolute path of
.../package> <and the rest is your platform specific flags>*



On Wednesday, November 28, 2018 at 2:15:44 AM UTC-8, Jean-Philippe Ouellet

Post by Jean-Philippe Ouellet
Hello,
tl;dr - What is the correct way to make well-known types available to
protoc when targeting Go?
In the protobuf documentation [1], well-known types are imported with
proto package path "google/protobuf/[...].proto".
In order for protoc to find such files, it is my understanding that they
must be available within a "google/protobuf" subdir of a provided import
path (protoc -I vendor/[...]).
However, github.com/golang/protobuf provides these files only under a
"ptypes/protobuf" directory, which can not be successfully pointed to by
protoc ("ptypes" != "google", import fails with not found).
They are available under google/protobuf within
github.com/protocolbuffers/protobuf/src and
github.com/googleapis/googleapis, among other places, but these are full
of other extraneous files.
I could also play games with protoc arguments like
`Mgoogle/protobuf/timestamp.proto=
github.com/golang/protobuf/ptypes/protobuf/timestamp`
<http://github.com/golang/protobuf/ptypes/protobuf/timestamp> to remap
every path from google to ptypes, but this most definitely seems wrong...
Any hints appreciated.
Thanks,
Jean-Philippe

[Jean-Philippe Ouellet]

Post by Chao-Hung Sun
You were trying to access those .proto from raw source code,

Correct, and this is intentional. I would like to have our build
system pull the protos directly from their canonical source so that
they can be easily kept in sync with any upstream changes (rather than
the alternative of forking a copy of them all into our monorepo).
Thanks, but I am already successfully bootstrapping protoc itself.
That is not the issue.

Post by Chao-Hung Sun
For this example, ".../package" is where the protobuf compiler and its resources are bootstrapped.
.../package/protoc
.../package/google/protobuf/*.proto <---these are all the well known types
in your protoc generation argument, you will then add "-I=.../package" to instruct protoc where to search for .proto
if you have a test.proto file that has import google/protobuf/any.proto in it. and test.proto resides in .../src, the protoc generation will look something like this
protoc test.proto -I=<absolute path of .../src> -I=<absolute path of .../package> <and the rest is your platform specific flags>

Sure, but what is considered the canonical upstream source of google's
well-known types? Is it indeed
github.com/protocolbuffers/protobuf/src? There are several other
protobuf repos also with local copies.

Given that we work exclusively in Go, would
github.com/golang/protobuf/ptypes be a more appropriate source? That
repo does not have the proto files under the same paths (e.g.
"google/protobuf/timestamp.proto") as seems to be indicated in the
protobuf docs though [1].

I suspect perhaps the .proto files may only be present in the
golang/protobuf repo for the purpose of generating the .pb.go files
in-place via `protoc --go_out=paths=source_relative:. [...]`, and
perhaps people get their .protos (to make available to protoc for
compiling protos with import the well-known types) directly from
github.com/protocolbuffers/protobuf, but I would love if someone with
more experience could confirm or deny whether this is a recommended
way to do things.

[1]: https://developers.google.com/protocol-buffers/docs/reference/go-generated

Thanks,
Jean-Philippe

Post by Chao-Hung Sun

Post by Jean-Philippe Ouellet
Hello,
tl;dr - What is the correct way to make well-known types available to protoc when targeting Go?
In the protobuf documentation [1], well-known types are imported with proto package path "google/protobuf/[...].proto".
In order for protoc to find such files, it is my understanding that they must be available within a "google/protobuf" subdir of a provided import path (protoc -I vendor/[...]).
However, github.com/golang/protobuf provides these files only under a "ptypes/protobuf" directory, which can not be successfully pointed to by protoc ("ptypes" != "google", import fails with not found).
They are available under google/protobuf within github.com/protocolbuffers/protobuf/src and github.com/googleapis/googleapis, among other places, but these are full of other extraneous files.
I could also play games with protoc arguments like `Mgoogle/protobuf/timestamp.proto=github.com/golang/protobuf/ptypes/protobuf/timestamp` to remap every path from google to ptypes, but this most definitely seems wrong...
Any hints appreciated.
Thanks,
Jean-Philippe