Creating gRPC code from scratch is a laborious undertaking that, for many companies, makes the cost of implementing gRPC outweigh the benefits. At the very least, starting from scratch means converting gRPC data structures (a.k.a messages) expressed in text and numbers to and from the Protocol Buffers binary format that's intrinsic to facilitating communication between gRPC clients and a server. (See Figure 1, below)
Figure 1: gRPC uses the Protocol Buffers binary format as the lingua franca of data exchange
Also, building your own gRPC API code from scratch means writing the business logic for the services defined in the API's .proto file. And, if that's not enough, you'll need to write the code that routes, calls to and from specific services defined for the gRPC API. As you can see, it's a lot of work.
Fortunately, the gRPC community has created a tool that does a lot of the mundane work that goes with creating code for a gRPC API. The tool is called protoc. protoc is a binary executable that allows programmers to auto-generate backing code for an API as defined in the API's .proto file.
The objective of this article is to explain how to get protoc up and running to auto-generate gRPC code in a variety of programming languages. We'll cover auto-generating code in JavaScript, C#, and GoLang. This code will be generated using the .proto file that's part of an example demonstration project named SimpleService. Also, we'll share links to a number of lessons in the Katacoda interactive learning environment that we at ProgrammableWeb have created for your benefit.
What is a .proto file?
A .proto file is a description of a gRPC API written in the Protocol Buffers language specification. Protocol Buffers is a binary format. It is NOT a self-describing language. Thus, there needs to be a common "dictionary" used by both the gRPC client and server to encode and decode text and numbers into the Protocol Buffers binary format. This common dictionary is the .proto file. While the diagram below that illustrates this shows a single .proto file accessible to both client and server, it's possible for the client and server to have access to their own copies of this file. Such an implementation would involve the dynamic model discussed in the previous part of this series (See the figure below)
The following is an example of a simple .proto file named coolcar.proto This .proto file defines a single data structure, car along with a service, CoolCarService that publishes a procedure GetCar(). The procedure retrieves a car by vehicle identification number (VIN).
syntax = "proto3";
package coolcars;
message Car {
string make = 1;
string mode = 2;
int32 year = 3;
string VIN = 4;
}
message CarRequest {
string VIN = 1;
}
service CoolCarService {
rpc GetCar (CarRequest) returns (Car) {}
}Understanding the Scope of .proto Plugins
As mentioned above, protoc is the binary executable used to auto-generate boilerplate code. protoc uses the plugin design pattern. Thus, a plugin developer writes an add-in for protoc that will auto-generate code for the given language in a particular way. For example, there is a JavaScript plugin that generates JavaScript code that represents objects that correspond to the messages in a .proto file. These auto-generated objects will contain the data fields that correspond to a particular message. Also, the auto-generated code attaches logic to each object that will serialize and deserialize the message's data to and from the Protocol Buffers binary format.
One the other hand, the GoLang plugin, for example, will also auto-generate code that describes the message's data along with its serialization and deserialization logic as well as the business logic for the services defined in the API's .proto file.
Each plugin works a little differently. Thus, it takes a bit of time to master the details of each. However, in terms of real-world adoption, developers tend to focus on the plugins that are relevant to their company's preferred programming languages.
Getting protoc
The protoc tool is operating system specific. There are protoc versions for OS X, Windows, and Linux. You can also build a protoc executable directly from source code using a C++ compiler. But, the easiest way to use the tool is to download the operating system specific executable directly.
You can get a complete list of protoc releases here. You'll access a particular release by clicking the hyperlinked heading for a particular release version. (See Figure 2, below)
Figure 2: The protoc executable is available for each release of Protocol Buffers
Then, you'll be taken to a list of zip files. Some of these zip files are relevant to working with protoc in a particular language. But, be advised, the prebuilt protoc compiler is not in any of these zip files. Rather, the prebuilt version of protoc is in the zip file that is operating system specific. (See Figure 3, below)
Figure 3: The protoc executable is available in an operating system specific zip file
If you want to learn the details about working with the autogenerated boilerplate code in the language of your choice, take a look at the tests and examples in the language-specific zip file as shown below in Figure 4.
Figure 4: The language-specific zip files contain examples working with the protoc compiler
Using protoc to auto-generate code
After downloading the operating system specific zip file, you'll unzip it and copy the file, protoc to a folder that is in your system's path. For example, if you are running on a Linux system, you'll navigate to folder that you unzipped from the downloaded zip file and then execute the following copy command:
cp bin/protoc /bin/protocOf course, this assumes that the folder /bin is defined in your system's PATH environment variable. Once protoc is added to your path, you'll be able to use it to auto-generate code from anywhere on your computer.
Once you have protoc installed on your computer, you'll use it to auto-generate gRPC code according to a particular plugin's scope. The command shown below in Listing 1 is an example of using protoc to auto-generate gRPC code for JavaScript. The details about the command follows.
protoc --proto_path=proto --js_out=library=simpleservice,binary:js_autogen \
proto/simple.proto
WHERE
- protoc is the command that invokes code auto-generation
- --proto_path is the option the declares the relative location of the folder that contains the relevant .proto files
- -js_out=library=simpleservice is the name of the Javascript file that will be created. This file will contain JavaScript objects that map to the messages defined in the .proto file. These objects will have functions for serializing and deserializing the object's data to and from the Protocol Buffers binary format.
- binary:js_autogen is the option setting that defines the target folder location for the auto-generated files. This folder must exist before auto-generation is invoked.
- proto/simple.proto is the location of the .proto file that is used as the basis for auto-generation
Listing 1: Using protoc to auto-generate gRPC boilerplate in Javascript
Figure 5 below shows the JavaScript code that is the result of running the protoc command described above in Listing 1. The code displayed within the Visual Studio Code IDE.
Figure 5: protoc for JavaScript will auto-generate objects that are described as messages in the relevant .proto file.
As mentioned above, protoc supports a variety of languages for auto-generation. Listing 2 below shows the protoc command for auto-generating code in .NET/C#.
protoc --proto_path=proto --csharp_out=cs_autogen --csharp_opt=file_extension=.cs \
proto/simple.proto
WHERE
- protoc is the command that invokes code auto-generation
- --proto_path is the option the declares the location of the folder that contains the relevant .proto files
- --csharp_out=cs_autogen is the name of the folder where protoc will store the auto-generated code. In this case, the output folder is cs_gen
- --csharp_opt=file_extension=.cs defines the file extension that will be applied to the output file. In this case, the file extension is defined to be .cs, which will result in an output file named Simples.cs. By default protoc names output files according to the name of the .proto file, which is simple.proto. Also, the plugin pascal cases the output file name. (The pascal case naming convention is one in which the first character in a word is upper case.) If the value of file_extension was set to --csharp_opt=file_extension=.cs, the output file will be Simple.pb.cs
- proto/simple.proto is the location of the .proto file that is used as the basis for auto-generation
Listing 2: Using protoc to auto-generate gRPC boilerplate in .NET/C#
Figure 6 shown below displays the results of using protoc to auto-generate C# code.
Figure 6: The results of using protoc to auto-generate C# code
Get hands-on experience auto-generating the ProgrammableWeb gRPC code on Katacoda
ProgrmmableWeb publishes free interactive lessons on Katacoda that demonstrate the concepts and practices for auto-generating gRPC code, as described in this section. For example, one lesson shows you how to use protoc to auto-generate boilerplate code in both JavaScript and C#. The boilerplate code contains objects that correspond directly to the messages described in the demonstration API's .proto file. Also, you can interact with the code you generate directly using the Visual Studio Code IDE that's built-in to the Katacoda Interactive Learning Environment
You'll find the JavaScript/C# scenario on Katacoda.
Once you've generated the objects for the messages defined in the gRPC API's .proto file, you can use them in your development efforts.
Be advised that in the case of JavaScript and C# code demonstrated in the examples above, the auto-generating boilerplate code does not provide all the logic you need to create a fully functional gRPC API. The boilerplate code creates objects that represent the gRPC messages defined in the .proto file. Each object will contain the serialization and deserialization logic that's necessary to encode and decode the object's data to and from a Protocol Buffers binary. However, you will still need to create a server and then program the procedures as well as the data access logic that your API needs to do its work. As mentioned above, it can be a daunting undertaking.
Fortunately, there are projects out there that provide such a framework. gRPC Elixir for Python is one such project.
Also, as we described above, some plugins will auto-generate the boilerplate code for gRPC services as well as messages. For example, the GoLang plugin for protoc generates both messages and services. You can learn the details by taking ProgrammableWeb's interactive lesson on Katacoda that demonstrates auto-generating code under GoLang. The lesson is based on the GoLang tutorial published on grpc.io. You can find the interactive lesson. A screenshot of the lesson is shown below in Figure 7.
Figure 7: The interactive lesson Katacoda that demonstrates how to use protoc to auto-generate message and service code under GoLang
Another project that generates both messages and procedures is grpc tools which can be found on the npm website. grpc tools, which is published by Google, provides an enhancement to protoc in the form of a binary named grpc_tools_node_protoc. grpc_tools_node_protoc will auto-generate boilerplate code in JavaScript for the gRPC messages and procedures defined in the given .proto file. To get a more detailed understanding of grpc-tools, take the interactive lesson on Katacoda that we at ProgrammableWeb created. You can find the lesson. Figure 8 below shows a screenshot of the interactive lesson
Figure 8: The Katacoda interactive lesson for auto-generating Node.js creates code for both the procedures and messages defined in the gRPC .proto file
The examples shown above are but a few of the many available on grpc.io. There are examples in C++, Dart, Objective C and PHP, to name a few. It's worth taking the time to review the gRPC tutorials for your chosen programming language. But, no matter what language you use, remember that getting an auto-generated code that's useful is not a matter of executing a single command at the command line. You'll also need to create a gRPC server that knows how to talk to the code that's auto-generated. Creating a server that uses auto-generated gRPC code is a topic we cover in the article, Writing a gRPC API Server, which you can find here.
Putting It All Together
Auto-generating boilerplate code provides many benefits for gRPC software development. Having developers create mundane code is a waste of time and money. It's easier to just auto-generate the boilerplate code for message and services as defined in a proto file and then have developers apply their expertise to mission-critical programming tasks such as implementing validation rules against incoming data, applying business logic in procedures and storing data.
However, as powerful as auto-generating code using protoc is, it's not an end-all or be all. The scope of auto-generation activity varies according to the protoc plugin being used. As a result, a programmer will need to add code that consumes the auto-generated code. The advantage is the programmer will be doing high impact tasks rather than mundane work. The gains in efficiency are apparent.
protoc and its derivative, grpc_tools_node_protoc, bring a lot of power to gRPC programming. The trick is to know the limitations of a particular protoc plugin in terms of auto-generation and be able to provide the necessary programming expertise after code auto-generation has taken place.