To deploy your own version of the calculator service, use the following link :) acorn.io/run/docker.io/dreamsofcode/grpcalculator:acorn?ref=dreamsofcode
So we're back 20+ years to SOAP with WSDL and XSD. This is not a bad thing per se but let's agree this is not an innovation. Yes, ProtoBuf is pretty efficient compared to the clunky XML messages in SOAP. And I'm excited about the Subscription capabilities in gRPC. Your videos are well-done and informative, so I'd be grateful for a video on Pub/Sub with gRPC. On the other hand, in gRPC, we lose the whole hypermedia concept of REST/HATEOAS/HAL. In well-done RESTful APIs you don't care about IDs and what endpoint to call because the resourse you GET contains hypermedia links with the exact endpoint to call (including the ID). To me, that was a truly remarkable shift in mindset when designing APIs. In closing, I appreciate this video. It illustates well that RESTful thinking does not fit to all problems, building a calculator being one of them. Thanks!
This is probably one of the best tutorials I've seen for non-noob programmers who just want to learn a new tech quickly and know the terminology already. awesome!
The main problem with grpc is that it requires your clients to add a step to their build process. It's also difficult to debug as values on the wire are very difficult for humans to read. And adding a build step to your own project is not zero cost as well. Just wait until you get confusing errors because you're running stale protobuf definitions. Generally I am a big fan of RPC interfaces. I don't do REST because you need to fit your problem to the REST format instead of modeling your API as relatively normal functions. Would love for these videos to talk about some.of the drawbacks of whatever tech you're discussing in addition to the benefits. There are few technical resources that honestly discuss drawbacks.
every few years, some new binary serialization protocol comes out. it requires code generation and build steps. then go back to text based protocols again json (or xml etc). the Hadoop ecosystem kept making a bunch of these. corba, wsdl, soap, grpc, avro, .... what we really want is a contract-first definition of the wire protocol between services so that it can be stable and versioned, and integrated documentation. we want to hide all the mTLS madness in setting things up (through sidecars/meshes, or libraries, etc). web sockets, etc. it seems like the protocols that win are the ones that are the easiest ones to reverse-engineer from a running system. grpc forcing you to not use stock HTTP/1.1 is a big drawback. now a huge problem.... you need to start transporting files and media. now the grpc stuff thinks of the world as tiny little messages. but in the middle of this, you return a link to a 4GB mp4 file. and it doesn't fit in with the rest of the system. you want to use grpc easily from javascript, etc. there is an AWS service that can take GraphQL and automate doing persistence for you. grpc has aspects of what we really want to do, but build messes and hard to reverse-engineer existing systems make this hard. building things based on on-the-wire-message definitions is good. but it's an error if you try to send a message to a service that is down. what you want is to be able to enqueue a message locally to go out. the network may be going up and down, and you turn errors into mere latency if things just queue up and drain when the network is up. offline-first clients from the browser, etc. there's a lot of half-solutions that get wired together; and we still don't have declarative protocols that keep the docs up to date, and handle the security and robustness issues; while still transporting over whatever is available. (ie: http/1.1, tls, and websocket issues).
@@mykyar9142in practice, you get forced to use tools like OpenAPI that all but assume http calls with json bodies. Your whole enterprise can force you to go through proxies; that can no longer handle non-http calls, or graphql, etc. in reality, REST assumes json bodies. i got so tired of fighting this when I was still using actual files (ie: mp4, jpg, etc), and going through proxies that just expected that all services are based on json. it's like multipart-mime, or large files are not a thing.
Protos are one of my favorite parts about dev at Google. Cods generation and static typing for not just DTOs but also method definitions is a gamechanger for internal services.
Worth noting that Protobuf isn't really an advantage specific to gRPC, as it's perfectly fine to use a binary serialization format like CBOR or Protobuf in RESTful APIs too
@@blender_wiki Because protobuf presents a sea of problems below the surface, For example, the foolish decision not to allow null and to represent 0 as null, so I have no idea for example if a temperature sensor displays 0 degrees Celsius or simply returned null, Oh sorry, you can also have a has function and then check each time during runtime. What a beauty, so useful!
@@fullmetalu320 Lol, this is absolutely bunk. Other serialization formats, like JSON and XML provide absolutely no type safety at all. The decision to NOT include NULL was deliberate! We DO NOT want the null problem, where arbitrary values can be valueless. We want the minimum amount of variants for our types, as this reduces complexity. Consider this: How many values can a JSON array named 'books' represent? A LOT, including null and "null" and 0 and {} and undefined. We have to, yes HAVE TO, handle all these possible variants in a robust application. We can make a schema, which helps a tiny tiny bit, but this still raises complexity because we have to change the schema along with our application code, or else it doesn't work. This introduces a new plethora of bugs that are not possible in protobuf.
This might depend on what you're working on but I disagree. REST is good for CRUD services, but CRUD services generally aren't that great in the first place. If you write services based around business logic/domains, RPC often fits better. That said I see a lot of pseudo-rest RPC services in the wild, where it should really be RPC but it's been written as if it's REST. For example you might see a get signed document where you post an unsigned document to it, but that's really a sign document RPC API. It's clearly not a nice API, but that's because someone was trying to use REST where RPC fit better, and they ended up with neither. Just a toy example but if you look around it happens all over the place.
I will call you on that. Tell me one real world use case that uses a *truly* RESTful API. Many say they do, yet I have seen absolutely none. REST absolutely does not fit the real world at all. That does not mean everyone should jump to gRPC as there are other options, each with different benefits.
@@aleksandarsusnjar9574 If you are just straight retrieving data, then rest has an edge. You can use a get request which can be cached more easily in internet infrastructure like cloud flare. gRPC uses post requests. 90% of what happens on the internet is retrieving data as well. If you are trying to update data objects I’d call it a toss up and for everything else gRPC has an edge. As for whether something is really “REST”, the true REST was never a real thing, just academic nonsense. The real REST is exactly what the community and professional world understand it to be, not what’s in an old outdated book that one guy wrote and almost no one uses. That said HATEOAS is an underutilized part of that original REST design, that more people should think about. The point is that real developers took the parts of “academic REST” figured out what was useful, discarded what got in the way and “Real REST” was born. You can’t truely define a real world API design until the real world actually uses it. Final note: You can make a strong argument that graphQL makes for a better data retrieval API in a lot of cases over REST, but it also doesn’t use get requests. So in the case that you are maximizing scalability for retrieval of fairly static data REST is quite peerless simply by virtue that it uses get requests and internet infrastructure can cache those better.
@@aleksandarsusnjar9574yeah, my wife used to work on a project as mobile dev. Backend API was restful written by old beard folks, they probably read a lot of interesting books... So on clients side mobile/site has to call like minimum 3 endpoints to collect necessary data for one entity, and sometimes even more sequential calls were needed as next response lead them to another urls. Needless to say the app was super slow and everyone blamed the frontend as by metrics backend responded pretty well
Re your points around 5:00 and open api specs. If you write your spec using a composition tool like boats then it's a lot nicer to write a good spec first in small chunks, then compile that into a full open API spec for your project, and use a good code generator to create a fully typed interface for your frontend and backend which has temporary mock data and automatic contract testing built in. Its a really nice way to work. It means your fe team can work on inplementing any endpoints whilst your backend team work in parallel to wire them up. Not super relevant to a gRPC video, but it's something I think apps where rest makes more sense can benefit from looking at.
Thank you for sharing the knowledge! I'm going to check out boats as you're correct, writing it in smaller chunks would be nicer. I like to take a contract first approach for the reasons you mentioned, so will be happy to have tooling to improve it!
@@dreamsofcode happy to help :) We use it for all our projects and it works really well for us. Takes a little getting used to but once you're familiar and have some nice helpers it's super fast to add or change any endpoints or schemas. Always up for finding out about new tools as well in case you come across any alternatives.
Cap'n'Proto is also worth mentioning -- from one of the original authors of protobuf, with the added benefit of server-side RPC chaining (hence the claim "infinitely faster than protobuf"). And it loads definitions at runtime, rather than having a mandatory compile-step.
I am sure you get this all the time: Your tech presentations are the most polished and visually appealing that I could find on YT. And that's only the cherry, but it's a big fat tasty cherry (for me at least). The cake is the content and how concise and easy to understand you present it. And the frequency of uploads is also spot on!
if you know any object oriented programming, think of RPC as calling static, prebuilt methods directly and perform an action - perform calculation, update database, add users, send data, receive data. REST, on the other hand simply creates a data model, which an end point can then use to perform calculations using its own methods. With REST you will have to duplicate your calculation code for each micro-service or end point, which could all be running different programming languages. If you must use REST, you would then need to have a server layer to centralize the calculations and pass the final result to each micro service end-point ... which could add delays and you will need to probably duplicate the calculation server or have it be very big to handle requests, especially if your application needs to scale and you may have latency issues if the calculations requires a lot of compute power. With RPC, you decentralize all this, and each end point can be running the same code, with predefined methods and do the required calculations on-prem, on-demand. Also, with RPC, validation, proof-of-work, conflict-resolution etc. are a lot easier. The coding process is also a lot simpler. This video is not super accurate, when it says RPC documentation is an issue ... all you need to do is understand the methods and replicate them. Definitely not for beginners, but as you begin use REST more and more, you start to understand the problems/limitations of REST, you naturally look for alternatives, like gRPC, for speed and durability.
Valorant the multiplayer game from riot games uses grpc with protobufs to synchronise players e.g such as when the server receives information when a player is hit it does an rpc call to the client to reduce player health and such. Read this in their technology article whereby they were trying to reduce the latency between the different players connected to the same server infact switching to this if i remember well reduced the latency i think by 30%. Interesting stuff
Pretty sure they use RPC, not gRPC. Game networking is usually done over UDP, while gRPC use HTTP/2. But yes, they did make a very interesting post about their networking called "VALORANT'S 128-TICK SERVERS"
Over the last year I've been using gRPC for the networking between different clients and an embedded device, such that multiple people are able to interact with some hardware through a given API. Due to protobuf, it is easy to both define well known and well structured data and methods, as well as being able to easily use this with multiple languages for the different clients.
2:57 I dunno… there’s little difference between REST and gRPC over HTTP. Both calling endpoints, both sending info. Both are requests to other computers to make the other computers do things. gRPC may be arbitrarily sending data to a generic endpoint, but it could send to different endpoints for specific purposes. REST could require the payload to be sent using protobuf or any other format. And you could devise a way to do both over a different transfer protocol, even raw sockets. It’s basically same same when you think about it critically and not try to shove it into boxes.
1. Can't you have OpenAPI generator generate that contract YAML or whatever? They have generators for like everything... 2. GRPCUI looks loosely similar to what a SwaggerUI would look for REST. I always love an easy way to use an API I am developing without having to terribly mess with curl or Postman. 3. I suppose gRPC could be used for real-time communication? Or what should we use for example for game servers game clients? Also amazing video! Love the clutterless and precise, yet brief introduction and explanation. Thank you!
I'm not sure if OpenAPI generates from code, but I know they can do Codegen to generate clients from the contract. I'll do some more research on it! You're correct, it can also be used for real time communication. Another viewer mentioned that Valorant uses it for game clients! Thank you! I'm glad you enjoyed it! Thanks for the comment as well, interesting discussion 😁
@@dreamsofcode Node Fastify is able to Generate well-documented opeaAPI 3. Spec from your JavaScript/TypeScript schema in your code. And with openapi-typescript you can generate TypeScript types from the open API 3 spec on the server for your client
I work as a data professional, a data engineer. REST APIs are a nightmare because you never know precisely the structure of your data, therefore you do not know its constraints and therefore your ability to extract information as semantic content to serve decision makers analysis is greatly diminished. gRPC is already much better and aligned with the idea of data contracts which in the end are simply a bad response to the lack of proper implementation of true data models as the relational model (not SQL which is not relational obviously).
5:28 - "Your OpenAPI contract isn't always guaranteed to match your actual service", this is only true when writing both code and OpenAPI spec by hand. I can understand why this can be frustrating. The actual recommended solution is to: - either write only your code and generate OpenAPI description from it, - or (and I recommend this solution, becomes your development becomes contract first, which is the better approach) write only the OpenAPI definition and use tools to generate code from it. E.g. for JVM, code generation (OpenAPI --> JVM bytecode) is usually done during build step, i.e. your service is guaranteed to match the definition specified in the OpenAPI file.
The lack of introspection and the incompatibility with HTTP error codes make gRPC a no-go for me. Lack of introspection means you need either home-made or gRPC-compatible tooling - all of which must have access to your gRPC definitions- to get any info out of the messages. Lack of compatibility with HTTP error indicators means that none of the vast array of existing network infrastructure that uses error codes for directing traffic, alerting, scaling, etc. will work with gRPC traffic. The benefits over REST would, at this point, need to be enormous to make such investments worthwhile.
I wish google had an official rust grpc package. Im aware tonic exists but i feel like it would make things easier for people if there was an official implementation or an existing implementation named official.
I agree. I tried around and found my way to Prost, but what's available in the ecosystem is usually lacking in features (reflection, for example) and is hard to actually configure. I do think that gRPC support is coming for Rust, though, as protoc has experimental support for Rust, although I haven't really tried it out. I also looked through the protobuf repo and progress is still being made for Rust support. Even so, Tonic does seem fairly mature. Still, an official package would be nice.
CapnProto, created by the same developers, should really be considered a replacement for protobuf and gRPC. Not only is its RPC system much lower latency, it enables you to express security properties right in the model.
You can build the OpenAPI spec directly from RESTAPI endpoints. There are many tools in most of the programming languages/server frameworks to generate accurate OpenAPI spec. We do it, so theoretically, I don't see benefit of gRPC over REST APIs when it comes to sharing/generating contracts.
Great tutorial! The lack of native web support is a bit of a bummer. The Firebase web client is also using gRPC so I guess the libraries are mature enough but still need to be downloaded to the client then.
quite interesting! having messed with D-Bus and such, i had a strange déjà vu when i saw the protocol being reflected back to the client (you can do something very similar with D-Bus via introspection) - eerily similar paradigms for IPC and RPC 😁
Wow I was hearing about protobuf but I never know it’s was link to gRPC and WOW ! I come from the world of rest then tRPC and now I do some golang but wow ! The typesafety and the autocompleting you get with that ease and performance is just incredible ! Thanks for this video, I know how to rebuild one of my personal services ^^
REST could be seen as easier than RPC because, when it's done properly (sic!) there is auto discovery ;). There is an idea behind that 2 different services could talk to each other without knowing each other. When both services know their counterparts, RPC could be the right choice. And, yop, gRPC would be a good choice. Back to the roots, to what we were doing before the web but with current technologies.
Amazing video ! Thanks for the work ! However, I think you forgot to show how to disable TLS if you are not running the server using Acorn but rather as localhost.
Deploy your own version of the calculator UI using the following link: acorn.io/elliottminns/acorn/run/docker.io/dreamsofcode/grpcalculator:acorn?ref=dreamsofcode
Great question! As long as you don't change any field numbers or types then you should be fine. gRPC will handle anything missing. Basically, don't do breaking changes!
So the Error type does not appear in the contract with the client? 🤔. Also, is there a way to include more “custom” validation in our contract?, for example in openapi you can put min/max limits, regex and other stuff? (or at least include a message to indicate something)
Hey, I know you have said before that you were going to show how you make your animations, I'm really impressed with them! In terms of tooling, you've said before that you use ffmpeg for terminal recording. What do you use for the animations or editing?
Hey! Thank you. I'm still planning on showing how I do them. I use Davinci Resolve for about 90% of my editing and then use After Effects for the rest. I'm likely going try moving to Premiere one day also!
if you want the flexibility of rest and graphql, forget grpc. it's not meant to be flexible. Few things to consider: date/time, envoy, and type generators for your web client, you just have to accept the oddity, not disadvantages, just different.
I read docs that browsers doesn't support gRPC it requires proxy server to convert json request to RPC call and get response in binary then conver back to json through proxy server. Can you please confirm ?
Wait... The sponsor is "Acorn"? I know that they are a different company, but the name is similar to that of the company which invented the ARM architecture, Acorn Computers Ltd. . Just reminded me of that. Nice video, anyways; the animations are great.
Actually I don't want to say but this is actually what Microsoft done 10 to 15 years ago except your methods can be specified in an interface forgot how it is called. It made a comeback as open source after Microsoft dropped it ah yeah CoreWCF. Anyway there are major differences between the 2 ways . I mean with RCP you got your object. Together with ORM you can really quick and dirty implement your stuff . With RESTful you have to do some reconstruction of what you have and the make an SQL request at the backend. Remember this isn't just for webpages mobil apps using this also so you have to reconstruct inside the app possible too. On top you have some overhead for your endpoints depending how and what you are using
Contract-first development is definitely the way to go. It just so happens that doing for REST using OAS is miserable. Protobuf is way easier and like you pointed out, when coupled with gRPC, it comes with other gains as well imo.
REST apis are sets of urls and json, something browsers can do out of the box Protobuf support needs to be implemented in javascript/wasm so it's not supported by the browser out of the box, meaning it's probably slower to decode
0:47 Why do you mix HTTP and REST. What do you know about REST? JSON is one of the ways to represent/encode data. You can even use a protobuf via HTTP. What makes this request an REST API? (HTTP is not required for REST. JSON is not required for REST.) The correct way to describe this is "I was creating a JSON API for HTTP 1.1. But gRPC is much better"
Thanks for the examples and the general walkthgrough but I am not so impressed with gRPC. For example in the django ecosystem with help of drf and drf spectcular you can achive automagical code dependent documentation and validation within a few lines for standard stuff. With a few lines more even customized serialisers are quickly documented and validated. My main problem in your example is that you still have to deal with the custom validation error message. Looks like the protocol doesn't even tell the user which field is invalid.
Yeah I think if you're not keen on gRPC then don't use it! For me it takes away a lot of boilerplate so I'm happy with it. As for better error handling, there's the richer error model which does allow for improved error messages, Ill probably do a video on it in the future but check it out if you're interested!
This really seems like the difference between OOP and functional programming to my admittedly pretty naive eyes. I can see where some blending is likely over time.
Ain't nobody sane writing their OpenAPI docs themselves, attribute configuration for example in C# will produce the correct results automatically and you can automate the endpoint mapping.
i still don't see the point, can someone explain why/when i should pick gRPC over REST? why is the solution in the video better than say a GET endpoint /add, which takes two numbers in query parameters and returns the result in a JSON response? (other than there being a single endpoint to run all the procedures)
/add isn't restful design. It violates what a restful app would look like which is to perform CRUD on resources. Add is an action, not a resource. You could still do it of course, you can technically do anything, but you'll end up with a poor DevEX and future devs will dislike the decision. The other benefits of grpc are: Contracts, performance, streaming, bidirectional communication, and more.
Actually, the use case of this video is a bit of a red herring. Nobody would send tiny action calls through the Intertubes for a calculator. Actions in most applications and use-cases happen on the client. The only use cases where finely granular actions need to be communicated through the internet are e.g. multiplayer games, chat clients or real-time Multi-User editing in documents (think Office365 or Canvas boards).
I like your videos, but pretending that there aren't packages to auto generate openapi documents seems a bit biased to me. I would've liked some better advantages over REST.
Have you found these to be useful / do you have any recommendations? Personally I've not found any that I've prefer as of yet, but I'm always interested in finding one for when I build out RESTful APIs
All I see with gRPC is trouble . How can someone even degug HTTP Wire and see the payload which is going out? Serialisation of data with a schema using a programming language? oh please. You need to check Middle integration tools before choosing protobufs.
Interesting, however also a bit annoying to learn yet another domain language that only interfaces with tools. I really have to look if there is a reverse solution as well, generating the proto from the data structure. I find it important to have the ability to define in both directions, as during initial development, I rather define first and contract later, while at the point when the api stands strong enough to cover all unknowns, i like to go over to the contract as source of truth. I find modern REST Frameworks that allow autodocumentation in the python world can do all this; the contract first approach was mainly a pain in the butt because of how json-schema and swagger, and later openapi conflicted with each other, and only since openapi 3.1 it is finally cross compatible; but that is rather a limitation of which library can do what better in the python world.
We have migrated all our APIs from REST to gRPC and will never go back. It was never easier to get a type-safe client implementation in almost every programming language in existence. And for legacy REST services we use gRPC Gateway to auto-generate a RESTful adapter layer.
You can send up authorization headers the same as you do in REST. So from there it's however you're performing rbac i.e. using a JWT or session based auth.
To deploy your own version of the calculator service, use the following link :)
acorn.io/run/docker.io/dreamsofcode/grpcalculator:acorn?ref=dreamsofcode
Link dead, seems like that service from acorn is dead. @dreamsofcode
@@ericowens3023 rip. Thanks for letting me know! I'll figure out another solution!
So we're back 20+ years to SOAP with WSDL and XSD.
This is not a bad thing per se but let's agree this is not an innovation.
Yes, ProtoBuf is pretty efficient compared to the clunky XML messages in SOAP.
And I'm excited about the Subscription capabilities in gRPC. Your videos are well-done and informative, so I'd be grateful for a video on Pub/Sub with gRPC.
On the other hand, in gRPC, we lose the whole hypermedia concept of REST/HATEOAS/HAL.
In well-done RESTful APIs you don't care about IDs and what endpoint to call because the resourse you GET contains hypermedia links with the exact endpoint to call (including the ID). To me, that was a truly remarkable shift in mindset when designing APIs.
In closing, I appreciate this video. It illustates well that RESTful thinking does not fit to all problems, building a calculator being one of them.
Thanks!
Actually we're back 30 years or so to COM and CORBA IDL files.
Who does GRPC pubsub? Redis is text.
This is probably one of the best tutorials I've seen for non-noob programmers who just want to learn a new tech quickly and know the terminology already. awesome!
You are doing great content! Love to see gRPC with Rust as for GO or Python you can find in the documentation.
The main problem with grpc is that it requires your clients to add a step to their build process. It's also difficult to debug as values on the wire are very difficult for humans to read. And adding a build step to your own project is not zero cost as well. Just wait until you get confusing errors because you're running stale protobuf definitions.
Generally I am a big fan of RPC interfaces. I don't do REST because you need to fit your problem to the REST format instead of modeling your API as relatively normal functions.
Would love for these videos to talk about some.of the drawbacks of whatever tech you're discussing in addition to the benefits. There are few technical resources that honestly discuss drawbacks.
every few years, some new binary serialization protocol comes out. it requires code generation and build steps. then go back to text based protocols again json (or xml etc). the Hadoop ecosystem kept making a bunch of these.
corba, wsdl, soap, grpc, avro, ....
what we really want is a contract-first definition of the wire protocol between services so that it can be stable and versioned, and integrated documentation. we want to hide all the mTLS madness in setting things up (through sidecars/meshes, or libraries, etc). web sockets, etc.
it seems like the protocols that win are the ones that are the easiest ones to reverse-engineer from a running system. grpc forcing you to not use stock HTTP/1.1 is a big drawback.
now a huge problem.... you need to start transporting files and media. now the grpc stuff thinks of the world as tiny little messages. but in the middle of this, you return a link to a 4GB mp4 file. and it doesn't fit in with the rest of the system. you want to use grpc easily from javascript, etc.
there is an AWS service that can take GraphQL and automate doing persistence for you. grpc has aspects of what we really want to do, but build messes and hard to reverse-engineer existing systems make this hard.
building things based on on-the-wire-message definitions is good. but it's an error if you try to send a message to a service that is down. what you want is to be able to enqueue a message locally to go out. the network may be going up and down, and you turn errors into mere latency if things just queue up and drain when the network is up. offline-first clients from the browser, etc.
there's a lot of half-solutions that get wired together; and we still don't have declarative protocols that keep the docs up to date, and handle the security and robustness issues; while still transporting over whatever is available. (ie: http/1.1, tls, and websocket issues).
I'm sorry, but what is REST format? (JSON is not a part of the REST, HTTP is not the part of the REST)
@@mykyar9142in practice, you get forced to use tools like OpenAPI that all but assume http calls with json bodies. Your whole enterprise can force you to go through proxies; that can no longer handle non-http calls, or graphql, etc. in reality, REST assumes json bodies. i got so tired of fighting this when I was still using actual files (ie: mp4, jpg, etc), and going through proxies that just expected that all services are based on json. it's like multipart-mime, or large files are not a thing.
If you design your API property sincecthe beginning gRPC have zero drawbacks for a private/internal API.
😂 protobuf in every language has fuction "printDebugString()" it will print human readable value of your message
I'm very proud that I'm following you for a while, you're making really great content. Keep it up ! :)
Protos are one of my favorite parts about dev at Google. Cods generation and static typing for not just DTOs but also method definitions is a gamechanger for internal services.
Worth noting that Protobuf isn't really an advantage specific to gRPC, as it's perfectly fine to use a binary serialization format like CBOR or Protobuf in RESTful APIs too
Protobuf is a burden, let's stop pretending it's not.
Why serialize in REST when you can use a proper tool? Laziness?
@@blender_wiki
Because protobuf presents a sea of problems below the surface,
For example, the foolish decision not to allow null and to represent 0 as null, so I have no idea for example if a temperature sensor displays 0 degrees Celsius or simply returned null,
Oh sorry, you can also have a has function and then check each time during runtime.
What a beauty, so useful!
@@fullmetalu320 I've never said so, and that's also why I mentioned CBOR. I was referring to the general advantages of binary serialization formats.
@@fullmetalu320 Lol, this is absolutely bunk. Other serialization formats, like JSON and XML provide absolutely no type safety at all. The decision to NOT include NULL was deliberate! We DO NOT want the null problem, where arbitrary values can be valueless. We want the minimum amount of variants for our types, as this reduces complexity. Consider this: How many values can a JSON array named 'books' represent? A LOT, including null and "null" and 0 and {} and undefined. We have to, yes HAVE TO, handle all these possible variants in a robust application. We can make a schema, which helps a tiny tiny bit, but this still raises complexity because we have to change the schema along with our application code, or else it doesn't work. This introduces a new plethora of bugs that are not possible in protobuf.
This is an impeccable video. High level explanation and a clear and pretty step-by-step example implementation. Well done.
gRPC is great, but REST fits most of the real world use case. We need more native tooling around gRPC
This might depend on what you're working on but I disagree.
REST is good for CRUD services, but CRUD services generally aren't that great in the first place. If you write services based around business logic/domains, RPC often fits better.
That said I see a lot of pseudo-rest RPC services in the wild, where it should really be RPC but it's been written as if it's REST. For example you might see a get signed document where you post an unsigned document to it, but that's really a sign document RPC API. It's clearly not a nice API, but that's because someone was trying to use REST where RPC fit better, and they ended up with neither. Just a toy example but if you look around it happens all over the place.
I will call you on that. Tell me one real world use case that uses a *truly* RESTful API. Many say they do, yet I have seen absolutely none. REST absolutely does not fit the real world at all. That does not mean everyone should jump to gRPC as there are other options, each with different benefits.
@@aleksandarsusnjar9574 If you are just straight retrieving data, then rest has an edge. You can use a get request which can be cached more easily in internet infrastructure like cloud flare. gRPC uses post requests. 90% of what happens on the internet is retrieving data as well. If you are trying to update data objects I’d call it a toss up and for everything else gRPC has an edge.
As for whether something is really “REST”, the true REST was never a real thing, just academic nonsense. The real REST is exactly what the community and professional world understand it to be, not what’s in an old outdated book that one guy wrote and almost no one uses. That said HATEOAS is an underutilized part of that original REST design, that more people should think about. The point is that real developers took the parts of “academic REST” figured out what was useful, discarded what got in the way and “Real REST” was born. You can’t truely define a real world API design until the real world actually uses it.
Final note: You can make a strong argument that graphQL makes for a better data retrieval API in a lot of cases over REST, but it also doesn’t use get requests. So in the case that you are maximizing scalability for retrieval of fairly static data REST is quite peerless simply by virtue that it uses get requests and internet infrastructure can cache those better.
@@aleksandarsusnjar9574yeah, my wife used to work on a project as mobile dev. Backend API was restful written by old beard folks, they probably read a lot of interesting books... So on clients side mobile/site has to call like minimum 3 endpoints to collect necessary data for one entity, and sometimes even more sequential calls were needed as next response lead them to another urls. Needless to say the app was super slow and everyone blamed the frontend as by metrics backend responded pretty well
What are your guys opinion on GraphQL?
I've been thinking about learning gRPC in Go for weeks now, and this video is perfect! Thank you!
Re your points around 5:00 and open api specs. If you write your spec using a composition tool like boats then it's a lot nicer to write a good spec first in small chunks, then compile that into a full open API spec for your project, and use a good code generator to create a fully typed interface for your frontend and backend which has temporary mock data and automatic contract testing built in.
Its a really nice way to work. It means your fe team can work on inplementing any endpoints whilst your backend team work in parallel to wire them up.
Not super relevant to a gRPC video, but it's something I think apps where rest makes more sense can benefit from looking at.
Thank you for sharing the knowledge! I'm going to check out boats as you're correct, writing it in smaller chunks would be nicer.
I like to take a contract first approach for the reasons you mentioned, so will be happy to have tooling to improve it!
@@dreamsofcode happy to help :) We use it for all our projects and it works really well for us.
Takes a little getting used to but once you're familiar and have some nice helpers it's super fast to add or change any endpoints or schemas.
Always up for finding out about new tools as well in case you come across any alternatives.
Cap'n'Proto is also worth mentioning -- from one of the original authors of protobuf, with the added benefit of server-side RPC chaining (hence the claim "infinitely faster than protobuf"). And it loads definitions at runtime, rather than having a mandatory compile-step.
I am sure you get this all the time: Your tech presentations are the most polished and visually appealing that I could find on YT. And that's only the cherry, but it's a big fat tasty cherry (for me at least). The cake is the content and how concise and easy to understand you present it. And the frequency of uploads is also spot on!
Super great video as always ! I wasjust looking for a video about RPC and LSP two days ago and here is yours! perfect timing, thanks !
if you know any object oriented programming, think of RPC as calling static, prebuilt methods directly and perform an action - perform calculation, update database, add users, send data, receive data. REST, on the other hand simply creates a data model, which an end point can then use to perform calculations using its own methods. With REST you will have to duplicate your calculation code for each micro-service or end point, which could all be running different programming languages. If you must use REST, you would then need to have a server layer to centralize the calculations and pass the final result to each micro service end-point ... which could add delays and you will need to probably duplicate the calculation server or have it be very big to handle requests, especially if your application needs to scale and you may have latency issues if the calculations requires a lot of compute power. With RPC, you decentralize all this, and each end point can be running the same code, with predefined methods and do the required calculations on-prem, on-demand. Also, with RPC, validation, proof-of-work, conflict-resolution etc. are a lot easier. The coding process is also a lot simpler. This video is not super accurate, when it says RPC documentation is an issue ... all you need to do is understand the methods and replicate them. Definitely not for beginners, but as you begin use REST more and more, you start to understand the problems/limitations of REST, you naturally look for alternatives, like gRPC, for speed and durability.
looks like somebody picked up the dropped ... SOAP 🤣
bad joke, i know, but this video triggered some dejavu's for me
good video, as always. thx
Haha I really enjoyed that one 😂
gRPC is great for internal server server but I think it falls short everywhere else.
Valorant the multiplayer game from riot games uses grpc with protobufs to synchronise players e.g such as when the server receives information when a player is hit it does an rpc call to the client to reduce player health and such. Read this in their technology article whereby they were trying to reduce the latency between the different players connected to the same server infact switching to this if i remember well reduced the latency i think by 30%.
Interesting stuff
That's really interesting! I'm going to see if I can find the article. Thank you for sharing
Pretty sure they use RPC, not gRPC. Game networking is usually done over UDP, while gRPC use HTTP/2. But yes, they did make a very interesting post about their networking called "VALORANT'S 128-TICK SERVERS"
Over the last year I've been using gRPC for the networking between different clients and an embedded device, such that multiple people are able to interact with some hardware through a given API. Due to protobuf, it is easy to both define well known and well structured data and methods, as well as being able to easily use this with multiple languages for the different clients.
The browser does not support it tho
2:57 I dunno… there’s little difference between REST and gRPC over HTTP. Both calling endpoints, both sending info. Both are requests to other computers to make the other computers do things. gRPC may be arbitrarily sending data to a generic endpoint, but it could send to different endpoints for specific purposes. REST could require the payload to be sent using protobuf or any other format. And you could devise a way to do both over a different transfer protocol, even raw sockets. It’s basically same same when you think about it critically and not try to shove it into boxes.
Would love to see gRPC implementation using Elixir
We use it at work. There should be lots of tutorials online
one of the best contents I have seen about grpc on yt. Thx
1. Can't you have OpenAPI generator generate that contract YAML or whatever? They have generators for like everything...
2. GRPCUI looks loosely similar to what a SwaggerUI would look for REST. I always love an easy way to use an API I am developing without having to terribly mess with curl or Postman.
3. I suppose gRPC could be used for real-time communication? Or what should we use for example for game servers game clients?
Also amazing video! Love the clutterless and precise, yet brief introduction and explanation. Thank you!
I'm not sure if OpenAPI generates from code, but I know they can do Codegen to generate clients from the contract. I'll do some more research on it!
You're correct, it can also be used for real time communication. Another viewer mentioned that Valorant uses it for game clients!
Thank you! I'm glad you enjoyed it! Thanks for the comment as well, interesting discussion 😁
@@dreamsofcode Node Fastify is able to Generate well-documented opeaAPI 3. Spec from your JavaScript/TypeScript schema in your code.
And with openapi-typescript you can generate TypeScript types from the open API 3 spec on the server for your client
okay, I'll say it. gRPC with Rust!
I work as a data professional, a data engineer. REST APIs are a nightmare because you never know precisely the structure of your data, therefore you do not know its constraints and therefore your ability to extract information as semantic content to serve decision makers analysis is greatly diminished. gRPC is already much better and aligned with the idea of data contracts which in the end are simply a bad response to the lack of proper implementation of true data models as the relational model (not SQL which is not relational obviously).
5:28 - "Your OpenAPI contract isn't always guaranteed to match your actual service", this is only true when writing both code and OpenAPI spec by hand. I can understand why this can be frustrating. The actual recommended solution is to:
- either write only your code and generate OpenAPI description from it,
- or (and I recommend this solution, becomes your development becomes contract first, which is the better approach) write only the OpenAPI definition and use tools to generate code from it.
E.g. for JVM, code generation (OpenAPI --> JVM bytecode) is usually done during build step, i.e. your service is guaranteed to match the definition specified in the OpenAPI file.
The lack of introspection and the incompatibility with HTTP error codes make gRPC a no-go for me.
Lack of introspection means you need either home-made or gRPC-compatible tooling - all of which must have access to your gRPC definitions- to get any info out of the messages.
Lack of compatibility with HTTP error indicators means that none of the vast array of existing network infrastructure that uses error codes for directing traffic, alerting, scaling, etc. will work with gRPC traffic.
The benefits over REST would, at this point, need to be enormous to make such investments worthwhile.
That’s pretty much why it’s only used in server to server communications at any big tech company
I wish google had an official rust grpc package. Im aware tonic exists but i feel like it would make things easier for people if there was an official implementation or an existing implementation named official.
I agree. I tried around and found my way to Prost, but what's available in the ecosystem is usually lacking in features (reflection, for example) and is hard to actually configure.
I do think that gRPC support is coming for Rust, though, as protoc has experimental support for Rust, although I haven't really tried it out.
I also looked through the protobuf repo and progress is still being made for Rust support.
Even so, Tonic does seem fairly mature. Still, an official package would be nice.
Is that a mac? Or a VM? I think gh you were using arch? If it is mac, can you share your mac setup? What terminal emulator are you using with tmux?
CapnProto, created by the same developers, should really be considered a replacement for protobuf and gRPC. Not only is its RPC system much lower latency, it enables you to express security properties right in the model.
You can build the OpenAPI spec directly from RESTAPI endpoints. There are many tools in most of the programming languages/server frameworks to generate accurate OpenAPI spec. We do it, so theoretically, I don't see benefit of gRPC over REST APIs when it comes to sharing/generating contracts.
Great tutorial! The lack of native web support is a bit of a bummer. The Firebase web client is also using gRPC so I guess the libraries are mature enough but still need to be downloaded to the client then.
quite interesting! having messed with D-Bus and such, i had a strange déjà vu when i saw the protocol being reflected back to the client (you can do something very similar with D-Bus via introspection) - eerily similar paradigms for IPC and RPC 😁
also reminds me of GraphQL a lot, but perhaps less confusing
rpSOAP looks good. Now do an automatic REST/JSON to gRPC translator API to use gRPC via JSON API
This is much cleaner than shipping an ODATA endpoint or trying to integrate with one (especially complex ones)
Wow I was hearing about protobuf but I never know it’s was link to gRPC and WOW ! I come from the world of rest then tRPC and now I do some golang but wow !
The typesafety and the autocompleting you get with that ease and performance is just incredible ! Thanks for this video, I know how to rebuild one of my personal services ^^
REST could be seen as easier than RPC because, when it's done properly (sic!) there is auto discovery ;). There is an idea behind that 2 different services could talk to each other without knowing each other.
When both services know their counterparts, RPC could be the right choice.
And, yop, gRPC would be a good choice. Back to the roots, to what we were doing before the web but with current technologies.
I remember writing COM and Corba idl files for the exact same reasons/promises. Good times.
Amazing video ! Thanks for the work !
However, I think you forgot to show how to disable TLS if you are not running the server using Acorn but rather as localhost.
Informative video and good explanation, thanks. There is only one small point: connection won't be closed because of log.Fatal()
Deploy your own version of the calculator UI using the following link:
acorn.io/elliottminns/acorn/run/docker.io/dreamsofcode/grpcalculator:acorn?ref=dreamsofcode
How do you make updates to an existing grpc? Wouldnt that require update frontend and backend at the same time so they follow the same contract again?
Great question! As long as you don't change any field numbers or types then you should be fine. gRPC will handle anything missing.
Basically, don't do breaking changes!
So the Error type does not appear in the contract with the client? 🤔.
Also, is there a way to include more “custom” validation in our contract?, for example in openapi you can put min/max limits, regex and other stuff? (or at least include a message to indicate something)
Out of the box. No.
But! There are plugins that can add validation rules with protobuf, protoc-gen-validate I believe is one of them
Hey, I know you have said before that you were going to show how you make your animations, I'm really impressed with them! In terms of tooling, you've said before that you use ffmpeg for terminal recording. What do you use for the animations or editing?
Hey! Thank you. I'm still planning on showing how I do them. I use Davinci Resolve for about 90% of my editing and then use After Effects for the rest. I'm likely going try moving to Premiere one day also!
if you want the flexibility of rest and graphql, forget grpc. it's not meant to be flexible. Few things to consider: date/time, envoy, and type generators for your web client, you just have to accept the oddity, not disadvantages, just different.
It's awesome to see you cover this topic!
I read docs that browsers doesn't support gRPC it requires proxy server to convert json request to RPC call and get response in binary then conver back to json through proxy server. Can you please confirm ?
It’s one of the last things he says at the end of the video
Wait... The sponsor is "Acorn"? I know that they are a different company, but the name is similar to that of the company which invented the ARM architecture, Acorn Computers Ltd. . Just reminded me of that.
Nice video, anyways; the animations are great.
Acorn Labs I believe is their full name! They're the same founders as Rancher, I believe. Really cool company!
is it possible to create the protofiles or the protobuffs in client without it being in the same directory structure? Using reflexion maybe?
When I worked at Google we had to used gRPC for our apis, the language we used was Java for a microservice
How can i design my terminal to look as good as yours?
gRPC: Huh, that's cool.
Using `touch` to trigger a file watcher: OMG THIS IS SO GENIOUS!
Actually I don't want to say but this is actually what Microsoft done 10 to 15 years ago except your methods can be specified in an interface forgot how it is called. It made a comeback as open source after Microsoft dropped it ah yeah CoreWCF. Anyway there are major differences between the 2 ways . I mean with RCP you got your object. Together with ORM you can really quick and dirty implement your stuff . With RESTful you have to do some reconstruction of what you have and the make an SQL request at the backend. Remember this isn't just for webpages mobil apps using this also so you have to reconstruct inside the app possible too. On top you have some overhead for your endpoints depending how and what you are using
Contract-first development is definitely the way to go. It just so happens that doing for REST using OAS is miserable. Protobuf is way easier and like you pointed out, when coupled with gRPC, it comes with other gains as well imo.
6:12 wow, which shell are u usinng?
Can somebody explain on what is meant by the sentence native web support? How is it there for REST but not gRPC? Any example if possible.
REST apis are sets of urls and json, something browsers can do out of the box
Protobuf support needs to be implemented in javascript/wasm so it's not supported by the browser out of the box, meaning it's probably slower to decode
Great video of grpc. Would love to see an example in nodejs
I shall add it to the backlog!
In terms of message size it should be smaller then rest? Right?
It reminds me of a mix between old bit-defined messages and more scalable REST.
0:47 Why do you mix HTTP and REST. What do you know about REST? JSON is one of the ways to represent/encode data. You can even use a protobuf via HTTP. What makes this request an REST API? (HTTP is not required for REST. JSON is not required for REST.)
The correct way to describe this is "I was creating a JSON API for HTTP 1.1. But gRPC is much better"
Great video but you didn't mention the streaming capabilities of gRPC
i have been using grpc and love it.A question grpc uses HTTP 2 is there a way to use rpc with HTTP 3.
Yes. But not all languages/frameworks support it (yet). ASP.NET support it
Thanks for the examples and the general walkthgrough but I am not so impressed with gRPC.
For example in the django ecosystem with help of drf and drf spectcular you can achive automagical code dependent documentation and validation within a few lines for standard stuff. With a few lines more even customized serialisers are quickly documented and validated.
My main problem in your example is that you still have to deal with the custom validation error message. Looks like the protocol doesn't even tell the user which field is invalid.
Yeah I think if you're not keen on gRPC then don't use it! For me it takes away a lot of boilerplate so I'm happy with it.
As for better error handling, there's the richer error model which does allow for improved error messages, Ill probably do a video on it in the future but check it out if you're interested!
Would love to see gRPC in action in C# or Rust
From what it seems of your video grpc is similar to soap with wsdl but much easier to mantain and program
love to see a video of gRPC with python.
This really seems like the difference between OOP and functional programming to my admittedly pretty naive eyes. I can see where some blending is likely over time.
Good video, An unrelated question for video, but how could I clear the path to folder after entering the command? I like that style 😅
gRPC is really good, I love using it. Only downside is that it needs envoy to support web client.
I sometimes find gRPC extremely slow, even when doing the simplest of RPCs
At 12:29 you are showing us the docker instance and not the deployment on acorn 🤔
Great video though.
Examples in java, typescript, python, and comparison
Your explanation was excellent!
Thank you!
REST was designed for hypermedia and its still the best option.
now gRPC is cool and all... but where can we find that ZSH theme?
Hey your video was very well produced I like the way you explained it!
Thank you!
That bluescreen triggered a small panic moment, and then I realized I'm running Linux now..
Ain't nobody sane writing their OpenAPI docs themselves, attribute configuration for example in C# will produce the correct results automatically and you can automate the endpoint mapping.
So effectively it's SOAP but binary
We've come full circle 🤣
Very cool video, I think I will try the grpc in some way
i still don't see the point, can someone explain why/when i should pick gRPC over REST? why is the solution in the video better than say a GET endpoint /add, which takes two numbers in query parameters and returns the result in a JSON response? (other than there being a single endpoint to run all the procedures)
/add isn't restful design. It violates what a restful app would look like which is to perform CRUD on resources. Add is an action, not a resource.
You could still do it of course, you can technically do anything, but you'll end up with a poor DevEX and future devs will dislike the decision.
The other benefits of grpc are:
Contracts, performance, streaming, bidirectional communication, and more.
Actually, the use case of this video is a bit of a red herring. Nobody would send tiny action calls through the Intertubes for a calculator.
Actions in most applications and use-cases happen on the client. The only use cases where finely granular actions need to be communicated through the internet are e.g. multiplayer games, chat clients or real-time Multi-User editing in documents (think Office365 or Canvas boards).
I like your videos, but pretending that there aren't packages to auto generate openapi documents seems a bit biased to me. I would've liked some better advantages over REST.
Have you found these to be useful / do you have any recommendations? Personally I've not found any that I've prefer as of yet, but I'm always interested in finding one for when I build out RESTful APIs
All I see with gRPC is trouble . How can someone even degug HTTP Wire and see the payload which is going out? Serialisation of data with a schema using a programming language? oh please. You need to check Middle integration tools before choosing protobufs.
Interesting, however also a bit annoying to learn yet another domain language that only interfaces with tools. I really have to look if there is a reverse solution as well, generating the proto from the data structure. I find it important to have the ability to define in both directions, as during initial development, I rather define first and contract later, while at the point when the api stands strong enough to cover all unknowns, i like to go over to the contract as source of truth.
I find modern REST Frameworks that allow autodocumentation in the python world can do all this; the contract first approach was mainly a pain in the butt because of how json-schema and swagger, and later openapi conflicted with each other, and only since openapi 3.1 it is finally cross compatible; but that is rather a limitation of which library can do what better in the python world.
finally gRPC/protobuf makes sense to me
This was an excellent video! Keep it up!
Nice channel. Valuable content here ! Subscribed.
Thank you!
Great video. Concise and well-explained.
Kind request for new video:
gRPC in Python with examples
So as the title, when is restful isn't enough that require rpc? You made a simple example, but rhat can be done with rest as well.
Unfortunately, the latency is higher in both Rust Tonic and Go FIber.
8:55 hmm, i could use that tip for invoking maven archetype generate commands too :)
using make for such small things can be neat (:
Sorry, I didn't get Dijkstra reference when talking about REST APIs, I don't think he would've cared?
GOTO
He was avidly against it
We have migrated all our APIs from REST to gRPC and will never go back. It was never easier to get a type-safe client implementation in almost every programming language in existence. And for legacy REST services we use gRPC Gateway to auto-generate a RESTful adapter layer.
WTF? I have total type safety with my REST interfaces based on OpenAPI, API gateways and Java Springboot and Angular JS.
How do you handle role level authorization?
You can send up authorization headers the same as you do in REST. So from there it's however you're performing rbac i.e. using a JWT or session based auth.
Would love to see gRPC in java. This is not a wish. This is a challenge
Great video dreams!
Thank you!
Great topic! What's this zsh theme🙄🙄?
what about Cap'n Proto? When grpc isn't enough
Excelent content. Thanks!
Would've been better to include streaming in the video as well.