What event queue would you recommend? We are currently doing a polling REST API for our longer running micro services and it feels "wrong" to continue this pattern while we are growing.
You're talking about reading data from different microservices so HTTP and gRPC are the correct way to go (if anybody is trying to read data by passing messages on queues then they're beyond help). I would also argue that gRPC isn't worth the complexity overhead if you're not transferring large amounts of data.
Also curious on what event queues you would recommend. Some I'm familiar with are Kafka for streaming data and Rabbit MQ. Totally based on use case. A video breaking down which queues to use when would be awesome.
Hasura is literally a server deployed as a microservice. It IS the server between the database and the client so it falls right in line with your "acceptable use cases for GraphQL" diagram. Memes and hot takes that paint the whole world black and white are great for driving Twitter and UA-cam engagement (here I am, engaging with your content) but at the very least you could try to do an honest review that adheres to your own "rules". If your take is that systems that have any kind of GraphQL -> SQL translation layer in their stack is evil at least be honest with yourself and viewers and say that instead of this nonsense about requiring a server in the middle.
I think it quickly becomes a mess when microservices talk to each other. You could do that with a message queue instead. Microservices can put messages into this queue and read/listen to messages, and suddenly your architecture's dataflow is way more understandable. If you think about it long and hard, the microservices talking to each other problem is similar to two-way data binding, and the solution I'm proposing is similar to what Redux does to solve the issue in a sense. Unidirectional Data Flows on the server^^
Oh I like that framing! It was a big ah-ha for me when I read the react-query docs (and reinforced by some of Thea’s videos) and shifted my thinking away from querying and to “state sync”. It’s a little more of a stretch to see an event bus that way, but could be very useful!
@@efkastner well yeah it's a stretch :D but if you have a service which is dedicated to reading the queue and mutating the database, then it could actually look like a reducer, that gets the current state from the db, and the "action" is the message. What you'd get out of this is very easy swapping of microservices, and replayability? Probably there are no race conditions anymore as well while writing the db. Never tried, though I'm just thinking in theory. (It feels like a bunch of unnecessary complexity, so maybe it doesn't make sense)
the between server communication is likely better dealt with as an event, posted to an event queue / log rather than a realtime connection that could / will fail.
Will repeat myself. Yes, you may use gRPC for microservices, there's no evil in it. Use event-driven architecture if you have very good reasons for it (having proven that synchronous doesn't work in practice *for your specific problem*, rather than just assuming it wouldn't or that it's somehow "not in the spirit of microservices"), if you have a damn good observability infrastructure and a strategy for dealing with transactions. Don't do event-driven because you've read it in a popular microservices book that takes a dogmatic stance on it, ignoring the enormous cost in added complexity and maintainability.
we use Hasura in production its is actually quite amazing... our backend is written in golang using the gqlgen library and we auto generate around 5M lines of compliant golang code because of the graphql schema Hasura provides... Theo you fail to understand that companies operate in different levels. If my company is worth 1B+ do you really think we would use Hasura? Of course not, its better to build your own infrastructure because you have the cash/manpower to do so. Hasura is fundamentally game-changing for startups unders ~25M and can easily handles thousands of concurrent subscriptions through Hasura Cloud. There is a reason Hasura is worth so much these days, and you fail to understand Hasura for what it truly is. You can easily eject out of Hasura because it does NOT interfere with your postgres database whatsoever
What even is the argument against Hasura? Or even supabase or whoever put gql straight into postgress? There's so many applications where 95% of the endpoints just pass data straight to/from the db. I genuinely don't understand why Hasura or some BaaS would be bad here
He called out Hasura for using GraphQL as SQL replacement on the backend, but then everything he described as the appropriate use of GraphQL can be done by Hasura. So Hasura is not the problem, only the problem is misuse of tools.
I think that gap that you show between "ServerClient" is a really good way to visualize that use case, and how useful GraphQL will be in comparison to the opposite use case when the server + client are "touching". I went down the path of using GraphQL a couple of years ago, and I can see it being great when dealing with multiple languages and/or dev teams, or with 3rd party APIs where I'm only the client. But for my big project where I'm the solo fullstack dev, and using TypeScript on both the nodejs backend + also the frontend... over the last couple of years I've slowly realized that I was just making heaps of extra work for myself dealing with 2 typing systems (TS + GraphQL) instead of just the 1 (TS). Plus all the extra complexity of having to write queries in another language, and all the extra tooling needed there. Overall it's slowed the whole project down a lot, without really any benefit for this project in particular. My use case in this instance was pretty much the opposite of when GraphQL shines. I'm just about to start removing GraphQL from this project. It's my own system (not for a client), and the frontend is still in its infancy... somewhat due to graphql slowing me down. I'll be taking a look at tRPC, and either using that, or building something customized for myself that has some similarities. Either way, my project isn't going to be any less typesafe than graphql in the end.
There’s a lot of tools that fix a lot of your problems. Such as nexus or type graphql to completely remove writing your code in the graphql schema language. Everything is kept in JavaScript with these tools. Which means your typescript is automatically created for you, and is the main thing you’re using
This is a surprisingly good video given that I disagree with the two contentious points you make: 1. GraphQL shouldn’t be used directly in front of DBs. 2. GraphQL shouldn’t be used for inter-service queries. And here’s why: 1. The case for DB backed GraphQL APIs (e.g. Hasura) I think you're arguing against the idea of having micro-services use GraphQL to query their own data, and I 100% agree with that. However, the value of this tech when used correctly is as a rapid project accelerator that won't tie you into loads of tech debt down the line. Creating a schema directly in front of a DB is fine provided that the schema is a completely logical view of your domain, since in that case you’re free to change the implementation details later. Hasura makes this possible to do since you can create mapping rules that allow you to present your DB logically. Given how fast it is to create very performant services this way, it’s actually a really good option for start-ups and green-fields, and I’d particularly recommend it, since although it’ll allow you to get your tech. stack implemented quickly, the logical schema you (with care) end up, will mean you don’t get stuck with a monolith when it’s time to scale, and you have a really simple path to moving to something more scalable on the back-end as needed. You even say as much yourself, and I quote: > This is what I love about GraphQL. None of this back-end implementation detail exists to me anymore. 2. The case for using GraphQL for inter-service queries If the thing you really love about GraphQL is that none of the implementation detail need concern you as a front-end dev, then isn’t that an equally useful trait for other services on the back-end? Or, stated another way, if the benefit to this formal interface is the ability to change implementation detail provided queries flow through this interface, then isn’t it better that more of the traffic flows through it? You don't explicitly say why you disliked this pattern, but perhaps your objection is the performance hit from going through some additional resolvers. If so, then it’s worth noting that this doesn’t apply in the case that services are implemented as subgraphs, since in that case you can then either query them directly, or query them via a low latency router that adds no perceivable overhead. Thoughts?
Thanks for making this video! I've been a graphql dev for 1 year and while I've gotten used to the mechanics of the system, I've never really understood the "why" behind it all. I work at an online retail company and this "middle tier" project was started more than two years ago and changed hands a couple times, so now people like me have no idea what the original point even was. This video helped me form a better idea of what it's for, and what my company is still missing in order to better leverage this technology. I might even be able to impress my manager by formulating a well-defined plan for improvements.
BTW, I am in a scrum group with front end developers and I hate it. I feel like team meetings are a complete waste of my time because they are only talking about React 99% of the time and I work on the Graphql server and resolvers. I'm going to make a new push to break up the stupid "full stack team" pattern we're using. It's stupid, at least for the way this company functions.
In case of distributing requests between microservices, GraphQL can address this on it's own via GraphQL Federation. The same applies for requests initiate from one service to another. Instead of calling the service directly it calls the federation. This removes the requirement for the service to know about each other. The only information that the service needs is where the federation gateway is. This allows you to easily merge and split services without need for application wide refactoring.
Good video, good points. Ever since I learned GraphQL, it made me appreciate the art of crafting API's, and made me like to *start* with GraphQL schema's when designing new object-oriented systems. However, *my* very favorite feature of GraphQL is its introspection capabilities. I do wish the spec would "require" it to always be public, so that clients could reasonably employ tools that could dynamically generate UI's, or even new machine-to-machine processes, based solely on schemas..
It feels good that the same arguments I had some years ago during an internal discussion, that GraphQL is probably best utilized as a gateway to your whole microservice clusterfuck rather than talking to the db's itself. The gateway also prevents the frontend developer to know about the slight differences in naming between the different microservices, as they only need to care about the contract GraphQL schema defines./
I'd argue that the GraphQL client should always be considered as part of the frontend domain as most UIs are data-driven and frontenders knows best what data dependencies each component has. Over time frontenders needs to make changes to the frontend code, and then it'll lead to a much more streamlined and frictionless process if they adjust the graphql schema / resolver at the same time. If the frontenders needs to align and communicate with the backenders every time they need to make a change impacting the graphql sschema, it may risk leading to technical debt instead, as there's a big chance that a workaround will be taken instead if time is an issue. The graphql server as I see it is a BFF (Backend For Frontend), meaning its for moving complexity and business logic from the frontend/client/browser to instead handle and prepare it in the graphql server. Therefore there are very strong arguments for writing the graphql server in node.js (javascript, typescript, rescript, purescript or some other compile-to-js language). Other than that I agree about server to server comm should be gRPC, and that is is only intended as a thin interface/proxy without direct db connectivity... but the split between FE and BE is where I disagree. This is based on my experience from working this way.
I was told GraphQL is used over REST primary because you can tell it to "filter" the response and thus you get exactly what you want. The problem is, you can emulate this with REST too if you really wanted. Either in the query params or inside the body you can tell the server what you want then your REST api will filter it based of your request.
If you design your graphQl schema you can literally fetch all the data in your database or just a single scalar value or anything in between in ONE REQUEST. It took me a while to understand how powerful this ability is but once I saw it, I can't go back to REST! The key is all in the design of you graphQL Schema plus custom DataFetchers to solve N+1 problems!
A nice thing that you brought up here Theo is Data contract which is probably the most common problem that any software solution without proper planning face. Explicit schemas as contracts is a must. For the most complex systems, we even have dedicated versioned schema repositories nowadays thank god. GraphQL was one of the moves towards this in a more narrowed sense than when talking about architectures. Info near the definition, flexible and extensible metadata, all come very handy. Also a very bad practice I see a lot is to try to come up from the beginning with an ubiquitous data model that will not be needing updates any sooner (that is an illusion and probably it will become harder to cleanup those unwanted attrs and add useful ones ). My tip is, unless it is a very fundamental attribute of the particular entity wait until you implement the feature that will need the attribute. I get the creeps when i see those humongous entities that look like they came from an enterprise software of the 90s.
You should look into federated graphql. With that at the gateway each service maintains it's own schema. This provides a consistent interface for inter service requests.
Came here to say this. The use case he made where the payment service needs to update the user is perfect for a combination of stitching and federation. You make the payment, that mutation returns the user, and you just spread the user, which, at least in Relay, updates that state.
You didn’t give any real reasons why graphql server is bad. I recently started a project with aws and when looking for resources to connect my dynamodb tables to my front-end, aws-amplify came up among the first solutions with graphql the default. I think I agree it’s probably not the best route, but I like to know why.
Hasura is great for base CRUDs, an events driver to fire off serverless fns, while adding your own apis (more serverless fns) for business logic in highly efficient and no boilerplate ways. Did I mention serverless? And Hasura is the (graphql) gateway. Theo is right: He doesn't know what he is talking about. He like most are too stuck on microservices. Go serverless and event driven, then Hasura makes sense. PS. You also must hate Firebase. Me too, Firebase is a trap. Hasura is conceptually like Firebase as a BEaaS, but better in every way: complexity and performance scaling, transparent and open source, graphql ecosystem
At work our app uses GraphQL stitching to combine Hasura with some custom NestJS APIs. We're in the process of porting to a single tRPC server and it is SHOCKING how much code we're removing. To be fair, a lot of that code is generated. However, the more significant change is to our dev process where everything is 10x easier.
Yeah I fell into the “wow Hasura is amazing” hype… but ended up having the same experience in the end: it just sugar coated the complexity, didn’t really deal with it. But it’s not bad! Just somehow I failed to find a really good use case for it even though I tried.
I went through Apollo’s free courses, which are super high quality, and learned GraphQL was not for the project I was working on. I’m really glad I went through the courses still, because now I’ll know when to use it.
With regards to microservices, GraphQL Federation can be used to provide a unified API while keeping services isolated; services define their schemas and ownerships. As for using GraphQL to communicate _between_ services, you'd be better off with using something like gRPC, Protobuf or other event queue systems alike. GraphQL only make sense when you want to expose APIs to one or more clients.
GraphQL for server to server is perfectly valid use case. I have used it in production with 7 microservice and it was a pleasure to use. Each service just needs to expose an internal graphql schema at say /internal/graphql that can be consumed by other microservices.
@@82TheKnocKY Yes exactly. This makes everything type safe. You can use graphql-codegen to make sure that all the queries are valid in CI so the contract enforcement can be automated. As far as GRPC is concerned, GraphQL can be just as fast if you use HTTP2 on server side (easy using a service mesh). The only saving using GRPC is that it has a smaller payload because of Protobuf encoding but the difference won't be significant.
@@gt4adolfo Event driven is not always good. For instance, if order service wants to make sure that the customer has a valid email, it should call customer service using HTTP and not Event. Event Driven is good for async communication.
@@quant-daddy is possible to achieve it using event driven... youre coupling microservices... for this reason exists Aggregation events. the same microservice have to check for example the "email". is not necesary coupling.
It would be interesting if you could properly address your beef with hasura. Perhaps a dedicated post explaining exactly why you don't like it? Or even discussion panel? I'll be there to meet your objections in a constructive way
I have been using Wordpress as a headless CMS, and using WPGraphQL gives you real power, and its so superior to the WP Rest API, in a lot of ways. It allows you to just don’t really care about API routes defined by Wordpress, and just fetch your things (with a proper caching solution, which can be a problem using GQL…).
We use javascript for things it's not meant for. Here's the thing, if the tool works well for your use case and there's no much bad trade-off, then you should ideal it.
Absolutely agree David. I am doing this since some time now: if it works i'll do it. It's not only a question of do I need to use GraphQL or not? The question is: what's the best tool I am confortable with to work with faster !
So you have multiple microservices that call the same database? Congrats! you have a distributed monolith) One of the biggest advantages of microservices is the ability to scale them independently. You just multiply it by 0 when you join everything in DB. With a schema like this you will end up using 600$ RDS DB instance on AWS (without traffic). I know this because I did this)
@@zzzz6538 In microservice architecture, every service should own its own data. If that requires a database, then the service should have its own database. This allows the service to define its own data models and scale its infra independently. Say you have 10 different services using the same DB. Most of the time each service needs 10 units of power (simplification for brevity), but occasionally each service will need up to 30 units of power. You could have 1 DB that provides 300 units of power all the time ( 10 services * 30 max units of power) or 10 DB that provide 30 units of power. 10 small DBs @ 30 units of power will be cheaper than 1 DB @ 300 units of power. Using current EC2 prices, 10 machines with 4 vCPU and 16GiB ram would cost $650/mo. vs. 1 machine with 40 vCPU and 160 GiB ram would cost $850/mo. It's about 25% cheaper splitting to multiple instances. Far more important than price though is independent data modeling and reducing complexity. If you outgrow a single DB instance and have to shard your data, determining how to shard data across 1000 tables using a single identifier for consistent hashing while maintaining foreign key integrity per sharded instance would be an absolute nightmare. My head is spinning just thinking about how to do it. If, instead, each of your services use their own DB: 1.) you're less likely to reach a scale where sharding is required. 2.) the relationship complexity between the few tables that belong to the service is going to be much easier to manage and determine how data should be split and which identifier is best for hashing while keeping related entities in the same shard.
I don’t often comment on videos but those I do are ones that help guide me in the right direction, light bulb moments if you will. It’s been a while since I’ve played around with programming and naturally things have changed a bit. I’ve been researching tools & tooling to possibly build small apps to manage my seemingly unmanageable life. TLDR: this was insightful
"when graphql happened for the first time ever there was a real line drawn between the backend and the frontend, that i didn't have to worry about the other side". That's not because of graphql, but the lack of basic knowledge about design patterns, SOLID principles, etc.
hey @Theo, heard about federated graphql? Have a look, it does let you have individual services also speak graphql, then use apollo router as your gateway (no code, just deploy), then your individual services can talk to other services through the router as well.
Another thing that I find GraphQL helpful is with defined schema, graphql client (apollo) can automatically merge same objects from different response which help consistency in the UI without manually doing in that in reducer or in react-query callback. How do you normally deal with this when you are using trpc/rest
Instead of just crying... "Dont do this.. dont put it here, it was not built for DB to server comms, ...this is just bad", a more mature aproach would be to state WHY it should not be used for DB to server interactions and to let people come to the conclusion about its usage by themselves. Everything that you said in the context of server & client applies to DB and server as well. There are many shops that have a dedicated DB team making modelling decisions, and providing sqls or SPs for middleware teams. For the server too, where business logic is to be applied, getting the data in a vendor agnostic way is desirable.. Imagine data stored in postgres, oracle and dgraph. Having a graphql interface between the DBs and the server really decouples the business layer from changes to the underlying model.
Strong agreements and disagreements lol. I think that your rant about why GraphQL works between the server and client is spot on. The part about not using it in the backend could use some more elaboration vs saying "don't do that." The reason why GraphQL has not been good in the DB layer is because it was typically implemented in mainly NoSQL or SQL databases i.e. Fauna, Hashua, etc. And life gets complicated when trying to use GraphQL between collections and tables. The ONE caveat to this IMO is a Graph database. Its literally called Graph Qurey Language lol. It lines up with how GraphDB's are structured way better than any custom GraphDB language. Its the reason why I've been using Dgraph for most of my projects and have been really enjoying it. Now even though I feel Dgraph is the only solution that got it right, it is certainly the exception and not the rule... and given that the dev team for the project has been all over the place, I don't like to say "it's the only solution that works and therefore use it". If anything my conclusion from using it is that GraphQL is not suitable for SQL/NoSQL databases but if GraphDBs ditched their own custom query language for GraphQL, the world would be a better place. Even EdgeDB (while still a Graph) is implemented on top of PostgreSQL which is probably why maintaining their GraphQL adoption has been difficult. A more accurate statement would've been "For the love of Christ, it's not called Collections & Tables Query Language" lol. Also as a side note, the other reason Dgraph works is because its designed to replace most of the server layer while also providing the capability to lambda call any server you want to setup. So in your diagram the DB would be the middle layer and it can still communicate easily with microservices. This approach has been great for building new applications and significantly reduces the code needed in my microservices. If this pattern was adopted by other DBs, the world would be a better place.
Totally agree with this and I commented something similar. Neo4j also has a library that generates GraphQL from your graph schema that essentially just writes cipherQL under the hood. This saves you from having to deal with cipherQL and instead you can use GraphQL for CRUD operations on your database. Small nit: graphDBs are a type of NoSQL DB, but I know you meant things like mongoDB.
With time and missions done, I realize that not so many people are understanding GraphQL. Do you think it's linked to its "not well explained" documentation?
I still don't understand how gql is better than trpc for communication (assuming you use TS for your server ofc)... You could define typescript types for your BE-FE contract just as easily as you define a gql schema....only you don't need to use a different syntax for it....
GraphQL can actually be a good solution of querying graphDBs. Neo4j with AuraDB has a library that autogenerates GraphQL from your graph schema your use for CRUD operations to your database (as a functional alternative to cypherQL)
@@t3dotgg I think that opinion comes from your focus on SQL databases. Plenty of graphDBs, like Neo4j, support the use of GraphQL for CRUD operations. Neo4j has a library where it auto-generates GraphQL from your schema. Under the hood this GraohQL calls CipherQL to write to the database, but this allows the user to use GraphQL’s often better syntax over CipherQL depending on the use case. Neo4j has an article about the library on their website.
I don't understand why GraphQL is bad on communicating backend and db (for example Hasura). It is very good to throw away writing a lot of code that is just getting some data from db. It is easy to make some fragments and auto-generate all other for project needs. The most disappointing is that I do not hear any reasons to not use GraphQL between servers and db. I agree, that at first GraphQL is built for using it between client and server and it is so good for it. But it doesn't mean, that it isn't for server side needs. I think with some work it can be one of the best solution for db and servers communicating. Maybe not in all cases, but in many cases.
My company attempted to adopt GQL for the exact thing you say not to adopt it for in this video ("hey I read somewhere that GQL is good for handling complex datasets or whatever, let's use it") for 5 disastrous months in 2021 before going back to REST. Guys, he is right... GraphQL is NOT an API/DB gobetween. Trust me on this.
Isn’t Hasura just auto-generating CRUD GraphQL (and recently REST) API, instead of querying DB’s using GraphQL? Also, can you share your thoughts on public GraphQL APIs?
I don't really agree with theo's position on this either, hasura has been all about having a low to no code backend in minutes for me. It solves the boilerplate nonsense of adding CRUD to your backend, and you can use actions for complicated logic. It's unbeatable when it comes to building something quickly You just define queries in your frontend, run graphql-codegen and use apollo-hooks, there, you're done One of the argument I've seen is that you should never expose your database schema "directly" to your client, without much explanation as to why. Well first of all you can use SQL views and functions and build your hasura graphql schema around that, and second: why not ? permissions in hasura are made to make sure your users see only what they're allowed to ... Anyway, hasura rocks, it will make your app building so much faster and it is ridiculous to dismiss it over some ideology, we've used and hacked technologies to use them where they were not supposed to for ever, won't stop now
Look kids. There is no reason you can't use GraphQL directly on top of a DB, but if you team is bigger than a handful of people you'll have a hard time not regressing something somewhere. Basically Theo is saying separation of concerns is good. And basically necessary as your team scales. There is not technical limitation here.
Theo can you explain why having graphql directly on a database is bad? Isn’t it just like a serverless implementation of graphql? I mean ultimately with hasura o assume there’s still a server in between the database and the client, or do I assume wrong?
All these things kinda sucks! Today no one has done the right solution for client-server communication that's frustrating . I will do my best to fix this
Awesome that you spent the time explaining this. I've seen crazy usages of GraphQL, even all the information on the client managing a client side database 🥺
I wondered about the title of this video, so I went to google trends and guess what? For the last 5 years, searchs related to GraphQL are going well, especially on 2022. People is indeed very interested on GraphQL. So there you have it.
What sort of a question is this? REST APIs is a staple and it won't go away ... ever. Not until we remove HTTP. They are simple to consume and fast to build. IMHO most new projects should start with a REST API and if they have some scaling issues then they should consider GraphQL, tRPC. REST is the perfect fit for small teams. Just don't overengineer it with dumb HTTP semantics. I've seen overengineered REST APIs and they are not fun to work with. Also keep in mind that REST doesn't mean JSON. You can use REST APIs and pass XMLs to keep extra object information. The downside of JSON is that it loses a lot of context. Granted most of the time you don't need the said context. But it just goes without saying.
I was wondering why is what hasura or platformatic do so wrong, yes ofc depends on the scope of the project, some are out of scope of what those frameworks can do, but for simple stuff why not?
I was thinking about the graphql between microservices - is it good idea to use GraphQL API for between microservices just to have a single API implementation over all infrastructure ? Then you can expose all these graph ql apis using the apollo federation/proxy service. The key thing to mention is, that microservices themselves will not communicate directly between each other but instead using kafka topics. What do you think about that Theo ?
A minute and thirty in I had to explain to my OH why I yelled 'fuck you Theo'. In all seriousness, this is great content. The misuse of GraphQL is a mess I've inherited from the version of me from last year who didn't really know what he was installing, and I wake up everyday thinking about pulling the trigger on a fat rewrite
So wait, i am conflicted now, am i an idiot for using graphql for storing data i need from a graphql api into my database, and then also for fetching the data i need for my frontend ? I am very confused by this video
If you speak in absolutes, you better have a good reason for your opinions. No reason is given for the opinions expressed in the video. For a lot of smaller applications having a built in GQL interface to a database can make a lot of sense, because you don't need to always hand code resolvers. You can use GQL wherever you want, including server to server communication, it's no worse than REST and you might not feel like dicking around with Protobufs and type definitions.
Very good explanation on how to use GraphQL. I have a question, what if we use GraphQL between each service and the Gateway? So, each Microservice exposes its own schema and the Gateway is in charge of resolving it according to the client's requests, in other words what I mean is, it is a bad idea to have graphql between the client and the server, but also to have it between each microservice and the gateway. Thanks
I agree 100%. Have you used federation? (E.g. Apollo's solution for joining multiple graphql endpoints into one) would be curious to see your take on that pattern :-)
Yeah the "GraphQL Resolver Gateway" should surely be "GraphQL Federation Server" and then GraphQL totally does handle the call end to end back to each micro-service. You also get the added benefit of writing queries that can cross over micro-service boundaries which all get resolved seamlessly.
But why? GraphQL makes trade-offs, and personally I find the trade-offs worth when communicating between client/server, not so much with server/server, cause now each microservice needs to also be written with GQL in mind.
@@kefkef5 good question. The reason is: say I want a "global" profile service for my users. But, depending on the department / apps they've interacted with in our ecosystem, the metadata is subject to change from a wholistic standpoint. So, federation would help us build a collective graph for the profile of any given user.
@@kefkef5 There are some situations where microservice1 { field: microservice2 { data } } is insanely powerful, and I'm not aware of too many alternatives. The caller doesn't need to know or care where any data is and the GraphQL Federation server glues everything together seamlessly with no significant performance penalty that my APM can detect.
Can't you do that anyway when you have your GQL service calling the rest services to ask for the data and glue it together on the GraphQL? Now the backend is also forced to provide you a GraphQL endpoint, which they might think it's not worth the time investment compared to exposing the data through rest.
The video is informative, but I sincerely hope that viewers don't adopt this particular approach for designing microservices architecture. My expectation is that the showcased method is solely for educational purposes. When it comes to communication between servers or microservices, it is essential to employ gRPC( Efficiency, Language Agnostic, Streaming, Contract-First Design ) for tasks requiring an immediate response, following the Function as a Service (FaaS) paradigm. On the other hand for scenarios where an immediate response is not necessary or possible, communication should be facilitated through an orchestrator or a stream-processing pipeline.
Hi Theo, I've only worked on the front end so far and now I just begun learning React for almost 1 week now learning useState and useEffect at the moment. What are the next things would you recommend I should focus on after React? I think Redux would be the next step? Maybe GraphQL after Redux? There is lots to learn and work on. At the moment I am focused on trying to create a scheduler App for myself. Kind of like a journal mixed with dates and goals you could set for yourself, not a to-do App. But my end goal is to land a job as a full stack developer with JavaScript as the client side and Java as the backend. I would appreciate your honest and brutal opinion, thank you Theo.
Have you watched my "your goals kinda suck" video yet? Think that will be helpful. Also don't learn graphql or redux if you don't immediately need them, both are SUPER heavy to take on w/ a solo project. Best learned in a big codebase that you don't run lol
I'm using graphql for a new SAAS project and we designed it as a supergraph/gateway for all rest microservices so client all access the same graphql api and the rest API remain private into the cluster. sure you won't add logic on the database nor creating any kind of monolithic api for transforming to sql
So the main thing that GraphQL solves, being the hard separation between FE and BE, is based on a middle contract layer. Doesn't Swagger/Openapi do this as well?
In the quest to make everything more complicated, graphql is great at that. How can we make it more complicated? One way of solving inter team workflow is to replace 2 teams with 1 team. Ie have devs which do both front and backend.
Learnt more from this guy in 5 videos than the entire last 5 years of working at my jobs. Excellent content man - please - keep it coming. Very grateful to have somebody with a good head on their shoulders who understands back-end as well talking React/TS. Very cool.
Thank you Theo, reading comments I realize i'm not alone, I've worked with it before but never really understood it. I finally get why and where GraphQL is valuable.
Graphql uses schemas, which defines a standardized contract between front-end en back-end. In REST you can also create schemas but you have to implement your own or use tools like OpenAPI. Another benefit of graphql is reducing data overfetching. In REST you might need to do several request to get data you need but in Graphql you just define the fields you need and make a single request
Api contracts in form of soap or grpc or res api provide a way to abstract the complexity of be from Fe and vice versa. Its not just graphql. If you cannot communicate between BE and FE, its people problem and not a tech problem. Work on each of above tech.
I disagree that GraphQL should not be used as an interface for a database. Especially when there is zero to none business logic and your application is basically glorified table editor direct access can make it way simpler. With proper precautions such as object type policies, complexity calculators, query timeouts etc... you can mitigate all issues that might arise. I'd argue that you in fact should delegate as much GraphQL schema as possible to the database as then you can reap benefits such as projections and single query (one execution plan).
Great video. I just signed on to use a data company who uses GraphQL to access their data. I have many years of SQL and so far I am not a fan of GraphQL. No default aggregate / Count?? Because it is like you said for the UI and not the database. I need to ask this company on why they are using GraphQL as their interface to their database.
Maybe you are not actually hitting their database directly, but rather are making a bulk entity request to one or more services? They may be considering all client requests equal, regardless of whether they are coming from an individual-entity record-viewing UI, or from a batch reporting job. But yeah, if you're trying to aggregate normalized, relational data, SQL would be better. But that doesn't mean they will be ok with giving access to query their database directly.
Isn't GraphQL also great for situations where you have many different consumers of your API? It's hard to predict every need of every possible client (website front-ends, mobile apps, games, smart watches, refrigerator screens etc.) so GraphQL makes sense, since the clients can define their own data needs in a query. This cuts down on superfluous requests and unnecessary data being transferred.
The critique that Hasura is an anti-pattern is based on the idea that you don't want to expose your db schema to public clients. It's valid. However Hasura has features to limit exposure of fields and modify permissions. The cost/benefit for small time devs is huge with Hasura. If you are in a highly-resourced company you can use Hasura and then put another GraphQL Yoga engine in front of you want another layer of abstraction and employ more devs.
First time i hear that people misuse GraphQL as a sort of new SQL. Eww. But i can see where the initial confusion could have come from. If you treat a backend service as just a Persistence thing that just happens to have a web interface you might get a wrong idea to use it there.
For the love of God, yes there are scenarios where people really have some ideas that graphql is somewhat a blackbox on top of your api which magically spits out the data that you need on the frontend. But if we ignore alot of bshit that is written on the internet and turn on our brains and read some documentation we come acrosd that gql is actually as a technology ment to be used to fetch only the data that the frontend needs eg. prevent fetching unnecessary data. 2nd its tech agnostic meaning that you could use gql with any framework or library thats out there react, angular for FE and node.js, java etc for the backend. 3rd thing to note is that devs generally have no clue what gql is about so when they see the promise of drilling the properties down in gql they think oh mann so sweeet we can now has our gql queries translated to sql and skip out all of our orm/sql/nosql thingys and just use gql ( which is common nonsense). 4th, there isnt yet a technology where you write 2 lines of code and it makes you a app which you need.
I don't understand the reference to Hasura, when it is interpreting postgres schema, plus reading some other metadata to generate an API (runs on a server). This API is communicated with via graphql to the client. I hardly expose any of the underlying database via Hasura, and mostly use more advanced postgres views and Hasura actions to define my full set of backend services, which then is all stitched together so my client can use it. Note, I am using Hasura for prototypes and getting into front end (which it has allowed me to do, while at the same time up skilling in postgres and getting my data model right. Am I misunderstanding how Hasura works? For me all it does is take the heavy backend effort away from maintaining a graphql able server. Definitely not going to attempt running it in my usual backend language, Python. However I know I could serve a model in Python and use Hasura actions or Remote schemas to integrate it (that part is not done using graphql)
To summarize what I am trying to say (Use case of chatGPT to phrase things more methodically, haha)... It's worth noting that the criticism you mentioned refers to the idea of using GraphQL as an internal communication protocol between the server and the database, which is not the intended use case for GraphQL. GraphQL is a query language, it can be used for flexible and efficient communication between client and server. Hasura, on the other hand, does not use GraphQL as an internal communication protocol between the server and the database. It uses GraphQL only as a query language for client-server communication. Hasura reads the schema of the database and generates the GraphQL API based on that schema. Once the API is generated, the client can communicate with the Hasura server using GraphQL queries and mutations, and the Hasura server will handle the communication with the database on the backend. In summary, Hasura sits between the client and the database, using GraphQL as the query language for client-server communication, and it does not use GraphQL as an internal communication protocol between the server and the database.
After having watched the whole video I still don't understand what your issue you have with Graphql between Client and DB. I think it would be great to have a video on that.
YES EVENT QUEUES ARE BETTER FOR MICROSERVICE COMMS I SHOULD HAVE MENTIONED THAT I AM SORRY
What event queue would you recommend? We are currently doing a polling REST API for our longer running micro services and it feels "wrong" to continue this pattern while we are growing.
Do you mean event driven architecture?
You're talking about reading data from different microservices so HTTP and gRPC are the correct way to go (if anybody is trying to read data by passing messages on queues then they're beyond help). I would also argue that gRPC isn't worth the complexity overhead if you're not transferring large amounts of data.
Also curious on what event queues you would recommend. Some I'm familiar with are Kafka for streaming data and Rabbit MQ. Totally based on use case. A video breaking down which queues to use when would be awesome.
Haha thanks was thinking that myself. That's what we did at Best Buy.
Hasura is literally a server deployed as a microservice. It IS the server between the database and the client so it falls right in line with your "acceptable use cases for GraphQL" diagram. Memes and hot takes that paint the whole world black and white are great for driving Twitter and UA-cam engagement (here I am, engaging with your content) but at the very least you could try to do an honest review that adheres to your own "rules".
If your take is that systems that have any kind of GraphQL -> SQL translation layer in their stack is evil at least be honest with yourself and viewers and say that instead of this nonsense about requiring a server in the middle.
This is what I was thinking
I agree. I would genuinely like to hear Theo's thoughts on Hasura beyond "No. Bad. Bad."
"Don't use stuff for things it was not designed to do" - wears a baseballcap while sitting at his desk 🤨
For anyone wondering what the actually name for "GraphQL Resolver Gateway" is, its called GraphQL Mesh, and that mixed with GraphQL Hive is magic
As someone using Mesh in a large project, get as far as you can from Mesh 😅
I think it quickly becomes a mess when microservices talk to each other. You could do that with a message queue instead. Microservices can put messages into this queue and read/listen to messages, and suddenly your architecture's dataflow is way more understandable.
If you think about it long and hard, the microservices talking to each other problem is similar to two-way data binding, and the solution I'm proposing is similar to what Redux does to solve the issue in a sense. Unidirectional Data Flows on the server^^
Oh I like that framing! It was a big ah-ha for me when I read the react-query docs (and reinforced by some of Thea’s videos) and shifted my thinking away from querying and to “state sync”. It’s a little more of a stretch to see an event bus that way, but could be very useful!
@@efkastner well yeah it's a stretch :D but if you have a service which is dedicated to reading the queue and mutating the database, then it could actually look like a reducer, that gets the current state from the db, and the "action" is the message.
What you'd get out of this is very easy swapping of microservices, and replayability? Probably there are no race conditions anymore as well while writing the db.
Never tried, though I'm just thinking in theory. (It feels like a bunch of unnecessary complexity, so maybe it doesn't make sense)
Kafka is you friend
the between server communication is likely better dealt with as an event, posted to an event queue / log rather than a realtime connection that could / will fail.
Will repeat myself. Yes, you may use gRPC for microservices, there's no evil in it. Use event-driven architecture if you have very good reasons for it (having proven that synchronous doesn't work in practice *for your specific problem*, rather than just assuming it wouldn't or that it's somehow "not in the spirit of microservices"), if you have a damn good observability infrastructure and a strategy for dealing with transactions. Don't do event-driven because you've read it in a popular microservices book that takes a dogmatic stance on it, ignoring the enormous cost in added complexity and maintainability.
we use Hasura in production its is actually quite amazing... our backend is written in golang using the gqlgen library and we auto generate around 5M lines of compliant golang code because of the graphql schema Hasura provides...
Theo you fail to understand that companies operate in different levels. If my company is worth 1B+ do you really think we would use Hasura? Of course not, its better to build your own infrastructure because you have the cash/manpower to do so. Hasura is fundamentally game-changing for startups unders ~25M and can easily handles thousands of concurrent subscriptions through Hasura Cloud. There is a reason Hasura is worth so much these days, and you fail to understand Hasura for what it truly is. You can easily eject out of Hasura because it does NOT interfere with your postgres database whatsoever
Hasura is great honestly. Has its limitations and kinks but definitely useful and great for getting started quick and reliably.
What even is the argument against Hasura? Or even supabase or whoever put gql straight into postgress? There's so many applications where 95% of the endpoints just pass data straight to/from the db. I genuinely don't understand why Hasura or some BaaS would be bad here
He called out Hasura for using GraphQL as SQL replacement on the backend, but then everything he described as the appropriate use of GraphQL can be done by Hasura. So Hasura is not the problem, only the problem is misuse of tools.
I think that gap that you show between "ServerClient" is a really good way to visualize that use case, and how useful GraphQL will be in comparison to the opposite use case when the server + client are "touching".
I went down the path of using GraphQL a couple of years ago, and I can see it being great when dealing with multiple languages and/or dev teams, or with 3rd party APIs where I'm only the client.
But for my big project where I'm the solo fullstack dev, and using TypeScript on both the nodejs backend + also the frontend... over the last couple of years I've slowly realized that I was just making heaps of extra work for myself dealing with 2 typing systems (TS + GraphQL) instead of just the 1 (TS).
Plus all the extra complexity of having to write queries in another language, and all the extra tooling needed there.
Overall it's slowed the whole project down a lot, without really any benefit for this project in particular. My use case in this instance was pretty much the opposite of when GraphQL shines.
I'm just about to start removing GraphQL from this project. It's my own system (not for a client), and the frontend is still in its infancy... somewhat due to graphql slowing me down.
I'll be taking a look at tRPC, and either using that, or building something customized for myself that has some similarities. Either way, my project isn't going to be any less typesafe than graphql in the end.
I have a feeling GraphQL will be perfect for you 🙏
@@t3dotgg did you read his whole comment?
@@succatash lmao
There’s a lot of tools that fix a lot of your problems. Such as nexus or type graphql to completely remove writing your code in the graphql schema language. Everything is kept in JavaScript with these tools. Which means your typescript is automatically created for you, and is the main thing you’re using
@@succatash prolly meant to type tRPC instead of GraphQL lol
This is a surprisingly good video given that I disagree with the two contentious points you make:
1. GraphQL shouldn’t be used directly in front of DBs.
2. GraphQL shouldn’t be used for inter-service queries.
And here’s why:
1. The case for DB backed GraphQL APIs (e.g. Hasura)
I think you're arguing against the idea of having micro-services use GraphQL to query their own data, and I 100% agree with that. However, the value of this tech when used correctly is as a rapid project accelerator that won't tie you into loads of tech debt down the line.
Creating a schema directly in front of a DB is fine provided that the schema is a completely logical view of your domain, since in that case you’re free to change the implementation details later. Hasura makes this possible to do since you can create mapping rules that allow you to present your DB logically.
Given how fast it is to create very performant services this way, it’s actually a really good option for start-ups and green-fields, and I’d particularly recommend it, since although it’ll allow you to get your tech. stack implemented quickly, the logical schema you (with care) end up, will mean you don’t get stuck with a monolith when it’s time to scale, and you have a really simple path to moving to something more scalable on the back-end as needed.
You even say as much yourself, and I quote:
> This is what I love about GraphQL. None of this back-end implementation detail exists to me anymore.
2. The case for using GraphQL for inter-service queries
If the thing you really love about GraphQL is that none of the implementation detail need concern you as a front-end dev, then isn’t that an equally useful trait for other services on the back-end? Or, stated another way, if the benefit to this formal interface is the ability to change implementation detail provided queries flow through this interface, then isn’t it better that more of the traffic flows through it?
You don't explicitly say why you disliked this pattern, but perhaps your objection is the performance hit from going through some additional resolvers. If so, then it’s worth noting that this doesn’t apply in the case that services are implemented as subgraphs, since in that case you can then either query them directly, or query them via a low latency router that adds no perceivable overhead.
Thoughts?
Thanks for making this video! I've been a graphql dev for 1 year and while I've gotten used to the mechanics of the system, I've never really understood the "why" behind it all. I work at an online retail company and this "middle tier" project was started more than two years ago and changed hands a couple times, so now people like me have no idea what the original point even was. This video helped me form a better idea of what it's for, and what my company is still missing in order to better leverage this technology. I might even be able to impress my manager by formulating a well-defined plan for improvements.
BTW, I am in a scrum group with front end developers and I hate it. I feel like team meetings are a complete waste of my time because they are only talking about React 99% of the time and I work on the Graphql server and resolvers. I'm going to make a new push to break up the stupid "full stack team" pattern we're using. It's stupid, at least for the way this company functions.
In case of distributing requests between microservices, GraphQL can address this on it's own via GraphQL Federation. The same applies for requests initiate from one service to another. Instead of calling the service directly it calls the federation. This removes the requirement for the service to know about each other. The only information that the service needs is where the federation gateway is. This allows you to easily merge and split services without need for application wide refactoring.
Good video, good points. Ever since I learned GraphQL, it made me appreciate the art of crafting API's, and made me like to *start* with GraphQL schema's when designing new object-oriented systems. However, *my* very favorite feature of GraphQL is its introspection capabilities. I do wish the spec would "require" it to always be public, so that clients could reasonably employ tools that could dynamically generate UI's, or even new machine-to-machine processes, based solely on schemas..
It feels good that the same arguments I had some years ago during an internal discussion, that GraphQL is probably best utilized as a gateway to your whole microservice clusterfuck rather than talking to the db's itself. The gateway also prevents the frontend developer to know about the slight differences in naming between the different microservices, as they only need to care about the contract GraphQL schema defines./
I'd argue that the GraphQL client should always be considered as part of the frontend domain as most UIs are data-driven and frontenders knows best what data dependencies each component has. Over time frontenders needs to make changes to the frontend code, and then it'll lead to a much more streamlined and frictionless process if they adjust the graphql schema / resolver at the same time. If the frontenders needs to align and communicate with the backenders every time they need to make a change impacting the graphql sschema, it may risk leading to technical debt instead, as there's a big chance that a workaround will be taken instead if time is an issue. The graphql server as I see it is a BFF (Backend For Frontend), meaning its for moving complexity and business logic from the frontend/client/browser to instead handle and prepare it in the graphql server. Therefore there are very strong arguments for writing the graphql server in node.js (javascript, typescript, rescript, purescript or some other compile-to-js language). Other than that I agree about server to server comm should be gRPC, and that is is only intended as a thin interface/proxy without direct db connectivity... but the split between FE and BE is where I disagree. This is based on my experience from working this way.
I was told GraphQL is used over REST primary because you can tell it to "filter" the response and thus you get exactly what you want. The problem is, you can emulate this with REST too if you really wanted. Either in the query params or inside the body you can tell the server what you want then your REST api will filter it based of your request.
Graphql sounds like another useless tech for most applications
If you design your graphQl schema you can literally fetch all the data in your database or just a single scalar value or anything in between in ONE REQUEST. It took me a while to understand how powerful this ability is but once I saw it, I can't go back to REST! The key is all in the design of you graphQL Schema plus custom DataFetchers to solve N+1 problems!
A nice thing that you brought up here Theo is Data contract which is probably the most common problem that any software solution without proper planning face.
Explicit schemas as contracts is a must.
For the most complex systems, we even have dedicated versioned schema repositories nowadays thank god.
GraphQL was one of the moves towards this in a more narrowed sense than when talking about architectures.
Info near the definition, flexible and extensible metadata, all come very handy.
Also a very bad practice I see a lot is to try to come up from the beginning with an ubiquitous data model that will not be needing updates any sooner (that is an illusion and probably it will become harder to cleanup those unwanted attrs and add useful ones ).
My tip is, unless it is a very fundamental attribute of the particular entity wait until you implement the feature that will need the attribute. I get the creeps when i see those humongous entities that look like they came from an enterprise software of the 90s.
You should look into federated graphql. With that at the gateway each service maintains it's own schema. This provides a consistent interface for inter service requests.
Came here to say this.
The use case he made where the payment service needs to update the user is perfect for a combination of stitching and federation.
You make the payment, that mutation returns the user, and you just spread the user, which, at least in Relay, updates that state.
You didn’t give any real reasons why graphql server is bad. I recently started a project with aws and when looking for resources to connect my dynamodb tables to my front-end, aws-amplify came up among the first solutions with graphql the default. I think I agree it’s probably not the best route, but I like to know why.
Hasura is great for base CRUDs, an events driver to fire off serverless fns, while adding your own apis (more serverless fns) for business logic in highly efficient and no boilerplate ways. Did I mention serverless? And Hasura is the (graphql) gateway.
Theo is right: He doesn't know what he is talking about.
He like most are too stuck on microservices. Go serverless and event driven, then Hasura makes sense.
PS. You also must hate Firebase. Me too, Firebase is a trap. Hasura is conceptually like Firebase as a BEaaS, but better in every way: complexity and performance scaling, transparent and open source, graphql ecosystem
At work our app uses GraphQL stitching to combine Hasura with some custom NestJS APIs. We're in the process of porting to a single tRPC server and it is SHOCKING how much code we're removing. To be fair, a lot of that code is generated. However, the more significant change is to our dev process where everything is 10x easier.
Yeah I fell into the “wow Hasura is amazing” hype… but ended up having the same experience in the end: it just sugar coated the complexity, didn’t really deal with it.
But it’s not bad! Just somehow I failed to find a really good use case for it even though I tried.
"we define a schema and don't have to talk to each other anymore" - remembers me the time we used SOAP
I went through Apollo’s free courses, which are super high quality, and learned GraphQL was not for the project I was working on. I’m really glad I went through the courses still, because now I’ll know when to use it.
With regards to microservices, GraphQL Federation can be used to provide a unified API while keeping services isolated; services define their schemas and ownerships.
As for using GraphQL to communicate _between_ services, you'd be better off with using something like gRPC, Protobuf or other event queue systems alike. GraphQL only make sense when you want to expose APIs to one or more clients.
Between services you own? Sure. If you don't own those services gRPC can be hell to debug
@@KManAbout Absolutely. gRPC is not suitable for public APIs imo.
@@dealloc yup glad we agree. It is super good for private one as you said. I should use it more tbh
With http3 things are going far
GraphQL for server to server is perfectly valid use case. I have used it in production with 7 microservice and it was a pleasure to use. Each service just needs to expose an internal graphql schema at say /internal/graphql that can be consumed by other microservices.
What specifically did you like over Rest? Did you manage to make the gql schema work as a 'contract' between the different teams?
@@82TheKnocKY Yes exactly. This makes everything type safe. You can use graphql-codegen to make sure that all the queries are valid in CI so the contract enforcement can be automated. As far as GRPC is concerned, GraphQL can be just as fast if you use HTTP2 on server side (easy using a service mesh). The only saving using GRPC is that it has a smaller payload because of Protobuf encoding but the difference won't be significant.
Antipattern avoid coupling when you re talking about MS architecture use Event Driven or Event Sourcing instead
@@gt4adolfo Event driven is not always good. For instance, if order service wants to make sure that the customer has a valid email, it should call customer service using HTTP and not Event. Event Driven is good for async communication.
@@quant-daddy is possible to achieve it using event driven... youre coupling microservices... for this reason exists Aggregation events. the same microservice have to check for example the "email". is not necesary coupling.
It would be interesting if you could properly address your beef with hasura. Perhaps a dedicated post explaining exactly why you don't like it? Or even discussion panel? I'll be there to meet your objections in a constructive way
I have been using Wordpress as a headless CMS, and using WPGraphQL gives you real power, and its so superior to the WP Rest API, in a lot of ways. It allows you to just don’t really care about API routes defined by Wordpress, and just fetch your things (with a proper caching solution, which can be a problem using GQL…).
One of the few channels where I don't mind the clickbait titles, the content is always good.
I really like Prisma lately. They seem to fix and work on the various issues.
We use javascript for things it's not meant for. Here's the thing, if the tool works well for your use case and there's no much bad trade-off, then you should ideal it.
Absolutely agree David. I am doing this since some time now: if it works i'll do it. It's not only a question of do I need to use GraphQL or not? The question is: what's the best tool I am confortable with to work with faster !
What's a problem to use gql as api gateway instead of REST(HTTP/JSON) API? For example to use `GraphQL Mesh` instead of Kong/Tyk/Ocelot/KrakenD etc.?
So you have multiple microservices that call the same database? Congrats! you have a distributed monolith) One of the biggest advantages of microservices is the ability to scale them independently. You just multiply it by 0 when you join everything in DB.
With a schema like this you will end up using 600$ RDS DB instance on AWS (without traffic). I know this because I did this)
Care to explain. I like to know more about that
@@zzzz6538 In microservice architecture, every service should own its own data. If that requires a database, then the service should have its own database. This allows the service to define its own data models and scale its infra independently.
Say you have 10 different services using the same DB. Most of the time each service needs 10 units of power (simplification for brevity), but occasionally each service will need up to 30 units of power. You could have 1 DB that provides 300 units of power all the time ( 10 services * 30 max units of power) or 10 DB that provide 30 units of power. 10 small DBs @ 30 units of power will be cheaper than 1 DB @ 300 units of power. Using current EC2 prices, 10 machines with 4 vCPU and 16GiB ram would cost $650/mo. vs. 1 machine with 40 vCPU and 160 GiB ram would cost $850/mo. It's about 25% cheaper splitting to multiple instances.
Far more important than price though is independent data modeling and reducing complexity. If you outgrow a single DB instance and have to shard your data, determining how to shard data across 1000 tables using a single identifier for consistent hashing while maintaining foreign key integrity per sharded instance would be an absolute nightmare. My head is spinning just thinking about how to do it. If, instead, each of your services use their own DB: 1.) you're less likely to reach a scale where sharding is required. 2.) the relationship complexity between the few tables that belong to the service is going to be much easier to manage and determine how data should be split and which identifier is best for hashing while keeping related entities in the same shard.
I don’t often comment on videos but those I do are ones that help guide me in the right direction, light bulb moments if you will.
It’s been a while since I’ve played around with programming and naturally things have changed a bit.
I’ve been researching tools & tooling to possibly build small apps to manage my seemingly unmanageable life.
TLDR: this was insightful
"when graphql happened for the first time ever there was a real line drawn between the backend and the frontend, that i didn't have to worry about the other side". That's not because of graphql, but the lack of basic knowledge about design patterns, SOLID principles, etc.
hey @Theo, heard about federated graphql? Have a look, it does let you have individual services also speak graphql, then use apollo router as your gateway (no code, just deploy), then your individual services can talk to other services through the router as well.
Another thing that I find GraphQL helpful is with defined schema, graphql client (apollo) can automatically merge same objects from different response which help consistency in the UI without manually doing in that in reducer or in react-query callback. How do you normally deal with this when you are using trpc/rest
Instead of just crying... "Dont do this.. dont put it here, it was not built for DB to server comms, ...this is just bad", a more mature aproach would be to state WHY it should not be used for DB to server interactions and to let people come to the conclusion about its usage by themselves. Everything that you said in the context of server & client applies to DB and server as well. There are many shops that have a dedicated DB team making modelling decisions, and providing sqls or SPs for middleware teams.
For the server too, where business logic is to be applied, getting the data in a vendor agnostic way is desirable.. Imagine data stored in postgres, oracle and dgraph. Having a graphql interface between the DBs and the server really decouples the business layer from changes to the underlying model.
Strong agreements and disagreements lol. I think that your rant about why GraphQL works between the server and client is spot on. The part about not using it in the backend could use some more elaboration vs saying "don't do that." The reason why GraphQL has not been good in the DB layer is because it was typically implemented in mainly NoSQL or SQL databases i.e. Fauna, Hashua, etc. And life gets complicated when trying to use GraphQL between collections and tables. The ONE caveat to this IMO is a Graph database. Its literally called Graph Qurey Language lol. It lines up with how GraphDB's are structured way better than any custom GraphDB language. Its the reason why I've been using Dgraph for most of my projects and have been really enjoying it. Now even though I feel Dgraph is the only solution that got it right, it is certainly the exception and not the rule... and given that the dev team for the project has been all over the place, I don't like to say "it's the only solution that works and therefore use it". If anything my conclusion from using it is that GraphQL is not suitable for SQL/NoSQL databases but if GraphDBs ditched their own custom query language for GraphQL, the world would be a better place. Even EdgeDB (while still a Graph) is implemented on top of PostgreSQL which is probably why maintaining their GraphQL adoption has been difficult. A more accurate statement would've been "For the love of Christ, it's not called Collections & Tables Query Language" lol. Also as a side note, the other reason Dgraph works is because its designed to replace most of the server layer while also providing the capability to lambda call any server you want to setup. So in your diagram the DB would be the middle layer and it can still communicate easily with microservices. This approach has been great for building new applications and significantly reduces the code needed in my microservices. If this pattern was adopted by other DBs, the world would be a better place.
Totally agree with this and I commented something similar. Neo4j also has a library that generates GraphQL from your graph schema that essentially just writes cipherQL under the hood. This saves you from having to deal with cipherQL and instead you can use GraphQL for CRUD operations on your database.
Small nit: graphDBs are a type of NoSQL DB, but I know you meant things like mongoDB.
@@prestonrasmussen1758 right exactly, and I meant NoSQL with collections.
"for the love of Christ stop using graphql for your databases" daam bro 😂😂
Theo, that was exactly the question I asked myself yesterday.
With time and missions done, I realize that not so many people are understanding GraphQL. Do you think it's linked to its "not well explained" documentation?
I still don't understand how gql is better than trpc for communication (assuming you use TS for your server ofc)... You could define typescript types for your BE-FE contract just as easily as you define a gql schema....only you don't need to use a different syntax for it....
Not always true about databases, there’s graph database called dgraph, and it looks really nice there
GraphQL can actually be a good solution of querying graphDBs. Neo4j with AuraDB has a library that autogenerates GraphQL from your graph schema your use for CRUD operations to your database (as a functional alternative to cypherQL)
If your gql is being "auto-generated", you're using gql wrong.
@@t3dotgg I think that opinion comes from your focus on SQL databases. Plenty of graphDBs, like Neo4j, support the use of GraphQL for CRUD operations. Neo4j has a library where it auto-generates GraphQL from your schema. Under the hood this GraohQL calls CipherQL to write to the database, but this allows the user to use GraphQL’s often better syntax over CipherQL depending on the use case. Neo4j has an article about the library on their website.
I don't understand why GraphQL is bad on communicating backend and db (for example Hasura). It is very good to throw away writing a lot of code that is just getting some data from db. It is easy to make some fragments and auto-generate all other for project needs.
The most disappointing is that I do not hear any reasons to not use GraphQL between servers and db. I agree, that at first GraphQL is built for using it between client and server and it is so good for it. But it doesn't mean, that it isn't for server side needs. I think with some work it can be one of the best solution for db and servers communicating. Maybe not in all cases, but in many cases.
My company attempted to adopt GQL for the exact thing you say not to adopt it for in this video ("hey I read somewhere that GQL is good for handling complex datasets or whatever, let's use it") for 5 disastrous months in 2021 before going back to REST.
Guys, he is right... GraphQL is NOT an API/DB gobetween. Trust me on this.
Isn’t Hasura just auto-generating CRUD GraphQL (and recently REST) API, instead of querying DB’s using GraphQL?
Also, can you share your thoughts on public GraphQL APIs?
I don't really agree with theo's position on this either, hasura has been all about having a low to no code backend in minutes for me. It solves the boilerplate nonsense of adding CRUD to your backend, and you can use actions for complicated logic. It's unbeatable when it comes to building something quickly
You just define queries in your frontend, run graphql-codegen and use apollo-hooks, there, you're done
One of the argument I've seen is that you should never expose your database schema "directly" to your client, without much explanation as to why. Well first of all you can use SQL views and functions and build your hasura graphql schema around that, and second: why not ? permissions in hasura are made to make sure your users see only what they're allowed to ...
Anyway, hasura rocks, it will make your app building so much faster and it is ridiculous to dismiss it over some ideology, we've used and hacked technologies to use them where they were not supposed to for ever, won't stop now
Yeah, I'm not sure why he is hating on Hasura. It's an incredible tool for generating an api.
Look kids. There is no reason you can't use GraphQL directly on top of a DB, but if you team is bigger than a handful of people you'll have a hard time not regressing something somewhere. Basically Theo is saying separation of concerns is good. And basically necessary as your team scales. There is not technical limitation here.
This confirms my thought. I currently can't see any value in Graphql if all it does is describe a schema.
Theo can you explain why having graphql directly on a database is bad?
Isn’t it just like a serverless implementation of graphql? I mean ultimately with hasura o assume there’s still a server in between the database and the client, or do I assume wrong?
Graphql was created because Swagger/Openapi syntax sucks. Change my mind.
This video really helped me understand the theoretical perspective of GraphQL. I was indeed viewing the whole concept like Hasura has designed..
Instructions unclear. Now using GraphQL for everything except client server state transfer
All these things kinda sucks! Today no one has done the right solution for client-server communication that's frustrating . I will do my best to fix this
Awesome that you spent the time explaining this. I've seen crazy usages of GraphQL, even all the information on the client managing a client side database 🥺
I wondered about the title of this video, so I went to google trends and guess what? For the last 5 years, searchs related to GraphQL are going well, especially on 2022. People is indeed very interested on GraphQL. So there you have it.
I like GraphQL but at the same time I don't like GraphQL.
does traditional REST APIs make sense in your experience?
What sort of a question is this? REST APIs is a staple and it won't go away ... ever. Not until we remove HTTP. They are simple to consume and fast to build. IMHO most new projects should start with a REST API and if they have some scaling issues then they should consider GraphQL, tRPC. REST is the perfect fit for small teams. Just don't overengineer it with dumb HTTP semantics. I've seen overengineered REST APIs and they are not fun to work with. Also keep in mind that REST doesn't mean JSON. You can use REST APIs and pass XMLs to keep extra object information. The downside of JSON is that it loses a lot of context. Granted most of the time you don't need the said context. But it just goes without saying.
@@xtremescript Well, that was kind of my point, why isn't REST mentioned in the video?
@@xtremescript rest is more suited for scalability than even gql imo
I was wondering why is what hasura or platformatic do so wrong, yes ofc depends on the scope of the project, some are out of scope of what those frameworks can do, but for simple stuff why not?
I was thinking about the graphql between microservices - is it good idea to use GraphQL API for between microservices just to have a single API implementation over all infrastructure ? Then you can expose all these graph ql apis using the apollo federation/proxy service. The key thing to mention is, that microservices themselves will not communicate directly between each other but instead using kafka topics. What do you think about that Theo ?
A minute and thirty in I had to explain to my OH why I yelled 'fuck you Theo'. In all seriousness, this is great content. The misuse of GraphQL is a mess I've inherited from the version of me from last year who didn't really know what he was installing, and I wake up everyday thinking about pulling the trigger on a fat rewrite
Okay, you are hands down the best tech UA-camr.
So wait, i am conflicted now, am i an idiot for using graphql for storing data i need from a graphql api into my database, and then also for fetching the data i need for my frontend ?
I am very confused by this video
Short review: Theo kills Hasura in 1 video
As soon as you said "do not use graphql for you database", the first thing I did was check to see if thats exactly what I did😅
If you speak in absolutes, you better have a good reason for your opinions. No reason is given for the opinions expressed in the video.
For a lot of smaller applications having a built in GQL interface to a database can make a lot of sense, because you don't need to always hand code resolvers.
You can use GQL wherever you want, including server to server communication, it's no worse than REST and you might not feel like dicking around with Protobufs and type definitions.
Very good explanation on how to use GraphQL. I have a question, what if we use GraphQL between each service and the Gateway? So, each Microservice exposes its own schema and the Gateway is in charge of resolving it according to the client's requests, in other words what I mean is, it is a bad idea to have graphql between the client and the server, but also to have it between each microservice and the gateway. Thanks
I agree 100%. Have you used federation? (E.g. Apollo's solution for joining multiple graphql endpoints into one) would be curious to see your take on that pattern :-)
Yeah the "GraphQL Resolver Gateway" should surely be "GraphQL Federation Server" and then GraphQL totally does handle the call end to end back to each micro-service. You also get the added benefit of writing queries that can cross over micro-service boundaries which all get resolved seamlessly.
But why?
GraphQL makes trade-offs, and personally I find the trade-offs worth when communicating between client/server, not so much with server/server, cause now each microservice needs to also be written with GQL in mind.
@@kefkef5 good question. The reason is: say I want a "global" profile service for my users. But, depending on the department / apps they've interacted with in our ecosystem, the metadata is subject to change from a wholistic standpoint. So, federation would help us build a collective graph for the profile of any given user.
@@kefkef5 There are some situations where microservice1 { field: microservice2 { data } } is insanely powerful, and I'm not aware of too many alternatives. The caller doesn't need to know or care where any data is and the GraphQL Federation server glues everything together seamlessly with no significant performance penalty that my APM can detect.
Can't you do that anyway when you have your GQL service calling the rest services to ask for the data and glue it together on the GraphQL?
Now the backend is also forced to provide you a GraphQL endpoint, which they might think it's not worth the time investment compared to exposing the data through rest.
The video is informative, but I sincerely hope that viewers don't adopt this particular approach for designing microservices architecture. My expectation is that the showcased method is solely for educational purposes.
When it comes to communication between servers or microservices, it is essential to employ gRPC( Efficiency, Language Agnostic, Streaming, Contract-First Design ) for tasks requiring an immediate response, following the Function as a Service (FaaS) paradigm. On the other hand for scenarios where an immediate response is not necessary or possible, communication should be facilitated through an orchestrator or a stream-processing pipeline.
Hi Theo, I've only worked on the front end so far and now I just begun learning React for almost 1 week now learning useState and useEffect at the moment. What are the next things would you recommend I should focus on after React? I think Redux would be the next step? Maybe GraphQL after Redux? There is lots to learn and work on. At the moment I am focused on trying to create a scheduler App for myself. Kind of like a journal mixed with dates and goals you could set for yourself, not a to-do App. But my end goal is to land a job as a full stack developer with JavaScript as the client side and Java as the backend. I would appreciate your honest and brutal opinion, thank you Theo.
Have you watched my "your goals kinda suck" video yet? Think that will be helpful.
Also don't learn graphql or redux if you don't immediately need them, both are SUPER heavy to take on w/ a solo project. Best learned in a big codebase that you don't run lol
@@t3dotgg Thank you Theo, I will look for the video you recommended, and I appreciate the guidance.
Could you please explain why querying DB from GQL server is bad idea, and what's wrong with Hasura?
I'm using graphql for a new SAAS project and we designed it as a supergraph/gateway for all rest microservices so client all access the same graphql api and the rest API remain private into the cluster. sure you won't add logic on the database nor creating any kind of monolithic api for transforming to sql
So the main thing that GraphQL solves, being the hard separation between FE and BE, is based on a middle contract layer. Doesn't Swagger/Openapi do this as well?
OMG!!! Who the f*ck uses graphql between server and database??? You left me speechless.
Apollo GraphQL Federation solves the problem he describes at 5:20
In the quest to make everything more complicated, graphql is great at that. How can we make it more complicated? One way of solving inter team workflow is to replace 2 teams with 1 team. Ie have devs which do both front and backend.
Learnt more from this guy in 5 videos than the entire last 5 years of working at my jobs. Excellent content man - please - keep it coming. Very grateful to have somebody with a good head on their shoulders who understands back-end as well talking React/TS. Very cool.
Thank you Theo, reading comments I realize i'm not alone, I've worked with it before but never really understood it.
I finally get why and where GraphQL is valuable.
Did I miss something or was standard REST not even mentioned? Isn't that also a way to separate Front and Back people?
Graphql uses schemas, which defines a standardized contract between front-end en back-end. In REST you can also create schemas but you have to implement your own or use tools like OpenAPI. Another benefit of graphql is reducing data overfetching. In REST you might need to do several request to get data you need but in Graphql you just define the fields you need and make a single request
Best explanation about GraphQL
What are your thoughts on Postgraphile?
Api contracts in form of soap or grpc or res api provide a way to abstract the complexity of be from Fe and vice versa.
Its not just graphql.
If you cannot communicate between BE and FE, its people problem and not a tech problem.
Work on each of above tech.
I disagree that GraphQL should not be used as an interface for a database. Especially when there is zero to none business logic and your application is basically glorified table editor direct access can make it way simpler. With proper precautions such as object type policies, complexity calculators, query timeouts etc... you can mitigate all issues that might arise.
I'd argue that you in fact should delegate as much GraphQL schema as possible to the database as then you can reap benefits such as projections and single query (one execution plan).
Great video. I just signed on to use a data company who uses GraphQL to access their data. I have many years of SQL and so far I am not a fan of GraphQL. No default aggregate / Count?? Because it is like you said for the UI and not the database. I need to ask this company on why they are using GraphQL as their interface to their database.
Maybe you are not actually hitting their database directly, but rather are making a bulk entity request to one or more services? They may be considering all client requests equal, regardless of whether they are coming from an individual-entity record-viewing UI, or from a batch reporting job. But yeah, if you're trying to aggregate normalized, relational data, SQL would be better. But that doesn't mean they will be ok with giving access to query their database directly.
RDBMS is a server, and User Service is a DB client application, so why, you say, shouldn't GQL be put between a client and a server?
Isn't GraphQL also great for situations where you have many different consumers of your API? It's hard to predict every need of every possible client (website front-ends, mobile apps, games, smart watches, refrigerator screens etc.) so GraphQL makes sense, since the clients can define their own data needs in a query. This cuts down on superfluous requests and unnecessary data being transferred.
I just wish they have the "select *" operator. Having to write out / link the primitives each time is silly.
You can use GraphQL snippets depending on your use case
Please have a look on hasura for graphql
Hasura is an anti-pattern lol
The critique that Hasura is an anti-pattern is based on the idea that you don't want to expose your db schema to public clients. It's valid. However Hasura has features to limit exposure of fields and modify permissions. The cost/benefit for small time devs is huge with Hasura. If you are in a highly-resourced company you can use Hasura and then put another GraphQL Yoga engine in front of you want another layer of abstraction and employ more devs.
First time i hear that people misuse GraphQL as a sort of new SQL. Eww. But i can see where the initial confusion could have come from. If you treat a backend service as just a Persistence thing that just happens to have a web interface you might get a wrong idea to use it there.
For the love of God, yes there are scenarios where people really have some ideas that graphql is somewhat a blackbox on top of your api which magically spits out the data that you need on the frontend. But if we ignore alot of bshit that is written on the internet and turn on our brains and read some documentation we come acrosd that gql is actually as a technology ment to be used to fetch only the data that the frontend needs eg. prevent fetching unnecessary data. 2nd its tech agnostic meaning that you could use gql with any framework or library thats out there react, angular for FE and node.js, java etc for the backend.
3rd thing to note is that devs generally have no clue what gql is about so when they see the promise of drilling the properties down in gql they think oh mann so sweeet we can now has our gql queries translated to sql and skip out all of our orm/sql/nosql thingys and just use gql ( which is common nonsense).
4th, there isnt yet a technology where you write 2 lines of code and it makes you a app which you need.
GraphQL was never really alive to begin with.
Why can't you use a rest API to do this though? Can't you replace the graphQL server in this architecture with a rest API to do the same thing?
I don't understand the reference to Hasura, when it is interpreting postgres schema, plus reading some other metadata to generate an API (runs on a server). This API is communicated with via graphql to the client.
I hardly expose any of the underlying database via Hasura, and mostly use more advanced postgres views and Hasura actions to define my full set of backend services, which then is all stitched together so my client can use it.
Note, I am using Hasura for prototypes and getting into front end (which it has allowed me to do, while at the same time up skilling in postgres and getting my data model right.
Am I misunderstanding how Hasura works? For me all it does is take the heavy backend effort away from maintaining a graphql able server. Definitely not going to attempt running it in my usual backend language, Python. However I know I could serve a model in Python and use Hasura actions or Remote schemas to integrate it (that part is not done using graphql)
To summarize what I am trying to say (Use case of chatGPT to phrase things more methodically, haha)...
It's worth noting that the criticism you mentioned refers to the idea of using GraphQL as an internal communication protocol between the server and the database, which is not the intended use case for GraphQL. GraphQL is a query language, it can be used for flexible and efficient communication between client and server. Hasura, on the other hand, does not use GraphQL as an internal communication protocol between the server and the database. It uses GraphQL only as a query language for client-server communication.
Hasura reads the schema of the database and generates the GraphQL API based on that schema. Once the API is generated, the client can communicate with the Hasura server using GraphQL queries and mutations, and the Hasura server will handle the communication with the database on the backend.
In summary, Hasura sits between the client and the database, using GraphQL as the query language for client-server communication, and it does not use GraphQL as an internal communication protocol between the server and the database.
GatsbyJS approves this message 👍🏼
HAHAHAHAHAHAHAHAHAHAHAHAHA
Dude this video is gold! And you are a star!
Thank for creating content ❤
I fail to see why it is "bad" at server to server communication.
After having watched the whole video I still don't understand what your issue you have with Graphql between Client and DB. I think it would be great to have a video on that.
Oh God people are doing client->db??
Why didn't I find this channel earlier... Just why..
Thx.... your videos are helpful as always, keep up the good work
Did you know the Mock Service Worker (npm: msw) , what dou you think about msw.
“I’m looking at you hasura”😂