Migrating Netflix to GraphQL safely
netflixtechblog.com
netflixtechblog.com
In my experience, it's a nightmare to 1/ secure 2/ version 3/ ensure qos
Securing it properly should make it a no-go in like 95% of cases. you get amazing flexibility on the front end with a heavy heavy cost on the backend. Also, in general being able to say "i want to do whatever" and request everything at once is an anti-pattern IMHO, especially after HTTP2 became mainstream and doing multiple requests is reasonably fast.
Versioning? Forget about it. Now you have "all the versions" and good luck figuring out what is used, what is not used or god forbid deprecate something. you're going to have a bad time.
As far as qos goes, if any client can range from "i'm asking a simple thing" to "i would like the whole world please - and btw you're only going to figure things out as you pull them in" qos becomes a pipe dream.
Last, I don't understand what Netflix does that is so complex that would warrant something like gql. I just don't. To the naive developer in me it seems that they 1) need to have a basic api to get whatever catalog they have + a few apps build on top of that 2) have figured out how to do streaming exceptionally well in order to scale to all the people watching. That's is and although a gross oversimplification I cannot think about a scenario where gql is needed.
I think in their desire to reinvent the wheel Netflix has jumped the shark. A lot of really good ideas have come out of the innovation they did in the past but lately it seems like they are doing things for the sake of doing them or they are pissing people off with password household policies and whatnot.
Seen that in some productive APIs, always fun.
When work or personal issues overwhelm me I go with this kind of meta analysis and stop usually at Keynes quote : In the long run we are all dead. This helps me get though the day.
It allows a full separation of the frontend dev velocity from the backend velocity.
The frontend is no longer dependant on backend engineers for API creation, nor negotiations on how those should look.
The backend can choose to observe the hot path of GraphQL and optimise those for your #3, but #1 and #2 don't differ from, say, REST.
I’m not a big fan of GraphQL, but I it’s not particularly difficult to secure.
how would you do this with REST? or any other kind of API? graphql, at it's core, is a tool for the frontend to describe to the backend what data it wants. how the backend replies to that is completely up to the backend.
> formulate complex queries where they ask for the kitchen sink
most concrete implementations i've seen include a way to limit complexity. in `graphql-ruby` for example, you can limit how many nodes, and you can apply a "cost" to a particularly expensive-to-calculate node if you like. Say you give an API only 10 "complexity points". You could make most nodes cost 1 point, and expensive nodes cost 3 points. Now, someone can _request_ the kitchen sink, but you can respond with a "no".
GQL itself doesn’t attempt to solve authZ or authN by design.
Edit: if you want to do this with nginx or within graphql / express, this is what that could look like https://www.apollographql.com/blog/backend/auth/a-guide-to-a...
Git Monorepo vs Polyrepo Microservices vs Monolith Kubernetes
Yes, of course! All problems are organizational problems at scale. I have no great love for gql, but painting it as a black and white decision pitting FE velocity against all other engineering concerns (including hyperbole like "sanity") is childish. There are legit tradeoffs discussions to be had, but if you think it's impossible to secure or scale gql then you're not a very imaginative or resourceful engineer.
> "Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure." - Melvin E. Conway
It's unavoidable. You can bury your head in the sand about it and demand that the technical structure only match your technical requirement, but then the organizational structure of the company will grind to a hault. People will always be people. Just because you give some of them a title like "Engineer" doesn't mean that we won't conflict on people problems.
Entire concepts like event driven architectures or microservices, while might have plethora of technical pros/cons are mainly pushed by management for their non-technical aspects. One of their main non-technical pro is organizational. If your component just emits events, handles events, and is implemented as a separate microservice then you can always just insert a team, or move a team in an organizational structure while minimizing the overall impact to a project.
So long as the backend engineers at least anticipated all the relationships, properties and objects that the frontend might ever need?
I work on a GraphQL backend and I can tell you, every time the UI needs some new fields we don't have in the GraphQL schema, I've gotta go and implement that just like in REST, the fields don't magically materialize into existence.
To your points:
1. I find GraphQL to be fantastic for security, primarily because all calls are strongly typed, so you can ensure by the time a request gets to your resolver it is guaranteed to match the defined input types. Furthermore, I always make heavy use of custom scalar types instead of String, which allows me to get rid of a whole host of potential security issues by limiting potential injection attacks before my code even sees it. As far as ensuring object ownership/permissions, this is trivially easy, or in any case just as easy as with REST.
2. Versioning is one of GraphQL's greatest strengths. The tooling makes it easy to deprecate some fields while adding replacement ones in a way that guarantees backwards compatibility. Far from "good luck figuring out what is used", the tooling e.g. in Apollo makes it very easy to see which clients are making use of which fields. Furthermore, if you do need to "hard version" a new endpoint, you can do it similarly to REST by including a version parameter in your URL (e.g. /v1/myGraphQLEndpoint).
3. Regarding qos, this again just gets to my original point about having well defined queries and mutations. Your queries don't need to say "allow me to see the world" any more than any other API framework. It's not hard to just have queries that only say "give me X by ID" and also a single bulk retrieval endpoint where you define the boundaries of what you want to return.
If anything, I think perhaps a lot of misconceptions around the problems of GraphQL have to do with these tools that basically say "expose my whole DB with GraphQL", which in my experience usually leads to tears in the end.
But starting with a thoughtful definition of your API surface area with GraphQL's type definition language can make it a joy to use and provides lots of benefits over a straight RESTful interface.
I wouldn't say that. It's simple and straightforward to get around some pitfalls of GraphQL, if you treat it as an enumerated set of RPC-style endpoints.
The commenter I was replying to stated "Thr fact that you're arguing that it should not be treated as a generic query language when it's in the name and this is how it's sold is hilaruous". Perhaps so. By far the biggest mistake I think the original designers of GraphQL made was putting "QL" in the name, as many people think it's in the same category of tools as SQL (I heard someone ask once on HN "Can you do joins in GraphQL?", which isn't even a question that makes sense). It should be treated as a competitor to OpenAPI, and there are reasons why I would prefer using GraphQL in many scenarios.
I think I'll probably write some longer form blog posts at some point about how I think GraphQL is easy to set up and get running quickly, and how to avoid some common misconceptions.
It isn't
> if you treat it as an enumerated set of RPC-style endpoints.
It's not that.
> I think the original designers of GraphQL made was putting "QL" in the name, as many people think it's in the same category of tools as SQL
"GraphQL is a query language for APIs" is literally the tagline at https://graphql.org/
And the frankly insane complexity of GraphQL comes from the fact that you now have to implement that query language, often over completely different sources of data.
Which leads to frankly inane solutions like the requirement to inspect the full request and the full response on the fly just to be able to cache responses.
No. The biggest mistake why not emphasising to people the usecase they solved which was having many many different front end consumers and having to create many permutations of the same endpoints to cover web, mobile, various applications in different states, time to deliver endpoints to the teams. Etc
Most people are not working on the same number of implementations that Facebook or similar need. But they want to be on the bandwagon.
REST arguably has better support for catching for read queries.
I've yet to see one good reason to use GQL for anything but public facing APIs, like GitHub API.
But isn't this "infinite object graph" like the only real valuable part of GraphQL? People get caught up on that because all the other things you mention aren't really an issue with normal REST API.
If you're saying you use GraphQL but don't use the key defining feature of GraphQL. Then what's the point of using GraphQL if you're just using a strict predefined set of things you can query/mutate?
Once the app is developed I think you might put up reasonable general limits to what queries are responded to in a general manner over the whole api.
Not to say either way around is inherently better, I just see it as a switch from backend-driven to frontend-driven.
(Personally I'd love to use something like OpenAPI, with tooling more mature than probably exists, to be properly schema-driven, with both backend and frontend stubs/clients derived from that.)
Yes. Where it can be logged, inspected and dealt with wholesale.
It's moveing a problem from intra-team verbal communication into software where it can be dealt with more efficiently with software tools and methods.
> Not to say either way around is inherently better, I just see it as a switch from backend-driven to frontend-driven.
Yes. I think that's the whole point and main benefit of GraphQL which makes a lot of sense since apps that evolve through their development usually have most changes UI driven.
Just as it could be in the backend? I don't follow that point.
> apps that evolve through their development usually have most changes UI driven.
That's a fair point certainly, but isn't completely generalisable - if you offer a third-party API at all (or think you might) then maybe don't special-case your own frontend.
GraphQL server is the part of the backend.
The problem is that client has a new requirement, it is verbally communicated through UI change request. Frontend team implements the change then has to make a verbal request to the backend team for new data to be exposed. There's a bit of back and forth, testing. Everybody waits for everybody. It all takes time and is most of the time brainless busywork. GraphQL makes it so that request to the backend team doesn't need to be verbal (jira is same as verbal for me). It can be mediated through software and even fulfilled automatically if the GraphQL server is open enough during development. The cost of that streamlining is that at some point security or performance issues might arise, but those then can be dealt with solely by the backend and db team, as they should be because frontend is always insecure.
> if you offer a third-party API at all
For one such app 99 have both server and the client developed in sync.
And even then with offering third party api I think it's way easier to fall into the trap of special-casing your frontend with poorly designed REST than with GraphQL, althought there might be performance and security challenges as there always are with anything public. So there might be a bit to learn about how to deal with that in GraphQL ecosystem.
You pretend that GraphQL is magic.
If there's requirement not exposed by the GraphQL server, the way to add it is literally exactly the same as for any other backend API.
This suggests that GraphQL adoption is a sign of impending business stagnation. This does in fact align with my experience around GraphQL.
On braindead busywork, yes. Please automate everything about it as soon as possible, including communication. I don't want to collaborate on it with a human the same way I don't want to collaborate with a human on my McDonald's order and I prefer to access their ordering software through a kiosk or a phone.
> This suggests that GraphQL adoption is a sign of impending business stagnation. This does in fact align with my experience around GraphQL.
That very well might be. But stagnation understood as predictably churning an app after an app.
Sometimes that solution is to use GraphQL to make it easier to manage client implementation fan-out and requirement diversity. But GraphQL is never a substitute for communication and collaboration.
Lucky you. In corporate environments I worked at it took easily half of the time.
It means that those fields are already exposed to you by backend. Had they not been, no amount of magic on your side would make them appear.
Behind the elegance of a query language that lets you get your perfect little payload, there is a complex server that could need to resolve and block on requests to multiple backends, touch different caches and databases, and do joins and sorts and projections. Front end devs will add the videoRating field and have no idea that this comes from a separate API and causes the response time to double.
Developing some gragantuan do-it-all meta API as a sandbox for frontend engineers is a terrible idea. You don't just slap a graph over your APIs and call it done. The GraphQL server is orders of magnitude more complex than a purpose-built API.
Ultimately once the FE devs have crafted their query to satisfaction it will hardly ever change. It's better to work together and decide whats needed and how to best get that data to your app.
I really favor collaboration on purpose built APIs than gargantuan do-it-all systems.
this makes the fallacy that it is FE devs driving change. it is not. it is product management. and to them (and indirectly to you), making it easy to make performant changes without 3 committee meetings is a Good Thing
Put it this way: you _could_ build a GraphQL resolver by means of mirroring your database schema or service structure, and let frontend engineers play in that sandbox while patching up holes. This would have the exact same result as building a REST API service that proxies requests to a SOA without understanding the performance or stability impact. Or the exact same result as exposing multiple REST services that must all be hit to render a single screen on the frontend, except you push the blocking to clients, which is not a proper solution if you care about the energy usage you don't pay for.
In all of these cases, the common thread is that the backend and frontend teams aren't working together and thinking of the entire product as a system instead of a frontend/backend split. GraphQL forces you to reason about the entire system and not a series of API calls, which can obviously cause problems with siloed teams.
This means I disagree with folks who attribute the power of GraphQL to "frontend-driven" APIs. Your frontend and backend is your product, so ultimately you must reckon with the boundary layer between the two.
Of course some people are better at one or the other so for very tricky things we might have them take over those pieces. But 99% of coding is fairly straightforward.
That's absolutely wonderful problem for the backend and db teams to detect and solve. Cache it, use another data structure, pre-calculate, build cunning indexes, if you can't do anything just throw hands in the air and say it cannot be done. Neither frontend team nor the client does care about backend woes. And frontend team doesn't want to be kept in the game of telephone as backend negotiates solutions for performance with the client.
That's what GraphQL does. Provides better separation of responsibilities between the teams and lowers the amount of conflicts and delays because of the reduction of verbal communication necessary.
> Developing some gragantuan do-it-all meta API as a sandbox for frontend engineers is a terrible idea.
It's not a sandbox for developers. It's for the clients. I know it's painful for the backend developers that front end doesn't want to be involved in challenges to the backend caused by customer's requests and how often they change. GraphQL makes those problems owned by backend devs and noone else.
> I really favor collaboration on purpose built APIs than gargantuan do-it-all systems.
Same as with the software itself, well separated parts, clear non-overlapping responsibilities, small, well defind interfaces between the parts result in better outcomes.
So which is it?
The front end arguement makes sense for someone like Facebook who has many different front ends and integrations. It’s unmaintainable to try do this any other way.
Can you explain this more? How can you avoid infinite graphs? If I have User {things: Thing[]} and Thing {owner: User}, you have to deal with this issue.
In GraphQL the query for the example you gave could look like this
```gql
query {
user {
things {
owner {
id
}
}
}
}
```
Now if your developers are intentionally creating infinitely deep queries, then you'd solve this the same way you'd solve an infinite loop in your code today. An issue you would catch during development or CI/CD. This can be easy to catch using a linter or during review.
```gql
query {
user {
things {
owner {
things {
owner {
things { } # etc
}
}
}
}
}
}
```
In addition you could introduce CI tools to enforce your devs stop writing such complex queries. Also see the @skip and @include directives that can further be used to control what data is queried. In practice, however, this isn't something that comes up too much. In cases where I have seen this happen, it's usually because a developer is trying to reuse fragments without considering what data they are querying, and whether they should be reusing those fragments.
In practice this is an issue for any publicly exposed GraphQL API.
That's why the only solutions to the issue are to:
- either deserialize and deep inspect the query to try and figure out if it's too complex
- or to only expose persisted queries, turning it into a bad replica of JSON RPC.
According to GraphQL, authorization belongs in the business logic layer[1]. A lot of projects ignore this advice and put it into the resolvers. This decentralized approach can be difficult to maintain and keep correct.
But I'll add one more: "Standardisation". Every RESTful API uses different URL paths, naming schemes, HTTP verbs, data structures, headers. Whereas those are defined by the GraphQL spec.
When consuming or creating a GraphQL API, those (usually meaningless) decisions are defined by the spec, so I only need to learn how the data is structured. Since it's a spec, my queries can be strongly typed in both frontend and backend.
GraphQL does add some risks IMHO, e.g. client can send a really complex query (even a recursive one, e.g. get accounts for user, get user for each account). We cannot say this will not happen as we define the queries ourselves, since sometimes the API is exposed to clients which are not in our control. Queries can be sent which can almost bring a server down, so you need to handle that.
Also error reporting / handling is trickier as compared to REST (e.g. https://the-guild.dev/blog/graphql-error-handling-with-fp).
I assume what they meant (and what I mean when I talk about how dire security is on GraphQL) is visibility first and foremost. Which users have access to which data. Things like multi-tenant or even just “you have access to X-model or this specific field on this model”.
Couple that with inconsistent performance (at least for RDMS) and inability to make use of state stores in a sane way (IMHO) on the frontend makes it a non-starter.
I'm surprised how so many HN posts about graphql just slander it. If you go to the website and look at the use case of why it was made, it makes it pretty clear when it could be a good to use. My experience with it has been very straight forward and a joy
You generally don't want to use HTTP error codes with field level errors in GQL.
And bo one is stopping you from returning a response that contains error details.
For example, the Pothos library implements this pattern automatically:
type Error {
message: String!
}
type Query {
hello(name: String!): QueryHelloResult
}
union QueryHelloResult = Error | QueryHelloSuccess
type QueryHelloSuccess {
data: String!
}
query {
hello(name: "World") {
__typename
... on Error {
message
}
... on QueryHelloSuccess {
data
}
}
}
If someone has solved the asshole corporate middle box problem using GQL, that’s really good news.
You can in fact just curl a GraphQL API. In simple projects where I don't need caching I just use the REST library and make a simple HTTP request with a graphql query string. I get back a JSON object to consume.
You can of course go other routes, we took a serius look at tRPC, but it seemed like it would end up being a lot of work. Actually quite similar to how doing the things you want from a frontend perspective is a lot of work if your backend is "just" rest.
> Last, I don't understand what Netflix does that is so complex that would warrant something like gql. I just don't.
If you dig into the other blog points that the OP link to you can see that it's because they needed better performance, more security and better control of the flow. I think you may underestimate how much you get for "free" when using graphQL. It has so much adoption, that there are so many tools that you won't have to build from scracth, even if you're Netflix. You can see our Odata library as an example of this, there are client libraries that we could've used, but they're bad. So we had to build our own. With GraphQL we wouldn't need to do that.
Here:
https://hackerone.com/hacktivity?querystring=graphql&filter=...
On the other hand, with constraint resources, you likely end up with better performance when using graphql because of the reduction of requests and decreased latency.
In most cases, graphql hits the sweet spot.
How?
GQL still depends on an API or a database, isn’t it? And depending on where that Api or DB is located in relation to the calling user/app latency is still going to be the same? What am I missing here?
If you are implying that GQL somehow intelligently caches data, then all the issues with cache invalidation etc still would apply, correct?
The problem with some of these debates about GQL vs Rest us that you have people either saying GQL is great or bad and same with REST. I wish there is someone that breaks it down and truly explains the pros and cons of each.
Obviously that's not exactly how it works, but if you give your frontend engineers the full query abilities of GQL and your backend the ability to monitor and then fix bottlenecks through the GQL load tools you simply remove a lot of the conflict zones in modern development.
You can do everything you can with GQL with Rest, but you're not going to want to do it. And speaking from experience, you're not going to want to do it with Odata either unless Microsoft magically improves the experience after 10 years of it sucking.
e.g. frontend says "backend, give me the users". Then later, for every user it says "backend, given me the users favourite images". However, with graphql the frontend can say "backend, give me the users and include each favourites images".
Even if the way to retrieve it from the database is the same for the backend, there is still reduced latency because it is only
frontend->backend->db (users)->backend->db (images)->backend->frontend
instead of
frontend->backend->db (users)->backend->frontend->backend->db (images)->backend->frontend
Never that, but maybe inexperienced? Nothing you say is wrong, but you need to look at it from the organizational point of view. The way we build (most, all of them where I work, but I'm sure people can point to places) "enterprise" apps today, is that we have a frontend that's build in Typescript, and then something behind it to move the data.
Your frontend engineers are going to want a query language that allows them to pick and chose the data they need. You can do that with rest, but you really, really, don't want to build it yourself. Some frameworks have options for parts of it, but in general you're out of luck when it comes to things like inheritance trees on objects. So you have Odata, RPC, GraphQL, and while Odata is nice on the client side, it's not nice on the server side. RPC's work great, but they are also a lot of work, and then you have GraphQL. What GraphQL allows you to do is the combination of having an engine that makes your frontend engineers happy, while allowing your backend engineers a lot of visibility on resource drains. This means that you can work on the heavy load areas, and "leave" the others, more easily than you can with Rest where you're likely going to "over engineer" a lot of your backend. RPC is sort of like Rest in that department, but because it's intended to be like that, its rarely an issue because you chose it for that reason.
Or to put it more simply. You use Odata and GraphQL and not Rest because they lessen the "working together" burden.
(Edit: it’s not even like this is a graphql vs rest change as Netflix have had graphql for years and they have a custom api system that seems a lot more like graphql than it does rest)
I’m curious what you think the best advantages of graphql are, particularly for a larger company like Netflix?
I would guess:
- reduces coordination/contention between different frontend/backend teams (eg there are many clients – desktop, mobile, various smart devices – but maybe other front ends too like something customer support might use).
- cuts down on ‘boilerplate’ api work. For example, maybe you have some property X of each item Y in your catalog, but which you don’t get on the request for the catalog, and that you want to display this now (maybe it’s some new experimental feature you need to thread through from some database table to the front end). You need to choose between adding X to the catalog (potential version woes…) or managing a bunch of requests for X for each Y (risks being slow or bad in a few ways), or add some non-restful api endpoint that takes a list of Ys and gives you their X properties. With graphql you need to teach the backend how to get X of Y but the frontend only needs to change the query.
- it potentially lets you apply access controls in a more straightforward way. If someone is from the US, maybe there are some shows they can’t see. In that case we probably want to hide all records in results about shows they can’t see. (Alternative example: various properties of other people’s accounts). If you have many separate api endpoints, it feels easier to me to accidentally screw up on one of them.
I’m not really very familiar with graphql but the middle option is the most compelling one for some of the things I work on (which are nothing like Netflix). One assumes that some people from Netflix had reasons for graphql and I’m interested in what they were rather than some imagined Venn diagram with the current api in the ‘not invented here’ area and graphql in the intersection of ‘invented here’ and ‘not yet a thing we have’. I’d also be curious to see how well it works out for them over time – perhaps they’ll regret it but maybe not for the reasons you oppose graphql; if there are organisational problems (like some team becoming a big new bottleneck or some teams being unhappy about it being ‘forced on them’), they can be hard to write about.
I would be curious how Netflix itself justifies this except for: we wanted to do this cool thing, so why the fuck not
It's ultimately not unlike how people wonder why the Uber app is so big, when everything seems so simple to them: Each interaction appears simple, but put all the interactions from every person together, and the problem is far harder than it seems.
Isn't the back and forth across the network the big problem?
Also, for a well architected backend you can spread load better on the backend and shockingly you can also leverage caching if you are doing a "dumb" rest api.
… on Book {
title
author
if_viewer_can_purchase {
price
shipping
}
}
But just like SQL, you have people who happened to learn SQL in passing and it’s like a table saw in the hands of someone who doesn’t know what to do.
Having spent over 15 years integrating with, and writing API's, GraphQL was a breath of fresh air! If you think of GraphQL as a query language, you will find it frustrating. If you think of it like you do a REST API with all the ugly pain points straightened out and standardised, then it is a dream to work with.
GraphQL is as much of a query language as REST is. For example, most REST API's end up getting super long lists of parameters in their urls. How many eCommerce API's end up having REST URL's like:
/api/products?category=guitars&include_variants=true&page=20&per_page=100&exclude_fields=description,specs
Whereas, GraphQL transforms that into a simple, readable query, which does not require reading pages and pages of documentation to understand.
But, if you are creating well designed REST API's, I would prefer to use them over somebodies poorly written Graph.
Just like a REST API, a well written Graph should be crafted to be fast and performant to execute. Just like with REST, a Graph designer needs to sit down and think about the ways people will want to use the API. A good graph does not allow endless relationships that will cause performance to blow out.
Seems pretty readable to me.
They can complement either API. From personal experience, choosing not to use them (as most single page web apps don’t) can make testing painful and makes deep-linking to specific states of your webpage impossible.
This doesn't make any sense, unless it's to point out that GraphQL is not a query language at all.
It's not about performance, it's about being able to trivially get the data you need in one request and get it back in a way that you've structured.
On top of that we are running DDD in the backend with ARs being spread over multiple different services.
We did experiment with rest early on, but we found out that we would need to make our frontned to be extremely complicated and vocal to handle the customization/data fetching requirements or build our own REST gateway on top of open API, doing, essentially, the same thing that graphql gateway does out of the box.
Adopting graphql was scary at first with low team exposure to the technology, but using REST would have made our life much much harder
I never understood why people insist on having strong opinions on things they clearly know nothing about.
To be fair, GraphQL has a real specification now but it didn’t back when it was taking off like wildfire.
Edit: I just noticed who I was replying to. Thanks for your work on Cydia and in the iOS jailbraking scene at large. I have lots of fond memories jailbraking and installing all kinds of weird tweaks on my iPhone 4 back in the day.
Netflix is an ecosystem, not just the frontend app for Apple TV. Netflix is pretty close to a vertical monopoly for some of its content. Fund-raising, planning, development, post-production, translations/dubbing, preparation for distribution, and distribution. Your idea of viewing a video content on the app is only the last 20%.
Also worth reminding people that the AppleTV app is not the only app Netflix supports for viewing. There have been dozens/hundreds of different apps and most are still actively supported.
Also, they are a company, so they are always looking to be able to optimize and cut costs.
It’s worth staying curious and asking why before assuming you know more than the engineers who have to work “behind the curtain”.
Would love to hear other opinions.
People here are making an assumption that the entrypoint to Netflix's microservice ecosystem is only for the "TV App". No -- their other domains (incl. talent and production) also depend on the same API gateway. When they upgrade the API gateway to GraphQL for one app and you might as well do it for all apps to get the benefits of the GraphQL toolset.
[1] https://www.infoq.com/presentations/netflix-scaling-graphql/
Ok, so you are saying this GraphQL API is also being used by numerous integrations behind the scenes that are--in your estimation, as I find this idea pretty awkward--managing the fund-raising, planning, development, post-production, etc. of the numerous random production houses that are working on projects for Netflix all somehow within Netflix's central database?
> Also worth reminding people that the AppleTV app is not the only app Netflix supports for viewing. There have been dozens/hundreds of different apps and most are still actively supported.
Is the claim here that one of these applications on other platforms has new functionality that isn't available on any of the major ones? I mean, I listed AppleTV as I can guarantee the app is both not changing and getting slower, but it isn't like I am not using Netflix on other platforms and it isn't as if I don't know it is also embedded into a ton of random one-off devices... I just don't see what point you are trying to make by expanding the scope here as it feels a bit like an attempt to nitpick rather than educate how I might be wrong here.
> Also, they are a company, so they are always looking to be able to optimize and cut costs.
I suggested this as an option in my comment. Do you believe this cuts costs? It wasn't at all clear to me that doing all of this work would cut costs, but if you can explain how that would be fascinating. (It, though, sounds like you don't know anything more about this than I do: you are just less skeptical for some reason. If you work for Netflix and would like to publish an article about why this update was a great idea I would totally be the first to read it and applaud the effort, though.)
> It’s worth staying curious and asking why before assuming you know more than the engineers who have to work “behind the curtain”.
You are literally responding to a comment that begins with "Why?" --one which then continues with a ton of questions that could simply be answered by someone at Netflix if they so cared (and frankly, I will claim, should have been answered in this blog post, as the subject matter begs this question and usually one starts a project postmortem with a comprehensive section on one's motivations)--with a comment trying to establish the moral high ground by... saying I should ask why?
Are custom listings/ordering the blocking issue that takes several seconds to load when the app starts? There’s so much amazing low level work that I’ve read from the Netflix streaming team, but the front end has always seemed lacking (why can’t I easily just get a list of all the videos)? It effects how I use the app?
Most likely that’s less a technical decision than a business decision, so that you never realize the true extent of Netflix’s catalog.
[1] https://github.com/Netflix/falcor
[2] https://github.com/Netflix/falcor/issues/1016#issuecomment-1...
However, for those who wonder what the disadvantages are, here are some. They are from real world production experience and they are not what people usually claim wrongly (like graphql would be harder to secure/version, ...)
No, some real pain points are:
1. Lack of a Map-type. There is no native type to return a list/array of objects but not a map with unique keys or even a set instead of a list. You can fallback to "convention" or custom types (like plain json) but it sucks.
2. Uploading big files (in chunks) is much more tricky than expected
3. There are union-types for the returned data. For example, I can query for an animal and get either a cat or a dolphin and if I use typescript or another capable language then the compiler will tell me that. But I cannot send such a request to the server, saying "here I send you an animal, the data is this ...". The server will have to provide a separate query/endpoint per animal. Very very annoying.
4. While graphql is awesome when it comes to batching requests, what it is not so awesome at is to improve performance by allowing the server to return the part of the data is already resolved. So while reducing the number http requests, now the previous slowest http request will become the bottleneck for all. This means the page might load subjectively faster because some parts are rendered later than possible. There is work going on here (like @defer) but it's not fully spec'd and also the solution doesn't work for some cases
There are more points but this gives you some idea of the ACTUAL problems. Otherwise, from development and setup time over maintenance, analytics, versioning/deprecation, tooling and so on, graphql is much better than your standard REST api in my opinion.
Why do you need special map types for transferring maps? Doesn't list of key-value pairs suffice?
Also, why am I wrong to even ask the above questions?
I do think it's worthwhile to ask the question, though. I have a strong personal preference to avoid encoding invariants like this in wire representations, because they are properties of the _schema_ and not the wire type (JSON object, in my most commonly-encountered case).
Of course, the entire point of GraphQL is to encode invariants in the data schema, so it's entirely reasonable to ask for a built-in Map type instead of building your own for every project.
1. I can signal to my api's users that there won't be any repeating keys without having to write documentation
2. The type can be mapped correctly into the target language. Most languages have a map-type and so it's nice if you don't have to convert it yourself
3. No manual conversion also saves time in most cases, since libraries can do it efficiently in one step. Otherwise you get a list<X> from your library and have to turn it into a map. Not good for bigger lists in performance sensitive applications
4. The same is also true the other way around when sending data to the server
5. It saves space in many serialization formats like json. Instead of [{"key": "key1", value: "value1"}, ...] you get {"key1": "value1", ...}
6. It's nicer to read and debug
7. It's nicer to manually create/write those values in tools like graphiql to test queries by hand and those tools can even warn you if you repeat a key by accident
Just some of the things out of my head, I'm sure I've missed a bunch.
...
5. I thought about something like [["key1", "value1"], ["... like in Lisp-y languages.
And while you can use a uniontype (like String | Integer) to have strings as keys and integers as values, your user now ends up with an ugly typ in their code and has to untangle it by hand.
So the user will have to 1.) know about that it is supposed to be a map, 2.) untangle the list types in their code, 3.) convert it into a map while not mixing up key and value
Therefore I usually go with explicit key/value objects if it's okay for performance or go with a custom JSON type if performance trumps the rest.
https://github.com/graphql/graphql-spec/pull/888
The issue with GraphQL is it tries to appease too many masters.
Similar to jsx. The language isn’t evolving.
The good thing is the spec is (almost) frozen, so there’s many implementations, the bad is it can’t encompass the flexibility of json schema can do.
Going with GraphQL means you have to have good control over both client and server. And like any tool, there are new ways to shoot yourself.
Could you expand on this? I have had the completely opposite experience.
- Falcor can be schemaless with optional json schema where it makes sense
- Falcor supports simpler to reason about and implement features that map perfectly to javascript objects so you can do zero boilerplate databinding with ES proxies without writing query and mutation wrappers
- Does not try to be a qyery language but only expose things you need to request the data that needs to be rendered in the UI
I think not even the creators of falcor fully understood how perfect it is if you just add in the last missing piece of fully automating the connection from the falcor model to the place in the UI templates where the data is needed. Most people tried doing this manually in component boilderplate which lead initially to preloading tens of json paths eg
falcor.preload([`todos.done.length`,`todos.done[0..10].name`,
`todos.done[0..10].date`, ...etc.])
load(`
todos { done 0...10 {name, data} } etc.
`)
The key mistake was that you need to instead think about your UI components ITSELF as the data quueries eg you can automatically derive all falcor quieries from a component like this without ANY boilerplate, and this is only possible with falcor, as everything else has so many features that it would not be possible to map the UI code to the intended data queries
{#each todos as todo}
<div>{todo.name} ({todo.date})</div>
{/each}
Graphql was a cool experiment, but I’m sticking with rest/protobufs
For anything else, it seems pointless.
In terms of code architecture, it drastically simplifies frontend code, and overall reduces backend code and boilerplate. It's extremely elegant and nice to use over creating custom endpoints for every use-case, but the N+1 SQL query problem is a giant thorn.
I see your point about reducing network requests (graphql lets you co-mingle mutations and queries), but I don't think that feature is used often enough, b/c what if one query fails, then everyone has to fail).
Or you use Accept header for its intended purpose
Accept: com.company.user-aggregated/json
Accept: com.company.user-aggregated-lite/json
etc.
And let the server return the fields defined by the relevant schemas.
I mean, the advise given for GraphQL in prod is to literally call predefined queries only.
Caching is trivial to do with REST.
The "really nice caching" that Apollo does literally deserializes and deep inspects both the request and the response to try and figure out caching on specific fields. Also, POST requests are not cacheable according to HTTP spec.
With REST you set an ETag and/or Cache-Control headers and let your existing infra and client browsers handle that.
1. Having to make a new endpoint every time your frontend changes
2. Not making a new endpoint, and just augmenting the existing one for a screen. Which would cause them to accumulate dead data over time (especially if multiple app versions had to be supported concurrently -- in the case of Facebook this was ~3000 versions).
3. Being lazy and finding a similar endpoint meant for another use case and piggy-backing it for your new usage.
All of these would create maintenance and organisational problems.
GraphQL is essentially a system that lets you write endpoint specs (in the form of GraphQL query documents) that the backend knows how to support. It's not a free lunch, you still have to do a lot of work on the backend to allow it to do this efficiently and securely (nowhere near as difficult as people make it out to be though), but it does a really good job at solving the endpoint maintenance problem.
It's worse because in a big system this becomes N+1 API calls to other systems (each with their own permissions, latencies, caching etc.)
Having not used, but only read about graphql, this was the first thing that struck me. It seems great at minimizing data transferred to the client, but it doesn't (seem to) stop that data being fetched from the DB - it just post-filters it.
I'm imagining a scenario like a dashboard, where you're getting a small amount of data from a lot of places. With REST, we'd generally do that by having a dedicated endpoint for the whole dashboard that executes a single query, letting the DB do what it's good at (efficiently retrieving required data). My naive impression is that graphql would execute a bunch of queries, trim what's not needed and join it all together. With anything non-trivial that seems like a performance nightmare.
It also bears mentioning that Netflix is, or pretty much should be, complete. They don’t need to make any changes to the service; if so, it’s very few. I’m sure they need talented engineers to scale up their operations, and great SREs, but what are they still writing code for?
If they were a tech company that'd be a SaaS provider for video streaming, in which case all of this would make sense - the proliferation of streaming providers certainly provides the use-case that someone could offer that as a branded service. But the time for that has come and gone as well - the other streaming services exist, and they work fine.
The de-facto GraphQL Java library comes with dataloaders built-in anyways:
The architectural beauty of GraphQL is that you can write isolated, functional nodes that only have to know about their object type. For example, the User node only has to know how to get the user's last_login, active status, and return a reference to another node type like a profile Image.
Inside the User node, you are free to write code (e.g. Java or Python) that can compute which Image node is the right one to return. i.e. you can write:
class UserNode:
def get_user_profile_image() -> ImageNode:
return some_arbitrary_function()
WRT random requests, there are libraries for query cost estimation you can use as a gate.
Seems like you could do the same batching/coalesce strategy for async Postgres for example, but I don’t see anything after a quick scan of the docs in SQLAlchemy. (Seems like it would be feasible since they already batch requests in unit-of-work, they just don’t coalesce to bulk operations AFAICT.)
For fields which hit external services, you can define types as "loadable" so that every time those are requested in a batch, they are loaded efficiently to avoid n+1.
Edit: Added Apollo
Here are the popular ones to my knowledge:
- Node.js: GraphQL Yoga + Fastify, Apollo Server
- Java/Kotlin: Netflix DGS (built on top of Spring Boot)
- Scala: Caliban, Sangria
- Python: Strawberry, Graphene Django
- C#: HotChocolate
> "On the client side(for mobile), AFAIK iOS(Swift) and Android(Kotlin) does not support GraphQL natively?"
If you want specialized tooling for them, Kotlin and Swift both have great strongly-typed GraphQL libraries.
Apollo publishes libraries for both:
https://twitter.com/fastifyjs/status/1670145210347577344?cxt...
Is that really popular? I'd rather use graphql-kotlin (expedia), graphql spring (yes, recent Spring module), quarkus graphql (code 1st) or just about anything else than that...
> Node.js: GraphQL Yoga + Fastify, Apollo Server
Typegraphql is / was hugely popular; there's also an Express based Graphql server
Aside: there's also Rust and async-graphql
Couldn't recommend it more.
Have to say. Most json-api impls suck, so graphiti has been a game changer for me.
N+1 are a problem buuuut it forces you into good model designs and good tools like graphiti and the latest rails show you the cause of the n+1s so it's not so bad to debug.
And pair this with their Blazor client called StrawberryShake. And your production levels are pretty nuts.
So to answer your question, yes people do use other alternatives
The exact same can happen with a standard REST api. In fact, it's even more likely since there is no type system to assist there.
Would you then call it REST injections too?
Is "zero downtime" relevant here?
For example, when migrating a database that is the source-of-truth for data, then yeah, downtime is something you have to contend with. But for a client-server API change, isn't it trivial? Companies do this all the time -- expose a new version of an API then gradually migrate clients over.
(It's definitely not a trivial task overall! For example, you need thorough testing to make sure you don't break existing functionality and the article does talk about that.)
It autocompletes so you find out more quickly that Netflix doesn't carry it, and what similar options are
I get that you are annoyed Netflix didn't have the movie you wanted, but autocomplete doesn't make that worse.
It’s just a poor solution.
When I search for "How to Train Your Dragon 2", I find Netflix doesn't have it.
But it has the original, plus a few series, so I turn on one of those instead.
"from what I can see in the landing page, REST can do everything it does anyway" is partially true, but is not useful, because how else would have gql picked up adoption if it can't provide similar benefits as REST? I'm smelling a lot of cope from people who have buried their heads so deep in the sand that they cannot fathom a world that doesn't prefer REST anymore. At least try the tech for your own and make informed opinions.
To me it sounds like they try to solve numerous problems: (1) be a query language, ie an improvement over sql, (2) a type safe alternative to REST ie web APIs (3) allow frontend developers to write queries without having to write endpoints, (4) keep clients lightweight, (5) be a single entry point to this hyper graph thing, and more.
None of these things seem compelling in isolation: simply replacing REST adds resolver complexity and doesn’t handle things like file uploads. Substituting sql restricts you to whatever sql is generated and you end up with two query planners. Allowing frontend to query db directly is insecure so you must have fool proof row level security or more resolver complexity. Clients I guess started out thin but now there are all these intelligent caching layers. And this hyper graph thing is only valuable with other graphql services.
So, I guess graphql can be good if you marry it and go 100% all in? But to me it’s a smell when technology claims that your pains can be resolved by using more of the same tech. How am I wrong?
On a gql server, you implement resolvers for the schema. There might be some frameworks that abstract SQL away from the resolver implementation but a resolver may need to make a combination of sql and network calls to other services. Or only network calls to other services.
> Allowing frontend to query db directly is insecure so you must have fool proof row level security or more resolver complexity. I'm not sure why you are thinking that graphql means the frontend is querying the db directly; a graphql server implementation is implement a callback for each graph node that returns the object. It's the server code logic to implement permission checks and the actual SQL or network calls to resolve the object. What part of this results in frontend code querying SQL directly?
It seems like you are getting caught up in the name of "query language" which may be partially a misnomer. It's more like a data fetching protocol
> It seems like you are getting caught up in the name of "query language" which may be partially a misnomer. It's more like a data fetching protocol
Yes, I gave multiple definitions and that’s one of them (2). While that’s certainly true, is that how it’s used in practice and sold? All popular frameworks I’m aware of are selling themselves as having automatic db resolvers with SQL dbs. And thus fast frontend iteration. N+1 solution. Etc.
> It's the server code logic to implement permission check
Isn’t that very difficult to do? Resolvers have recursive relations that can be very tricky to secure, since it’s a high abstraction level. So you tend to rely on RLS or another system to be configured, no?
I really don't think it's any more difficult than if you're not using graphql. If you're hitting a db then you can store auth/permission context for the lifetime of the request. Each time you hit the db to resolve a node, you're going to use the same types of patterns you'd use when fetching objects from the db using sql or an orm. It's customary to have a request context object available in all of the resolvers for just this reason.
If you resolving by calling other services, auth scopes would typically be forwarded on in request headers.
> All popular frameworks I’m aware of are selling themselves as having automatic db resolvers with SQL dbs.
I think I agree if some frameworks for full stack aimed at prototyping where a solo frontend-focused developer needs a backend it abstracts too much, including the db. In these cases, it's probably unnecessary and if you're solo dev on a full stack framework and probably doesn't matter either way.
I've worked on graphql at scale as an frontend service that acted as an API gateway of sorts. The resolvers were not accessing the db but calling other services.
When working on a new application, I would probably use REST and probably would be a long while and would need to have a set of problems to solve with graphql. I think it's quite good at federating services.
To put it simply, even with REST, you're already dealing with the concept of "how does clients get a handle of a particular entity?" With REST, the answer is usually muddled by client data fetching mechanisms and backend architecture that you've opted to use - so just because two projects are doing REST does not mean they're handling these entities in a similar, understandable nature and it requires a lot of background context, same with auth.
graphql makes all that a deliberate matter instead. So take whatever's in your db, decide which fields are you going to expose, and write resolvers that return just that - you can't get any more foolproof than that, it's basically the same thing you do with REST, just without all the client-side fumbling and coordination.
But what I'd argue the thing it does best, is that with REST, you'd probably write code for handling Entity A, and if Sub-Entity A and B rely on Entity A, you'd probably write specific code to ensure the right entities are returned in your endpoints. With graphql, you just reference that entity and it's resolved automatically, because you have written the resolver for that entity already. Resolvers have access to the parent entity that's referencing it, so if Sub-Entity B has stricter controls in what it should expose, the Entity A's resolver can be written to accommodate that (see directives among other ways for how to scale this better).
As for auth, whatever middleware you're already using in your REST endpoints can also be reused one way or another for the gql endpoint, no issue.
> this hyper graph thing is only valuable with other graphql services.
Even without other graphql consumers/services, there's huge value in not having to write multiple ways of handling a certain entity anymore, among other things.
> So, I guess graphql can be good if you marry it and go 100% all in?
All of my recent work related to graphql happened in projects that incrementally adopt it - there's no requirement whatsoever to fully commit to it 100%. It definitely helps that you can reuse whatever your REST endpoints use to comprise code that builds up towards resolvers, like util/helper fns. Nothing stops you from serving the same entity in both REST and graphql forms, heck there's specific usecases that benefit from it (i.e. maybe your mobile app can't keep up with it yet, and you're trying it out for web for now etc)
Say, in such kinds of projects, what I can recommend is identifying a subset of entities that your backend serves that you think (at least superficially) will benefit from it e.g. a relatively new entity that is essentially WIP across releases, and you'd benefit from at least not having to redo the entire REST API endpoint subset for this entity whenever you're making drastic changes, or a particularly old one that could use a rethinking anyway, among other use cases. After a couple of releases you'll get a feel for how it works, and then it'll be enjoyable to port more of the entities later on.
> decide which fields are you going to expose, and write resolvers that return just that
To be clear this means writing custom resolvers? That seems like the most sane way to use graphql, should I ever revisit it.
What’s the logical authorization entity? In rest, it’s (typically) the endpoint itself. What is it in graphql? Does a specific entity resolver have authorization? What does it look like in pseudocode?
How hard is it to write custom resolvers that also produce efficient sql? A gql query can be arbitrarily complex (ie nested), no? How to curb that complexity in practice?
> What’s the logical authorization entity? In rest, it’s (typically) the endpoint itself. What is it in graphql? Does a specific entity resolver have authorization? What does it look like in pseudocode?
You can have both API-wide and resolver-defined auth. In your graphql server config, there's a `context` config option you can pass a function to, and it has access to the whole HTTP request (typically, depends on the integration but I assume all of them treat it the same), it's where you'd probably e.g. check for auth headers and run them against db or the cache layer for auth etc. You can already throw errors and whatnot within this function, so that's the API-wide part.
The returned value of this context function is then included in the params of any executing resolvers, so you can have that resolver throw if e.g. the user stored in context does not have the sufficient roles to access this entity.
There's also directives: say you want to have an `@auth` directive that you can just slap on typedefs that makes auth logic reusable across a subset of entities, but you don't want to handle it on both API-wide and resolver levels, you just write a transform fn, register it in the config, and put that directive on the schema itself.
> How hard is it to write custom resolvers that also produce efficient sql? A gql query can be arbitrarily complex (ie nested), no? How to curb that complexity in practice?
Yes, a gql query can be complex and nested, but you only need to write the resolver in a way that all the ways you need to resolve that entity are taken into account. A query resolver returning one instance of Entity A via their unique ID in most cases is enough, for example - it does not matter how deep into the gql query the entity appears, graphql will do the heavy lifting and refer or run every resolver fn until all entities in that query have been resolved.
The next question is, that means graphql will hit the db one-to-many-times depending on the schema, and yes, that'll happen, but there are again, granular ways to handle that depending on the integration - Apollo at least lets you configure your own in-memory cache or use Redis for example, and be able to cache results in the schema-wide, resolver and response levels.
On my project, we generate SPARQL queries [to simplify, it is not so different from a GraphQL query] and we send it to a SQL translator that then sends that SQL to a Spark cluster. It is reasonably efficient, and extremely flexible.
Of course, this is a single database use case, but I don’t see any n+1 select involved here. And I do not understand how single database graphQL servers cannot work exactly the same.