Comment by jitl | Hacker News Reader

Comment by jitl | Hacker News Reader

Parent Full thread

We’ve use an API style similar to tRPC at Notion, although our API predated tRPC by 4 years or so.

You can build this kind of thing yourself easily using Typescript’s mapped types, by building an object type where the keys are your API names, and the values are { request, response } types. Structure your “server” bits to define each API handler as a function taking APIs[“addUser”][“request”] and returning Promise<APIs[“addUser”][“response”]>. Then build a client that exposes each API as an async function with those same args and return type.

We use this strategy for our HTTPS internal API (transport over POST w/ JSON bodies), real-time APIs (transport over Websockets), Electron<>Webview APIs (transport over electron IPC), and Native (iOS, Android)<>Webview APIs (transport over OS webview ipc).

For native APIs, the “server” side of things is Swift or Kotlin, and in those cases we rewrite the request and response types from Typescript by hand. I’m sure at some point we’ll switch to a binary based format with its own IDL, but for a single cross-language API that grows slowly the developer experience overhead of Protobuf or similar hasn’t seemed worth it.

In our current project with a TS frontend and Python backend, we use an OpenAPI schema as the source of truth and openapi-typescript-codegen [0] to interface with it on the client side. While not perfect, it provides a very nice interface to our API with request/response typings.

I also wrote a 10-line mock API wrapper that you can call as mockApi<SomeService["operationName"]>((request) => response), and it will type-check that your mock function implements the API correctly and return a function that looks exactly like the real API function.

[0]: https://github.com/ferdikoomen/openapi-typescript-codegen

Can second this approach. At a past job we did the same, except to connect the frontend to a Go backend.

I really like that the openAPI approach is language agnostic, and makes it relatively simple to support SDKs for many other languages if needed. For any company where the API itself is a product, OpenAPI is great.

We use OpenAPI internally for communication between systems. It eliminates an entire category of bugs, and we can offload testing of schema conformance entirely to third-party tools. We've never had a bug due to a mistake in calling an internal API that I know of. And we get both internal documentation and a web UI for free via SwaggerUI and Redoc.

Good zero-config OpenAPI support is one of the best features of the FastAPI framework in Python. The "fast" part refers to the speed of basic product up and running.

Another great library to generate TS types from OpenAPI is https://github.com/drwpow/openapi-typescript . It provides the types as single objects you access via indexing, which is pretty nice. There's a partner library to generate a typed fetch client.

Do you know if there's something similar for Python? Or what approach would you use for Python?

Like generating pydantic models or dataclasses for an OpenAPI schema? I haven't needed to go in that direction myself, but this[0] looks promising!

Apologies if I've misunderstood your comment

https://koxudaxi.github.io/datamodel-code-generator/

This looks interesting, thank you!

I'm basically wondering what would help us produce (automatically?) a Python HTTP client based an OpenAPI spec, and achieve a development experience similar to using drwpow/openapi-typescript.

drwpow/openapi-typescript will generate the whole schema, along with paths, request and response models, as well as path and query parameters. In the IDE, all client methods will be type-safe and autocompleted.

Example:

  client = createClient<GeneratedSchema>();
  
  client.get("/my/autocompleted/endpoint/{some_param}", { params: { path: { some_param: "foobar" } } })

koxudaxi/datamodel-code-generator seems to generate the response models, but not much else. It seems we would have to manually write wrapper methods for each endpoint, manually specify the parameters and request models, and use a type hint for the return value of each method with the generated response model.

Example:

  def my_wrapped_endpoint(...) -> ResponseModel:
    pass

  client.my_wrapped_endpoint(some_param="foobar")

I'd like to avoid any manual wrapping or at least minimize the amount of it. Basically: to achieve a similar experience to what we have in TypeScript.

I hope my question is a bit clearer now. I'm not that familiar with Python, so I will appreciate any guidance regarding this problem. :)

Gotcha. I’m not sure what options are out there. Haven’t needed to do any codegen to python - only from python.

For all its issues, the JS world typically provides an excellent developer experience and takes types far more seriously than python.

Hope you find something that helps, and if you do I’d love to hear about it.

Thanks! I'll keep looking for a solution and keep you in the loop once we figure something out.

This is the way. tRPC adds unnecessary complexity over simply inferring types. My theory is that no well maintained and promoted library adopting this approach has emerged, and that’s why you don’t see it discussed very often.

Is there some trick to doing validation of request data using this process? That's a valuable part of using something like tRPC, JSON Schema + type generation, zod, etc.

We use an internal validator library that we infer request types from. It’s similar to Zod (but also predates it by a year).

I’ve also spent some time on a Typescript type to X compiler. My first prototype is open source and targets Thrift, Proto3, Python, and JSON schema: https://github.com/justjake/ts-simple-type/tree/main/src/com...

I’m not happy with the design decision in that codebase to try to “simplify” Typescript types before compiling, and probably won’t continue that implementation, but we have a few internal code generators that consume TS types and output test data builders and model clases we use in production.

I want to open source some of those bits but haven’t found the time.

Deepkit is a fantastic solution for this. It uses a compilation step to inject metadata about types into plain JS.

https://deepkit.io/

Deepkit looks really cool, but it’s so complex on the inside and leverages a forked/patched Typescript and requires full typecheck before emit.

What happens if the Deepkit guy retires? What if I want to run my code without waiting for 11 minutes of typechecking? What if there’s a bug somewhere in there?

There’s way too much risk for me to consider Deepkit for production.

Yeah, I’m reading their sample code and wondering if this is just type-imbuing wrappers on top of XHR calls. It even asks you to provide the generic argument in the invocation (this sucks for trying to keep your dependency tree in order).

This works quite well actually:

https://github.com/mikew/transmission-material-ui/blob/maste...

https://github.com/mikew/transmission-material-ui/blob/maste...

That's seems to be completely missing validation. Typescript types are at their worst when they are lies, and the actual shape of the data is something completely different.

It's copied and adapted directly from their RPC docs, validation would just be busy work.

https://github.com/transmission/transmission/blob/main/docs/...

For Transmission, maybe. In general, validation is very much not "busywork".

On top of automatically generated types (eg. openapi, graphql), I would say it is.

Seems like you're just inserting yourself and ranting about typescript where it's not really applicable.

Lovely attitude, you have a nice day too.

There's others expressing the same concerns, for example https://news.ycombinator.com/item?id=37101393

What's your solution though?

To me, if the server has updated it's schema and the client has old code, the server responds with an error, the user sees "something went wrong".

And if the validation fails on the client side, the user sees "something went wrong"

And if the client isn't doing validation of what's returned, and an error gets thrown because it tried to access a now missing field, the user sees ... "something went wrong"

Intentionally maintaining multiple versions of an API, or making sure your API changes are backwards compatible (like adding new fields and marking old ones as deprecated), those are solutions but definitely require organizational effort.

All totally expressible via Typescript though!

With an explicit error you get from validation (like Zod, as used in tRPC), you could notice that your client is out of date and either refresh automatically or prompt the user to refresh[1].

The benefit from validation is that the error is easily recognizable & can be handled nicely, even in just one place. Without validation, you get things that Typescript claims are impossible, for example exceptions on code paths that look like they cannot throw, and you can't easily tell why any of that happened -- that's a recipe for miserable debugging. Of course, with less busy, less critical, sites you might not notice or care.

[1]: Refresh could cause loss of user input (e.g. text just entered), depending on the app that might matter and need browser-side storage or something along the lines of the "local first" manifesto. Easy kludge for simple apps with smart users is to make the user copy the text, click refresh, paste.

Isn't that only for one case of errors though (server not matching expected response)?

One solution I thought was clever, was a friend's single-page app would hard-reload on navigation, if there was an update to the assets. Not sure how feasible that would be for schema errors though.

Between that and catching 400 Bad Requests coming back from the server, you're pretty much catching any communication errors resulting from version skew.

SvelteKit can poll to check server version and refresh, both on timer and if assets are 404. But as you implied, that doesn't help with API evolution and e.g. users interacting with a form (apart from increasing the likelihood the page will refresh soonish due to navigation).

For cross-language, I can recommend Fern, which works with OpenAPI

http://buildwithfern.com

You can recommend it in what context, from openapi (as they claim https://github.com/fern-api/fern#starting-from-openapi ) or from their ... special ... definition schema?

For those wanting less talk, moar code: https://github.com/fern-api/fern-java/blob/0.4.2-rc3/example... -> https://github.com/fern-api/fern-java/blob/0.4.2-rc3/example...

Regrettably, I wasn't able to readily find a matching openapi example, likely cause they really seem to believe their kid-gloves format is the future (or lock in, depending on how bitter one is feeling)

---

confusingly, their repo has a "YCombinator 2023" badge on it that just links to the badge itself. Some Algolia for Launch HN didn't cough up anything, but there was a Show HN I found: https://news.ycombinator.com/item?id=34346428

Disclosure: I'm a contributor to Fern.

This is good feedback that we need to provide more examples starting with OpenAPI instead of with the Fern Definition. For some context, we convert the OpenAPI spec into a Fern definition and then pass that into the code generators.

If you want to see some real world examples, check out these links:

- https://github.com/vellum-ai/vellum-client-generator/tree/ma... -> https://github.com/vellum-ai/vellum-client-python

- https://github.com/Squidex/sdk-fern/tree/main/fern/api/opena... -> https://github.com/Squidex/sdk-node

Worth calling out that you can go from the Fern Definition -> OpenAPI anytime to prevent lock in.

Similarly, we built a typed-client for inter-service communication using lambdas at Robocorp.

The requests and responses are inferred from the interface (published following semver) defined in Zod (including the errors that are following HTTP-like conventions: 401, 409…).

It also includes “hooks” for pre and post processing on the server side (effectively middlewares)

I built something that sounds very similar at my last company called crosswalk. Open sourced here: https://github.com/danvk/crosswalk

One thing that worked surprisingly well: codegen TypeScript types from your database and use those in your API schema.

Reply on news.ycombinator.com