Show HN: Orbital – Dynamically unifying APIs and data with no glue code | Hacker News Reader

Show HN: Orbital – Dynamically unifying APIs and data with no glue code

github.com

86 pointsmartypitt15 comments

Hey HN,

I'm excited to share Orbital, a new approach for unifying APIs and Data sources!

Rather than relying on glue code to bridge endpoints, Orbital leverages annotations in schemas & API specs to build the integration dynamically.

The traditional method of crafting glue code often becomes repetitive and burdensome to maintain in the long run.

With Orbital, developers embed tags to their existing API specs (OAP, Protobuf, etc), indicating where data can be sourced, and publish these specs to Orbital (which runs self-hosted).

Consumers query these tags with our TaxiQL language, and Orbital generates the integration on the fly. That could be merging multiple APIs, blending API and database queries, or enriching event streams to craft custom message payloads.

It feels a lot like writing GraphQL, but there's no resolvers to maintain, and producers are free to use a variety of API Spec languages.

The beauty of using tags over field names is the adaptability. As API developers update and publish their specs, the integration remains seamless and automatically adjusts.

Under the hood, the tags (and associated query language) are actually Taxi - an OSS meta-language and toolchain we build (and have shared previously). Orbital is a query engine that executes TaxiQL queries, generating the integration.

We've been working on this for a while, and have a number of production deployments. We recently made the move to make the source available on Github under a mix of Apache 2 (Open Core) and BuSL.

In a nutshell, Orbital excels in Data Composition (covering APIs, DBs), crafting Bespoke Event Streams, and streamlining ELT workloads into databases.

Excited to hear your thoughts and feedback!

Open articleView on HN

> the source available on Github under a mix of Apache 2 (Open Core) and BuSL.

I for one don't care for "logic" in license files <https://github.com/orbitalapi/orbital/blob/7b167ed37c31cf866...> since it makes it very hard to reason about the license usage without, as in this case, viewing every single file to look for license-traps

In reality, most larger projects are already like that - except with no logic spelled out. It's completely ad hoc. Instead you have to hunt in commit logs and compare with old versions of open source projects they based their source on. Sometimes the upstream changed their license too at some point in the past, and if you are lucky, you can still find a reference on some web page. The upstream project can have the same situation itself, in a sort of fractal way. It's turtles almost all the way down, until you end up with an early transcription of the Fortran BLAS routines to C by way of Pascal, with the intermediate Pascal routines lost somehow in the mists of time. Our historians now believe it all started after the moon landing in what they called Jan 1 1970 00:00 or "0" or origin. After seeing for themselves how inhospitable Space is, the humans decided to stay on Earth and focus their efforts on the study of source code. Some argued that it would eventually lead to the encompassing of all human thought and thus to Enlightment for all. Yet others feared a world where human thought has no consequence. Most are too busy with work to worry either way. After all, at the end of the month another name of God in the markets is what pays the bills¹ and who has time to worry about whether the Monad⁰ comes² or even exists.

0: https://en.wikipedia.org/wiki/Monad_(Gnosticism)

1: https://cyclonedx.org/

2: https://stackoverflow.com/ a / 1732454

This is a fair comment, and something I think we could do better at... bear with us, we're moving from entirely closed source, to Open Core.

I've partially tried to address this with the FAQ[0]. Maybe we could move the codebase around, so there's community/ and commerical/ top-level dirs, and associated licenses under there. (Naming TBD).

[0]https://github.com/orbitalapi/orbital/blob/develop/LICENSE_F...

This looks very neat! I like the approach of including all data (API, messaging, database, files) as I went through similar thoughts and implementation recently with my project.

A growing pain for data integrations is data validations. The schema can be correct but the data itself may have incorrect values. I noticed you had a section about the structure and semantics of the data here (https://orbitalhq.com/docs/background/intro-to-semantic-inte...) but don't see where you could validate records using it. How do you handle these scenarios?

Currently, we don't have validations implemented.

We've spiked it in the past, and had some nice designs that attached validation rules semantically to the scalar types (which are very sharable), and models (which are more system-specific).

The early designs felt nice (albeit early), but the deal we were working on didn't eventuate, so we didn't push it through.

I think Validations become really interesting in both evaluation flows (ie., ingesting in a pipeline or through a mutating API), as well as quality scoring for data-at-rest.

I'm excited to explore these in more detail, given a use-case.

Interesting tool! I watched the video in which you built a query using the query builder and mentioned that you can copy and paste it into your project. Do you have plans to further integrate it with projects (in form of some API call, or your SDK), or is it already part of the product? It can be cumbersome to repeatedly copy and paste code snippets every time something changes.

Thanks!

Yep, we have (early) SDK's for Typescript and the JVM, to submit the query directly.

Also, you can publish queries as standalone APIs, in a BFF-style pattern.

Every time I try to skip the glue code, it bites me. You need the glue layer to allow for indirection, tight coupling through codegen or reflection will work early on but eventually a dependency will need to be shimmed, proxied or otherwise modified- having the glue layer gives us a way to make that change.

In software eng, the one true metric for delivering is how fast can you iterate on a piece of software. We already have constructs to deal with code duplication (classes, etc).

What's a difference between this and sth like GraphQL Mesh or Wundergraph?

There are no resolvers to maintain in those either but you get "standard" GQL instead of TaxiQL which I assume has much smaller ecosystem. What are the strengths of TaxiQL?

For teams that want to adopt GraphQL, I think those tools are pretty great (and I've sent customers to them). However, we know from our customers there's lots of teams who don't want the shift of GraphQL (which tends to lead to GraphQL everywhere), but still want API composition.

At a language-to-language feature level, there's some core differences:

GraphQL-without-resolvers still relies on imperative joins defined somewhere. AFAIK, Wundergraph pushes this to the query layer. GraphQL Mesh pushes this to .yaml files. This means that as the sources change, there's centralized config needs to be updated in lock-step, or things break. This means the API producer is now obligated to work closely with consumers to release changes.

TaxiQL infers join logic by introspecting the schemas. This is especially powerful where joins require hops across multiple sources (eg., Id resolution where services use different ID schemes). Because type metadata is embedded at the producer schemas, producers are free to make changes and the query layer adapts.

GraphQL promotes a single graph that everyone cherry-picks from. TaxiQL lets query authors define their own data contract, and own data shapes. This moves transformation of code into the query layer, which we think is a better place for it.

GraphQL doesn't allow extensibility in the language. (We've seen others hack around this with framework-specific annotations, but that's always felt quite awkward). TaxiQL supports extensibility, adding custom functionality into the query layer, and ships with a pretty rich stdlib for things like avg(), first(), last(), etc.

Have been looking into Taxi over the last year. Really neat and powerful tool. I'd be so curious to see what could be built by creating a plugin for a LLM!

We've been playing with LLM's and OpenAI to build integration from plain text.

There's a video of an early version of it in action here: https://www.loom.com/share/a78c70f0e2bd42a28b7cb0a5d175c3aa

I think what's nice about using LLM's to build Taxi queries (which is what's actually happening here) is that we're not having to push things like API Specs and DB Schemas to the LLM. Instead, we train it on the Taxi types that are available, get the LLM to build a Taxi query, and let Orbital generate the integration.

It means a lot less information is pushed out.

Still quite early, but it's really powerful.

Sound similar to what WunderGraph is doing

I'm wondering whether GPT-4 can be used for the same task? Only that instead of chatting you will provide a common schema for the integrations and let GPT-4 figure out how to make the requests and fill out the fields, etc.

What if a piece of data exists in multiple services? How does the tool know which service it should use to fetch the data, given the link is just a semantic representation - but that representation can exist in multiple places?

Generally, data that comes from two different sources, and has different values, aren't semantically equivalent, and so get assigned different types.

eg:

   type ImdbFilmReview extends FilmReview
   type RottenTomatoesFilmReview extends FilmReview

Which allows you to query for either "All film reviews", or just a subset.

We've found that actually covers about 80% of use cases.

However, some users have asked for more control in the query langugae, which we're working on improving the TaxiQL query syntax for this right now.

Here's what it looks like:

   find { Film[] } as {
     id : FilmId
     rottenTomatoesReviews : RottenTomatoes::FilmReview[]
     imdbFilmReview : Imdb::FilmReview[]
   }

Reply on news.ycombinator.com