Hacker News Reader

Comment by forkandwait | Hacker News Reader

So why not pure SQL, which everyone knows already?

Complexity?

1) Parsing SQL is complex, and full of decades of warts in expectation (dialectal differences between databases, among other things). There are few "drop in" SQL parser libraries that are trustworthy and support much more than a bare subset of the language.

2) Query Planning/Optimization is the "secret sauce" of most SQL databases, and is extremely hard in general. To use SQL as a general API outside of an implementation tied to a specific SQL vendor you'd presumably need a "general" query planner/optimizer that becomes a mini-version of an SQL database in itself.

GraphQL's faults aside, it is quite straight-forward to parse (with multiple well known implementations of the parser), and for the most part query planning/optimization in GraphQL follows a simpler, more straight-forward map/reduce-esque "resolver" pattern than the Relational Calculus and general complexity of SQL query planners/optimizers.

> There are few "drop in" SQL parser libraries that are trustworthy and support much more than a bare subset of the language.

Libraries, no, but you don't need a parser library to parse SQL; SQL [even in its DBMS-specific dialects] is a regular language, and regular languages can be entirely formalized using a declarative grammar. You just need a grammar file in a standard format, and a parser-generator like GNU Bison to throw it through.

Conveniently, existing FOSS RDBMSes like Postgres already generate their parsers from grammars, so they have grammar files just laying around for you to reuse (or customize to suit your needs): https://github.com/postgres/postgres/blob/master/src/backend...

> Query Planning/Optimization is the "secret sauce" of most SQL databases, and is extremely hard in general. To use SQL as a general API outside of an implementation tied to a specific SQL vendor you'd presumably need a "general" query planner/optimizer that becomes a mini-version of an SQL database in itself.

Definitely the bigger problem of the two.

Personally, I would suggest reversing the whole problem formulation. You don't need to take your existing system and make it "speak SQL." Instead, it's much simpler (and, in the end, more performant) to take an existing, robust RDBMS and hollow it out into this "mini-version" that queries through to your service.

I've personally done this before: I created a Postgres Foreign Data Wrapper for my data service, then stood up a Postgres instance which has my data-source mounted into it as a set of foreign tables. I then created indices, views, stored procedures, etc. within Postgres, on top of these foreign tables; and then gave people the connection information for a limited user on the Postgres server, calling that "my service's SQL API."

In this setup, all the query planning is on the Postgres side; my data source only needs to know enough to answer simple index range queries with row tuples. I don't really have to do much thinking about SQL now that it's set up; I just needed to understand enough, once, to write the foreign data wrapper.

> a regular language

SQL varies somewhere between a context-free language and a context-sensitive language between dialects and/or edge cases.

Pedantry aside, certainly there are open source grammars for particular dialects. But it is the whole package of "drop in" library including at least some basic semantic modelling that doesn't exist, certainly not to the extent of say an XML or a GraphQL parser library.

> Personally, I would suggest reversing the whole problem formulation. You don't need to take your existing system and make it "speak SQL." Instead, it's much simpler (and, in the end, more performant) to take an existing, robust RDBMS and hollow it out into this "mini-version" that queries through to your service.

Which again, was complexity that I mentioned trying to avoid ("outside of an implementation tied to a specific SQL vendor"). You might not think it a big deal to tie your application to a single database vendor, but that's not always a good option. GraphQL as a language/platform is largely agnostic, supports a large variety of options for backend storage and makes it very easy to support heterogeneous data sources, because it is almost always as simple as "write a function to get the data" (and maybe "write a function to guess how expensive it would be to get the data", if you are optionally using a basic quota system). It does sound like Postgres Foreign Data Wrappers make some of that possible in that very specific database environment, but certainly it's a lot more complex of an option, which gets back to my root problem statement that "complexity" is most probably the reason SQL isn't getting used as much traction as an API option as for instance GraphQL.

> "complexity" is most probably the reason SQL isn't getting used as much traction as an API option as for instance GraphQL.

There's a much more obvious reason: there's no such thing as an "ANSI SQL wire protocol." If you want to take advantage of an existing ecosystem of client libraries that speak SQL to your server (and why wouldn't you?), you literally have to choose a specific vendor's RDBMS wire protocol to mimic, and then tell your clients to configure their client library as if your service was an instance of that RDBMS.

Amazon's Aurora? You need to speak the MySQL wire protocol.

CockroachDB? It speaks the Postgres wire protocol.

If there was such a thing as an "ANSI SQL wire protocol"—and especially if there was such a thing as a "packed binary ANSI-SQL wire protocol"—then I think you'd see adoption of SQL as a lingua franca for things that are not, themselves, SQL RDBMSes take off. As it is, SQL can't be a lingua franca of anything–because there is no "SQL standard" in the sense of an IETF RFC networking standard you can just build your server around. "ANSI SQL" is a language standard (specifically, a mapping from a grammar to a semantics); not a protocol standard.

There isn't even an SQL-wire-protocol standardization committee. Not even a working group. Not even a random FOSS project to attempt to create a "reference ANSI-SQL protocol specification." It's just... not there. (It's kind of bizarre, actually.)

But ODBC exists. Or am I missing something?

Would you use ODBC to communicate to a web API?

Also, ODBC is not quite the abstraction layer I think parent was referring to, because ODBC only (barely) abstracts getting SQL from a client to a server, it doesn't abstract any of the semantics of the communication because it entirely punts that to a "driver" model. ODBC is slightly better than RS-232, but it's about the same abstraction layer from what I've seen.

GraphQL has a lot of advantages over SQL, API-wise; namely it's a lot more natural to describe what you want to get back. But without getting into that, the point is that you usually don't want to give direct data access, with all that entails.

You still want to be declaring the fields and properties of the data that users will be querying, and that usually sits a layer above your raw data.

You can make it a SQL view if you like, and give access to those instead. But it's just a lot of hassle to solve a solved problem in an unfamiliar way.

_______

Edit: Clarifying my stance on SQL views. I'm not dismissing it as a "bad tool", but there's good reasons it's not a popular one today. Writing SQL is not easy for everyone, and it's especially not easy for potential API consumers.

The best APIs are those you grok. If you grok SQL, congratulations, but most devs don't, and APIs are usually designed for most people.

Obviously, if SQL is the right tool for the job, use SQL. But don't start doing stuff like this for ideological reasons.

I mean SQL views have been a thing for literally decades longer than GraphQL. If any technology should be accused of causing a 'lot of hassle to solve a solved problem', it's graphql, not SQL.

If your original data is flat and your app can consume it flat, no worries. But SQL is a worse fit for structured data, so you end up with an ORM on top, which is what GraphQL sort of is. Typically web apps are written with web technologies that work with structured objects, and you want something like {title: "Stapler", price: 1230, category: {id: "123", name: "Office supplies"}, mainPhoto: {url: "https://...", width: 600, height: 300}} back to feed directly in the UI.

Some databases (Postgres comes to mind) are better at letting you express this as a query which builds the structure using JSON functions in the SELECT part of the query. But it's not even close to ideal.

"Some databases (Postgres comes to mind) are better at letting you express this as a query which builds the structure using JSON functions in the SELECT part of the query. But it's not even close to ideal."

Depending on which version of postgres you have(Want to say pre 9.5.) it can be a pain. There are various functions that allow a JSON output and another to make it human readable. With regards to performance I couldn't say but I imagine it would not be as fast as GraphQL.

You don’t need an ORM, just a big JOIN.

JOINs also return flat data structures, no?

Just iterate through the results and recover the hierarchical structure. It’s not exactly rocket science:

    rows = sql("SELECT * FROM parent LEFT JOIN child ON parent.id = child.parent_id")
    results = []
    for row in rows {
        if row["parent.id"] != results.last.id {
            results.add({ id: row["parent.id"], children: [], ... })
        }
        if row["child.id"] != null {
            results.last.children.add({ id: row["child.id"], ... })
        }
    }

but now it's no longer a declarative query. What if you want to include grand children? or if you want to fetch just one child but include it's parent and it's siblings of another type like a product, it's category and the categories tags:

   {id: 23, name: 'Trackpad', category: {id: 42, name:'Equipment', tags: [{id:...}, {id:...}]}}

Now instead of just adding two joins you have to build a custom loop with hand crafted conditions. Sure you can find a way to generalize that loop and put it into a library but still the query itself is separated from some kind of post processing that has be kept in sync.

Nice bit of code.

Just to clarify, wouldn't you need to order by parent_id for the row["parent.id"] != results.last.id to work? Otherwise, say you're ordering by child.date_modified, you could end up with two identical parent_ids in your array?

Yeah, I forgot the ORDER BY clause, but you get the idea.

You have to handle arbitrarily nested hierarchies.

You also need to support referential equality — so that objects with the same type and ID map to the same in-memory JS instance — which requires building an identity map that can live across multiple paginated result set pages.

You have to support untangling the object trees for UPDATE statements, which will require tracking dirtiness to avoid unnecessary requests. Then something similar to deconstruct new objects into multiple INSERTs. Both require a knowledge of foreign-key relationships unless you run all constraints in DEFERRED mode.

Now you're building an ORM.

(Your query also makes assumptions about order and structure that you cannot make in the general case. The devil is in the details.)

In this context, what do you think of Datomic? Wish they make it the usual open source with a paid version with more features.

I don't know Datomic very well, but everything I've read makes it sound a bit like a modern take on an object database. Certainly interesting, but as long as it's commercial software I'm not going to touch it. Its close coupling (AFAIK) to Clojure is also not interesting to me.

I’m a huge proponent of sql, for me it’s virtually impossible to beat as a query language.

We played with wrapping our dB layer with graphql recently and I would say that the experience was pretty good. In part it’s because you’re working a layer up where you can have a more pluggable model. In the first run you can model your local dB structure, but it’s easy to glue more stuff into it from other sources and generally evolve the structure.

I find it very easy to imagine a better SQL.

Two things I'd fix:

* make it explicit when a join is expected to have 1:1 semantics vs 1:n, because it's very easy to end up with duplicated rows otherwise

* increase composability of the syntax; why do we need HAVING, why isn't WHERE enough? We have relational algebra, projections, filters, joins, folds, union, intersect etc. But the syntax of SQL is so idosyncratic, and at times antagonistic to the stream and set-like intuitions behind relational algebra.

Regarding your first point, I’m trying to imagine what that would look like. So you mean something you could put on there to say (join x [1:1] on blah)? In general the mistake that’s easiest to make is to get duplicate rows from another join later, which this wouldn’t help with - but maybe I’m misunderstanding.

I don't really understand your comment about it always being join x+n that causes the problem instead of join x. What is different about later joins that is not true when the later join is the current join, when you get around to writing it? If joins for tables T1::T2 and T2::T3 are 1:1, they don't magically become 1:n when you do T1::T2::T3.

To do it right, you'd need to mark up the relations with expected cardinality (effectively unique constraints), so that the DB would be able to verify ahead of time whether a join is going to be 1:1 or 1:n. That would be a better solution than a join working fine up until the cardinality expectation is violated and it suddenly stops working.

If we had to stick with SQL syntax, it might be something like:

   select a.*, b.*
   from a
   join one b on b.id = a.b_id

(Unique constraint from primary key on b.id. It may still filter down the set of rows in a, but it won't duplicate.)

Or:

   select a.*, b.*
   from a
   join many b on b.some_key = a.some_key

(No unique constraint on b.some_key)

I think the interesting cardinality distinctions are 1:1, 1:n and 1:0.

(The above syntax is ambiguous with aliases, so it wouldn't fly as is. But it gives a flavour.)

So the idea is that when you write the queries with these explicit join requirements, you'll want the query to fail if someone changes the schema later in such a way that the requirement is no longer met? If so, it seems like something that comes up so infrequently as to not warrant the noise, but maybe I'm still misunderstanding.

What I was saying was that often people have T1::T2 (1-1) and then throw in a later join T2::T3 (1-many) without considering the implications for the T1 (maybe they were doing a count(T1.id)). At least that seems to be an aspect that confuses developers newer to sql (that the many join is duplicating other parts of the data).

It comes up quite a bit if you have repeated data in batches where the batches are a first class concept. Any relations need to be qualified by both normal fk id and batch id, or else you bring back across all batches.

Think something like measurements samples that come in an envelope with metadata, or account updates that come in a statement, that kind of thing.

HAVING is not the same thing as WHERE. WHERE filters the query to produce a result set. HAVING filters the result set.

    select id, count(*) - 1 as nduplicates
       from parts where color = 'red'
       group by id
       having count(*) > 1;

WHERE cannot filter on the count(*) because WHERE is applied during the query before the count is known.

I agree SQL is kind of warty, and it would be great if it were easier to compose from programs, but HAVING is not redundant to WHERE.

I believe this should have been in response to its sibling. The set theorist in me agrees - the two operations are working on 2 different sets at 2 different levels of abstraction.

Yes, replied to the off by one post.

But that's equivalent to the following query which doesn't use having, only select

    select * from (select id, count(*) - 1 as nduplicates
       from parts where color = 'red'
       group by id) _inner
       where nduplicates > 0;

Now obviously that doesn't read as easily, but WHERE can filter out the count(*) perfectly.

> You can make it a SQL view if you like, and give access to those instead. But it's just a lot of hassle to solve a solved problem in an unfamiliar way.

It's kind of odd to use that to dismiss the solution that was established and widely available for 30 years longer, and which more people are probably familiar with.

See my edit. I'm not dismissing them, but the range of cases where they're the correct tool over a REST API, GraphQL, or something else... is very narrow.

Out of curiosity (didn't had the time to get up to date on it yet), how does GraphQL let you describe your returning data better? Is it because it has hierarchical types (types and collections inside types) or it's something else?

With GraphQL, you are creating what is essentially a 1:1 mapping of the structure of the data you want to get back. It's a lot more intuitive than joins and such.

A join in GraphQL looks like this:

{ getOrganisation(args){ name, employees{ Name, Position }}

(I added commas to make it easier to read, but they are not supposed to be there)

I know SQL, so I don’t think it’s necessarily easier, but what is easier, is non-sql-savvy people making efficient queries because GraphQL does a lot of the heavy lifting for you.

Why is writing SQL so hard for "most devs" I was moved to a Oracle based project (the management system for the uk's core internet) and only had a weeks training on PL/SQL.

I got a high performance award after that project was completed and I don't consider myself a Rockstar developer

You have a week's worth more experience of training than "most devs".

"Most devs" don't directly deal with SQL these days, because ORMs are good enough for day-to-day operations (and the non-day-to-day ends up being handled by people who actually know SQL). In fact, "most devs" rarely interact with databases. A lot of data sourcing is API driven these days.

Now, I don't claim to like or dislike that particular state. But you have two choices: You can innovate, which carries risks (of being wrong; of not selling your idea correctly; of reinventing the wheel; of making the same mistakes others did; ...). Or, you can play it safe, and create APIs that most people will know how to use without having to go through a 1-week course, because short as that may sound, that's 167.5 more hours than anyone will reasonably be willing to invest in understanding your API.

So at the end of the day: Use the right tool for the job. SQL for APIs is usually not the right tool.

Reply on news.ycombinator.com