Hacker News Reader

Comment by d33 | Hacker News Reader

It's a pity that the thing has to get reimplemented all over so many times. For example, you'd think you could reuse PostgreSQL's parser, but its code is autogenerated and riddled with side effects:

https://github.com/postgres/postgres/blob/master/src/backend...

We've a lot of got languages and usually their parsers are tightly coupled, basically forcing people to reimplement them if you want a flexible parser. I'd say it would be great if we had one parsing library for each language, ideally used at its core. I think that would greatly simplify implementing linters, static analysers and maybe even tools that generate code. Also, perhaps this would enable even more cooler hacks?

View on HN

Just in the off-chance this is useful to someone: If you find yourself needing the Postgres parser elsewhere, e.g. in some custom tooling to analyze SQL queries, I extracted it into a library some time ago:

https://github.com/lfittl/libpg_query

Your comment still applies though, as a lot of the parser output is very specific to the chosen Postgres version (10 at the moment) and can change unpredictably.

I am in the middle of a project similar to what uber is doing and it leans very heavily on this library. Thank you so much for this! I would not have been able to pull it off without libpg_query.

There's also HSSQLPPP for Postgres, and it's written in Haskell

https://github.com/JakeWheat/hssqlppp

Or simple-sql-parser by the same author which is just a parser.

https://github.com/JakeWheat/simple-sql-parser

I agree, I wish every vendor would publish an official open source parser for their SQL dialects, or that the community would create a repository of standard parsers or grammars for them all.

We are almost there with ANTLR grammars... https://github.com/antlr/grammars-v4

The trickiest bit isn't parsing per-se, but disambiguating references, so grammar alone isn't enough.

And unfortunately, disambiguating references doesn't work the same way everywhere - for instance, tables on Hive will shadow a CTE.

Definitely agreed on the amount of duplicate effort. Apache Calcite seems to be a good solution but usually gets ignored: https://calcite.apache.org

Interestingly, Uber does use Calcite themselves for AthenaX: https://github.com/uber/AthenaX/

When I looked at Calcite, the branding was very focused on query optimization, which was sufficiently unrelated to what I was doing that I didn't dig in. Had I, it might've been the starting point instead of my personal project, although at the time it didn't have Vertica support (I think that was ultimately added by a different group at Uber).

Yes, it's not marketed much, but then again it's a very specialized project that isn't relevant for many people.

It is pretty good though and used by plenty of other Apache projects like Apex, Drill, Flink, Hive, Storm, etc: https://calcite.apache.org/docs/powered_by.html

Reply on news.ycombinator.com