Teaching Rust the SQL Language
SQL is not compiled straight to machine code, but building a SQL engine forces you to face a similar question: how do you get one language (Rust, in my case) to understand and execute another language (SQL) that its own compiler knows nothing about?
A database does not grow out of a pile of existing code. It grows out of a set of concepts: rows, columns, expressions, joins, aggregates, indexes, transactions.
The starting point for my engine is a generic SQL dialect heavily influenced by SQLite and DuckDB, because they both provide a large corpus of tests in a common format called SQLLogicTest. My theory was that an engine that could pass those tests would have a solid baseline of compatibility. My goal is not to invent a new flavor of SQL, but to support enough of what those systems already run that I can reuse their test suites and their ecosystem of queries.
I did not write my own parser. I use `sqlparser`, which turns SQL into an abstract syntax tree (AST), a graph-shaped in-memory representation of the query. From there I map that AST to my own set of Rust enums and data structures. That is the point where SQL stops being text and starts being a set of concepts the engine can reason about.
Instead of starting with an optimizer, I started with a test harness. I wired it up to run the SQLLogicTest corpus that SQLite publishes and added some support for DuckDB-style tests. Today all of the SQLite-provided tests I run are passing. That does not mean the engine is bulletproof; this is still early greenfield code. It does mean that the behavior of the engine is anchored to something outside my own head: when a test fails, it is my implementation vs. a known result. That suite has become the de facto specification; any change to the planner or executor has to keep it passing.
At the beginning I thought I would write my own compute kernels for every data type I wanted to support. When you are inexperienced in a domain, your mind plays tricks on you and convinces you that reinventing is the only way to get it "right." I eventually moved the engine onto Apache Arrow's memory model. Arrow gives me a columnar representation and a set of kernels for common operations. Instead of hand writing and maintaining all of that, I can focus on planning queries and mapping SQL semantics onto Arrow arrays.
This project is a solo effort. To compensate, I lean on LLMs. On any given day I bounce between Claude Sonnet 4.5 and GPT-5.x Codex to help sketch out new features, reason about lifetime issues, or explore alternative designs. They are not reliable enough to trust blindly, but fast enough to act as noisy, context-aware pair programmers.
This creates a balancing act: I have a relatively large test corpus that must keep passing, I want new features to be efficient, and I do not want subtle correctness regressions hiding behind "clever" LLM-generated code. The tests form a guardrail. If a suggested optimization breaks behavior, it gets thrown away. If it passes but looks fragile, I refactor it.
I have never written a compiler. This project is the closest I have come. It sits somewhere between a query engine, an interpreter, and a lesson in how far you can get by standing on the shoulders of other systems: SQLite's tests, DuckDB's style of queries, Apache Arrow's memory model, and LLMs that sometimes suggest the right idea in the wrong shape.
If nothing else, it has answered my childhood question. Computers do not "just understand" new languages. Someone has to build the bridge.
(If you like this story, search for "rust-llkv" and give it a star.)