HNHacker News
TopNewBestAskShowJobs

phrasecode

22 karma · joined July 23, 2019

Language, Runtime, and Systems Engineer; Principal Engineer & Lead Researcher at Convergent Machines Lab.
submissionscomments
phrasecode··on Show HN: LinkedQL – Live Queries over Postgres, MySQL, MariaDB
You're welcome.

And of course achieving that "exceptionally high throughput and performance" is the ultimate goal for a system of this nature.

Now, yes — LinkedQL reasons explicitly in terms of transactions, end-to-end, as covered in the paper.

The key structural distinction is that LinkedQL does not have the concept of its own transactions, "since it doesn’t initiate writes". Instead, it acts as an event-processing pipeline that sits downstream of your database — with a strict "transaction-through rule" enforced across the pipeline.

What that transactional guarantee means in practice is this:

Incoming database transactions (via WAL/binlog) are treated as "atomic" units. All events produced by a single database transaction are received, processed, and propagated through the pipeline with their transactional grouping preserved, all the way to the output stream.

Another way to think about it:

You perform high-throughput writes (multi-statement transactions, bulk writes, stored procedures, batching, etc.)

  → LinkedQL receives the resulting batch of mutation events from that transaction
  → processes that batch as "one" atomic unit
  → emits it downstream as "one" atomic unit
  → observers bound to the view see a "single" state transition composed of many changes, rather than "a flurry" of intermediate transitions.
Effectively, a systems that thinks in terms of batching and other throughput-oriented write patterns. LinkedQL just doesn’t initiate its own transactions — it preserves yours, end-to-end.
phrasecode··on Show HN: LinkedQL – Live Queries over Postgres, MySQL, MariaDB
Thanks — this is a really good scenario to walk through, and I’m happy to extend the conversation.

First, I’m implicitly assuming your 10 ECS tasks are talking to the same Postgres instance and may issue overlapping queries. Once that’s the case, WAL slots and backend orchestration naturally enter the story — not just querying.

A few concrete facts first.

PostgreSQL caps logical replication slots via `max_replication_slots`. Each LinkedQL Live Query engine instance uses one slot.

Whether “10 instances” is a problem depends entirely on your Postgres config and workload specifics. I’d expect 10 to be fine in many setups — but not universally. It really does depend.

---

That said, if you want strong deduplication across services, the pattern I’d recommend is centralizing queries in a separate service.

One service owns the LinkedQL engine and the replication slot. Other backend services query that service instead of Postgres directly.

Conceptually:

[API services] → [Live Query service (LinkedQL)] → Postgres

From the caller’s point of view this works like a REST API server (e.g. `GET /users?...`), but it doesn’t have to be "just" REST.

If your technology stack requirements allow, the orchestration can get more interesting. We built a backend framework called Webflo that’s designed specifically for long-lived request connections and cross-runtime reactivity — and it fits this use case very naturally.

In the query-hosting service, you install Webflo as your backend framework, define routes by exposing request-handling functions, and have these functions simply return LinkedQL's live result rows as-is:

  // the root "/" route
  export default async function(event, next) {
    if (next.stepname) return next();

    const q = event.url.q;

    const liveResult = await client.query(q, {
      live: true,
      signal: event.signal
    });

    // Send the initial rows and keep the request open
    event.respondWith(liveResult.rows, { done: false });
  }
Here, the handler starts a live query and returns the live result rows issued by LinkedQL as "live" response.

  * The client immediately receives the initial query result
  * The HTTP connection stays open
  * Mutations to the sent object are synced automatically over the wire and the client-side copy continues to behave as a live object
  * If the client disconnects, event.signal is aborted and the live query shuts down
On the client side, you'd do:

  const response = await fetch('db-service/users?q=...');
  const liveResponse = await LiveResponse.from(response);

  // A normal JS array — but a live one
  console.log(liveResponse.body);

  Observer.observe(liveResponse.body, mutations => {
    console.log(mutations);
  });

  // Closing the connection tears down the live query upstream
  liveResponse.background.close();
There’s no separate realtime API to plumb manually, no explicit WebSocket setup, and no subscription lifecycle to manage. The lifetime of the live query is simply the lifetime of the request connection.

---

In this setup:

  * WAL consumption stays bounded
  * live queries are deduped centrally
  * API services remain stateless
  * lifecycle is automatic, not manually managed
I haven’t personally run this exact topology at scale yet, but it fits the model cleanly and is very much the direction the architecture is designed to support.

Once you use Webflo, this stops feeling like “realtime plumbing” and starts feeling like normal request/response — just with live mode.

phrasecode··on Show HN: LinkedQL – Live Queries over Postgres, MySQL, MariaDB
Would you clarify what a "transaction" in this instance would mean?

LinkedQL definitely optimizes at multiple levels between a change happening on your database and the live result your application sees. The most significant of these being its concept of query windows and query inheritance which ensure multiple overlapping queries converge on a single "actual" query window under the hood.

You want to see the engineering paper for the full details: https://linked-ql.netlify.app/engineering/realtime-engine

phrasecode··on Show HN: LinkedQL – Live Queries over Postgres, MySQL, MariaDB
Great questions — happy to clarify how deployment and lifecycle work today.

Let me begin by answering: what exactly is this engine? It's simply a computation + cache layer that lives in the same process as the calling code, not a server on its own.

Think of a LinkedQL instance (new PGClient()) and its concept of a "Live Query" engine as simply a query client (e.g. new pg.Client()) with an in-memory compute + cache layer.

---

1. Deployment model (current state)

The Live Query engine runs as part of your application process — the same place you’d normally run a Postgres/MySQL client.

For Postgres, yes: it uses one logical replication slot per LinkedQL engine instance. The live query engine instantiates on top of that slot and uses internal "windows" to dedupe overlapping queries, so 500 queries that are only variations of "SELECT * FROM users" still map to one main window; and 500 of such "windows" still run over the same replication slot.

The concept of query windows and the LinkedQL inheritance model is fully covered here: https://linked-ql.netlify.app/engineering/realtime-engine

---

2. Do all live queries “live” on one machine?

As hinted at above, yes; each LinkedQL instance (new PGClient()) runs on the same machine as the running app (just as you'd have it with new pg.Client()) – and maps to a single Live Query engine under the hood.

  That engine uses a single replication slot. You specify the slot name like:

  new PGClient({ ..., walSlotName: 'custom_slot_name' }); // default is: "linkedql_default_slot" – as per https://linked-ql.netlify.app/docs/setup#postgresql

  A second LinkedQL instance would require another slot name:
  
  new PGClient({ ..., walSlotName: 'custom_slot_name_2' });
We’re working toward multi-instance coordination (multiple engines sharing the same replication stream + load balancing live queries). That’s planned, but not started yet.

---

3. Lifecycle of live queries

The Live Query engine runs on-demand and not indefinitely. It begins to exist when at least one client subscribes ({ live: true }) and effectively cleans up and disappears the moment the last subscriber disconnects (result.abort()). Calling client.disconnect() also ends all subscriptions and does clean up.

---

4. Deployments / code changes

Deploying new code doesn’t require “migrating” live queries.

When you restart the application:

• the Live Query starts on a clean slate with the first subscribing query (client.query('...', { live: true })).

• if you have provided a persistent replication slot name (the default being ephemeral), LinkedQL moves the position to the slot's current position and runs from there.

In other words: nothing persists across deploys; everything starts clean as your app starts.

---

5. Diagram / docs

A deployment diagram is a good idea — I’ll add one to the docs.

---

Well, I hope that helps — and no worries about the questions. This space is hard, and happy to explain anything in more detail.

phrasecode··on Show HN: LinkedQL – Live Queries over Postgres, MySQL, MariaDB
Honestly, that was solely crafted by me. That comment needed to be formatted and properly sectioned.
phrasecode··on Show HN: LinkedQL – Live Queries over Postgres, MySQL, MariaDB
Ah yes — good catch. The commit history definitely isn’t following Conventional Commits right now. Things got a bit loose during fast iterations, but I’ll follow the convention going forward.
phrasecode··on Show HN: LinkedQL – Live Queries over Postgres, MySQL, MariaDB
Thanks for spotting that — to clarify: the current production implementation of Live Queries is Postgres only.

MySQL/MariaDB support is in progress (binlog-based) and is why the docs say “coming soon.”

The post wasn’t meant to imply that MySQL/MariaDB are already live; the intention was to describe the overall design rather than claim full parity. I’ll update the wording to avoid that confusion.

phrasecode··on Show HN: LinkedQL – Live Queries over Postgres, MySQL, MariaDB
Author here. Sad to hear that you perceive the docs and comments here as LLM generated. I'm genuinely curious what in particular gives you that impression.
phrasecode··on Show HN: LinkedQL – Live Queries over Postgres, MySQL, MariaDB
Author here — thanks for checking it out.

Short answer: the core LinkedQL live query engine runs on the backend today, and there’s an embeddable variant (FlashQL) that runs directly in the frontend with the same LinkedQL capabilities – live queries, DeepRefs, etc.

1. Pure frontend / local data

For data that can live entirely on the client, you can spin up an in-browser FlashQL instance:

const client = new FlashQL(); // runs in the page / worker

await client.query(` CREATE TABLE users ( id UUID PRIMARY KEY, name TEXT ) `);

// Live query works the same way as on the backend: const result = await client.query( 'SELECT * FROM users', { live: true } );

From there, result is a live result set: inserts/updates/deletes that match the query will show up in the rows, and all the same features (live queries, DeepRefs, etc.) behave as they do on a backend instance.

At the moment FlashQL is in-memory only; persistence backends like IndexedDB / LocalStorage are on the roadmap.

2. Remote database from the frontend

If your source of truth is a remote Postgres/MySQL instance, the model we’re building is:

a LinkedQL engine next to the database, and

a FlashQL instance in the frontend that federates/syncs with that backend engine.

That federation/sync path is in alpha right now (early docs here: https://linked-ql.netlify.app/flashql/foreign-io ), so today the “stable” story is:

run LinkedQL on the backend against Postgres/MySQL,

expose whatever API you like to the frontend,

and use FlashQL locally where a client-side store makes sense.

The goal is that the frontend doesn’t need a special framework — just a LinkedQL/FlashQL client wherever JavaScript runs.

phrasecode··on Show HN: LinkedQL – Live Queries over Postgres, MySQL, MariaDB
Thanks for reading through and for these questions. I'll take them in their order:

---

Auth / RLS

Yes — LinkedQL works with Postgres Row-Level Security. Each LinkedQL connection is equivalent to a regular DB connection (e.g., new LinkedQLClient(connectionInfo) is like new pg.Client(connectionInfo)). There’s no new permission model to maintain — the DB remains the enforcement point.

Live queries always execute under the same authenticated role you provided, so RLS policies apply on every refresh or incremental update. LinkedQL never uses a “superuser” backend that could widen visibility.

--

Replication limits & scaling

Right now, each database connection supports one logical replication slot. LinkedQL dedupes overlapping live queries on top of it — so 1,000 clients watching the same underlying SELECT only cost the DB one change stream.

We plan to support a distributed architecture as well — multiple instances of the live query engine coordinating load for high-traffic deployments.

---

Why an npm package (and future extension)

Right now LinkedQL plugs directly into JavaScript apps, matching how many teams already query Postgres from frontend or backend code.

We definitely have a Postgres extension in the roadmap for your exact use case – tighter operational integration.

---

Schema migration story

This is also one I’m personally excited about. We previously had an automatic schema versioning layer in the earlier LinkedQL prototype:

https://github.com/linked-db/linked-ql/wiki/Automatic-Schema...

https://github.com/linked-db/linked-ql/wiki/Migrations

The goal in the current version is a cleaner rewrite of that whole feature. So, migration support is returning – with everything we learned in the previous baked in.

For example, while the previous implementation of the diff-based migration feature spoke JSON for schema declarations, we plan to let that be pure SQL – yet, diff-based.

---

Thanks again for the thoughtful look! We can zoom into any other area of your choice.

phrasecode··on Show HN: LinkedQL – Live Queries over Postgres, MySQL, MariaDB
Author here — a bit more detail on architecture and guarantees

Happy to dig into internals if anyone’s curious — how live updates propagate, how JOINs and complex queries resolve, consistency expectations, worst-case scaling, etc.

To keep the main post short, here are deep-dive links if you want to explore:

• Live update mechanics https://linked-ql.netlify.app/capabilities/live-queries

• Engineering paper (replication pipelines, differential projection, query inheritance) https://linked-ql.netlify.app/engineering/realtime-engine

Totally open to questions — I’m hanging around the thread to learn what concerns matter most.

phrasecode··on Show HN: Linked QL – a "magic wand" query client for your database
I'm glad at that. Would you also consider leaving us a star on github?
phrasecode··on Show HN: Linked QL – a "magic wand" query client for your database
Thanks for a particularly exciting feedback. I'm sure you'll have more to say after you've tried this out.

That said, I agree that I haven't done a perfect job there in the sample schema.json area. Nice catch! I'll be change that next.

phrasecode··on OOHtml – Object-Oriented HTML Implementation
I love how we can talk about differences and preferences here like this. Now, I can't wait to put my thoughts further together on the pros and cons of a HTML-first approach like OOHTML's vs a JS-first approach like React's and why there seem to be strong opinions either way. I'm happy to share a link under here when I do put that together.
phrasecode··on OOHtml – Object-Oriented HTML Implementation
Monkey-patching native DOM APIs is really how polyfills are made. This way, people are able to simulate an idea and have a more productive conversation around it. All with a view to having it succeeded by a native implementation.
phrasecode··on OOHtml – Object-Oriented HTML Implementation
LOL
phrasecode··on OOHtml – Object-Oriented HTML Implementation
Thanks for your sharing your thoughts.
phrasecode··on OOHtml – Object-Oriented HTML Implementation
Your feedback really stands out here dude. I'm hoping that the whole conversation here and some of the frustration will lead to something even better for OOHTML.
phrasecode··on OOHtml – Object-Oriented HTML Implementation
Author here. Looks like you missed something about the last update thing. I do actively work on it and ever poised to incorporate feedback.

Don't hesitate to open an issue on project repo when you do use it on a hobby project.

phrasecode··on OOHtml – Object-Oriented HTML Implementation
This is a great feedback and I'm sure that this conversation and some of the frustration around will help us get even better.
phrasecode··on OOHtml – Object-Oriented HTML Implementation
*building a Progressive Web App
phrasecode··on OOHtml – Object-Oriented HTML Implementation
This is a great feedback and I'm sure that this conversation and some of the frustration around will help us get even better.
phrasecode··on OOHtml – Object-Oriented HTML Implementation
Why not we think in terms of "tradeoffs"? You'll find that there a million alternative syntaxes that could work but which would require seeking the right balance. Evan You's on this comes to mind again: https://youtu.be/ANtSWq-zI0s?si=fDbJLfA9T5VLvgsZ.

Meanwhile, I am happy to put up a quick post regarding the thinking over here. I might have done a bit of that in the introductory article: https://dev.to/oxharris/revisiting-the-html-problem-space-an...

phrasecode··on OOHtml – Object-Oriented HTML Implementation
I do understand the frustration with syntax. The good thing is, none of that is finalised and we are in fact seeking people's idea of whats best. But one thing that always holds on this subject is "tradeoffs"! It's an inconvenient word but we wouldn't do anything on the web without it. See also Evan You's talk on seeking that balance: https://youtu.be/ANtSWq-zI0s?si=fDbJLfA9T5VLvgsZ.

I'm happy to incorporate constructive feedback on this project, and it's early age affords us a great opportunity to do just that.

So what's your thought on the challenge?: a native way to bind an element's css properties, class list, attributes, etc. to application data?

phrasecode··on OOHtml – Object-Oriented HTML Implementation
Feels good to know. It really already saves the day for me.

If I get you correctly, is it that the other features like scoped styles and scripts, or html imports, don't look as good?

For data binding, I can see peoples frustration: syntax! And that's to say we've got to have more extensive conversations around here.

phrasecode··on OOHtml – Object-Oriented HTML Implementation
I might have done of lot of hating on DSL just like this. In fact, if you had followed the progress of OOHTML, there was never data binding.

But there's something you find when you follow a practice driven process, and the realisation that brought data binding in OOHTML is one of them.

I'm happy to write a post about this when I have more time. But thanks for your feedback.

phrasecode··on OOHtml – Object-Oriented HTML Implementation
Nice catch. But yes we're currently buying a Progressive Web App right now with OOHTML. I'm happy to share after we've made progress.