epicepicurean

Much better than Opus 5. prompt:

> hi, can you explain how the scheduler works. keep it brief, but include important correctness details

some excerpts:

>Flow: 1. Data arrives. The appender calls prepare/commit around the krolik write. The prepare step leaves a phantom row, so a write that is still in flight is tracked. The commit publishes STREAM_INPUT. 2. Poll. It first reclaims expired leases. Then it sweeps sdirty (a candidate index) in a rotating order, oldest since first, which prevents starvation. It re-derives the exact period and rewind for each candidate from snode_out/snode_in, then claims it: writes the sassign lease plus a per-edge sassign_edge snapshot. 3. Complete. Each edge's watermark is set to the snapshot taken at dispatch. Anything published after that gen stays dirty automatically. A failed or partial job writes nothing, so its dirt persists.

> - Folds are monotone single statements. gen only goes up, extents only grow, processed periods only union, rw_start only moves earlier. So replays, out-of-order writes, and concurrent writes can at worst hold a watermark back. The worst outcome is a spurious rerun, never lost work. There's no read-modify-write and no truncation of the dirty period, so a write that lands during a job can't be swallowed.

> - Rewrites are declared by the publisher, never inferred from overlap. An unflagged write that overlaps already-processed data means "unchanged", which makes retries and redelivered tasks free. Each publisher has its own reason for being sound (see the table in scheduler/CLAUDE.md). A new publisher needs a row in that table.

> - NULL means dirty, and DELETE is the fence. Every node and edge has a row from the moment it's created. A lost parent or a settings-only edit can't be derived, so both go through one forced-rerun path: capture_rewinds reads the processed span before the DELETE, and apply_rewinds publishes it as a rewrite on a config root.

All the non-standard programming jargon is stuff from the repo. I can actually read it and understand what it's talking about. I used Fable to handle Opus 5 as I just couldn't stand it. With this I'll probably go back to Opus.

They are not human, but it helps to prompt them similarly. See: https://www.anthropic.com/research/emotion-concepts-function

any explanation of why the context window is only 272K?

When I say crash tolerance, I mean the entire system going down. Given the emphasis on async BEAM processes, which all work in memory, I find it hard to understand why they're more reliable than the "standard" approaches of SQL dbs or crash-tolerant queues like kafka.

Take this example from the article:

  def handle_call({:process, order}, _from, state) do
    customer = Customers.fetch!(order.customer_id)
    charge = PaymentGateway.charge!(customer, order.total)
    Notifications.send_confirmation!(customer, charge)
    {:reply, :ok, state}
  end

I'd assume we want PaymentGateway to commit to a DB. But there's no transactionality with notifications, hence notifications can be lost if the entire runtime goes down. For an article trying to "sell" BEAM to me, I just don't see the value.

> I guess maybe the question is why use BEAM if it also doesn't solve the general systems are problem?

I interpreted the tone of the article to mean it does solve all these problems. Resulting in my general confusion as to the actual advantages. I think this whole actor business somewhat reminds me of the Smalltalk people saying it's all about message passing, but I just don't understand what's the difference between passing a message to and object, and doing obj.function(message). At least for BEAM the whole supervisor tree seems neat, but other than that, it sounds like go routines with channels, or just a queue in python.

A rewrite of a stateful application written in python with postgres would be more illustrative of how you're solving the same problems but better. Do BEAM applications not use an actual databse? How is crash tolerance guaranteed? In a typical application I'd write crash tolerance would be handled by the DB. So would transactionality. Without it, one would have to persist each message to disk and be forced to make every action idempotent. The former sounds like a lot of performance overhead, the latter like a lot of programming effort overhead. I assume these problems are solved, but the article doesn't demonstrate the solutions.