eatonphil

Yeah I do have the JavaScript side in place, was just loathe to start totally owning this data instead of Fathom.

I know you can't be super specific because you'd just be enabling the companies who get around this (who, funnily, I also index on The Consensus), but I have spent a few rounds on this over the months and haven't yet been able to cut them out.

Some of the things that seemed like they'd cut out lazier bots/scripters involved browser fingerprinting that I didn't want to yet have to deal with disclosures or approval for. But also even some of these browser fingerprinting things were disable-able via flags when you start up Chrome or whatever.

I think what would work overall is to track all requests for a pattern of page access and start blocking other systems that are part of that pattern. But that also wouldn't account for like... landing on the front page of HN, which happens not infrequently.

All of this is aside from per-IP or per-user proof of work. Whenever someone lands on a more expensive page I make the browser do proof of work and store it in a cookie, unique to the IP or user. (Signed in users don't have to do proof of work). This is a perfect solution except for the IP rotation I'm talking about above.

edit: ah also, I don't do SSL termination so can't spot the SSL configuration. But Cloudflare does. And all of this is even after Cloudflare paid bot traffic systems are in place.

edit 2:

> tiny percentage of bots will still get through but the load becomes much more manageable.

The load is not the problem. It's only at most a few bot hits per second. I'm just having trouble removing bot attribution from view counts.

I get tens of thousands of IPs making 1 or 2 requests against pages on The Consensus (job search, code viewing, etc.). They're not super expensive pages. But I don't want to deal with the bot traffic (mostly because it messes up view counts I use to sell sponsorships). But then if I require an account (even a free one), valid readership is also going to drop with the new friction.

I could build something more complex to discover the patterns, or use Turnstile maybe. I'm not really sure at the moment.

Like West Wing. Makes it hard to watch. TNG a bit less so, thankfully.

No allocator available isn't the same as being able to enforce no allocations.

I agree this isn't the absolute ideal form. I'd rather it be built in as a language directive or something. noalloc does parse compiler escapes emitted by the Go compiler with `-gcflags=-m -S`. One problem is that these are not really a standard format. But I hope that with testing across versions (over time anyway, I only run this on Go 1.27) it'll be stable enough.

Hey folks, this is my project. I really liked OxCaml's zero_alloc feature [0], something you don't get in Rust or Zig or Go*, and have been trying to get a handle on allocations in The Consensus's code search engine.

It's an extremely early project but it's been helping me out a bunch already. Would love to hear your bug reports or requests if you try running it on your codebase.

[0] https://theconsensus.dev/p/2026/06/27/the-feature-in-oxcaml-...

* I think you can get this in Clang and maybe GCC with `__attribute__((nonallocating))` though.

> For real judgements on database products, I'd look to https://jepsen.io/. That author tends to be refreshingly straightforward in a field where many vendors love to cheat by different names

As far as I'm aware, Jepsen solely focuses on correctness (which is more than enough for someone to do!), not performance.

I was not precise there sorry. They were not crashes but Lightpanda failed to render them. It was these URLs

https://duckdb.org/2025/07/04/ducklake-02.html

https://ducklabs.com/news/2024/06/03/duckdb-version-100-rele...

https://neovim.io/doc/user/dev_arch

https://www.twelvelabs.io/blog/%EB%B9%84%EB%94%94%EC%98%A4%E...

I have a research folder of some findings comparing Chrome, headless-shell, lightpanda, jsdom, etc. Lightpanda still did quite well, though with much smaller multiples compared once you've mildly tuned Chrome/headless-shell, but it crashed more often than Chrome and headless-shell.

I haven't yet had time to write it up on The Consensus. So here are some findings you can take or leave.

  Configuration                       CPU s  Mem MiB  Median  Matched  Errors
                                      /page   (peak)    page  (of 40)
  Chrome, default flags                2.31    1,229    4.4s      ref       0
  Chrome, tuned                        1.98    1,244    3.1s       40       0
  Chrome, tuned + block all trackers   1.31    1,338    2.2s       37       0
  headless-shell, today's settings     2.30      748    3.9s       39       0
    + stop at network idle             2.16      772    2.6s       39       0
    + block ads/analytics              1.79      748    2.1s       39       0
  headless-shell, tuned                1.67      723    2.1s       39       0
  headless-shell, tuned + no frames    1.29      696    1.9s       39       0
  headless-shell via Playwright        2.58      805    4.2s       38       0
  Firefox via Playwright               6.94    2,270    5.6s       38       0
  WebKit via Playwright                4.76    4,125    5.2s       36       0
  Lightpanda, default                  1.31      540    3.6s       35       4
  Lightpanda, tuned                    1.04      509    1.3s       35       4
  jsdom                                2.66    2,102    4.3s        4       8
  Plain HTTP fetch, no JS             0.002       97   0.05s        0       0

hahaha that is a good one.

For sure, there are ~23,000 articles in the feed (which only goes back 5 years, but grows naturally as people publish and as we index more companies in software infrastructure).

The point is to index interesting jobs and articles to 1) help people follow the technologies they're interested in and 2) to help us figure out what to write about.

Articles get tagged with mentioned infrastructure technologies (e.g. Postgres [0] or Lean [1]) and get tagged semantic categories too (e.g. compilers [2] or network protocols [3]). (And all the same tagging happens for jobs as well.) The weekly newsletter [4] features articles and jobs from a few key semantic categories. But you can also build your own RSS feeds or email watchlists for whatever you're interested in.

There's a lot of filtering done to keep out marketing content, listicles, etc. And checking AI tells is the next stage of that.

Limiting to only good quality articles lets us build broader intelligence like the ranking of programming languages [5] and databases [6] and helps us figure out which companies are most focused on the technologies and so on.

[0] https://theconsensus.dev/feed.html?q=projects%3DPostgreSQL

[1] https://theconsensus.dev/feed.html?q=projects%3DLean

[2] https://theconsensus.dev/feed.html?q=topics%3D%22Compilers+%...

[3] https://theconsensus.dev/feed.html?q=topics%3D%22Network+pro...

[4] https://theconsensus.dev/weekly.html

[5] https://theconsensus.dev/languages.html

[6] https://theconsensus.dev/databases.html

> This is the opposite of how most places use the sparkle icon!

I heard this from someone else today too. I'll have to think of a different icon perhaps then.

> It was relaxing to see some minor typos and stylistic awkwardness in this article, I have to be honest.

whoops

That's right.

Synchronous replication to a quorum is a lot more expensive than an fsync.

There'd be no benefit. They're good general purpose databases but for any specific workload you can do significantly better on your own. For search in particular even if you use Postgres you wouldn't want to use Postgres FTS you'd want to at least use Tiger Data's extension.

I have seen this happening already at two different companies. And I'm also doing it as well. Particularly for search indexes where there's no risk of data loss.

And some additional context from a few months ago, building off the postmortem.

https://theconsensus.dev/p/2026/04/16/who-even-uses-jemalloc...

> I'm only sad that I didn't put SQLite under test earlier in the year, or I would have found this issue right away

Unless you guys routinely add assertions to libraries I don't see how you would have caught this?

I mean for sure databases should guard against lost writes but how would you have known to add that particular assert?

I did a review of Mojo a few months ago focused on Python compatibility if anyone's curious about that angle. Admittedly it is a narrow angle. I want to take another look at Mojo sometime not focused on Python.

https://theconsensus.dev/p/2026/03/12/mojos-not-yet-python.h...

> The "fan-out" is synchronized with a classic consensus algorithm called 3PC (three-phase commit) so that a push is only accepted if a majority of the nodes acknowledge it.

Doesn't 3PC require all nodes to agree, not just a majority?

If TigerBeetle is the Zig flagship then Zig is untenable for almost any team, because Tiger Style is untenable for almost any team. No, I think that Tiger Style makes Zig mostly unimportant.

On the other hand, as a corporate backer for Zig, TigerBeetle is definitely a big deal.

But in terms of exemplary and imitable projects in the ecosystem I would think something like Ghostty is the safer label for "flagship".

They are indeed different. In Zig you might get the allocator via a struct field or just via importing the global allocator.

  const std = @import("std");

  fn makeBuffer(n: usize) ![]u8 {
      return std.heap.page_allocator.alloc(u8, n);
  }

Here are a few examples from the last release of Bun before the rewrite into Rust.

https://github.com/oven-sh/bun/blob/bun-v1.3.14/src/install/...

https://github.com/oven-sh/bun/blob/bun-v1.3.14/src/shell_pa...

The allocator via function parameter is only a convention and only even one convention. It is certainly a strong convention (throughout the standard library anyway) but it is just a convention.

Recent Rust also has the ability to pass allocators but that's the same thing. And even if you use no_std in Rust you might call into some other library's no_std function that allocates and you wouldn't be able to grep for that.

Supabase customers run it (marked alpha).

https://supabase.com/blog/orioledb-launch

And they continue to work on it.

https://supabase.com/blog/orioledb-patent-free

Snowflake acquired Crunchy Data yes and Craig was at Crunchy Data.

Pangram says the text is entirely AI generated but I don't know how trustworthy Pangram is. (I would love to hear what others think about it.)

Thank you! I've made an edit crediting you. This also led me to notice the nonallocating attribute in Clang, which I've now mentioned as well.

I was also always curious about this and recently wrote a few implementations of an http file server to teach myself the key differences.

https://theconsensus.dev/p/2026/05/18/serving-files-three-wa...

It's nice to have some proof to show people.

Tart was just bought by OpenAI so its future is suspect.