5 minutes is the sweet spot for many (enterprise) data pipelines too, in my experience.
2,052 karma · joined June 22, 2025
If you need fresher data, let me know. I will open source the whole pipeline later.
Also, I wanted something that is easy to consume anywhere. With this setup, you can point DuckDB or Polars directly at the data and start querying, no catalog or special tooling required.
The main reason I built this was to have HN data that is easy to query and always up to date, without needing to run your own pipeline first. There are also some interesting ideas in the pipeline, like what I call "auto-heal". Happy to share more if anyone is interested :)
A lot of the choices are trade-offs, as usual with data pipelines. I chose Parquet because it is columnar and compressed, so tools like DuckDB or Polars can read only the columns they need. This matters a lot as the dataset grows. I went with Hugging Face mainly because it is simple and already handles distribution and versioning. I can just push data as commits and get a built-in history without managing extra infrastructure (and, more conveniently, if you read the README, you can query it directly using Python or DuckDB).
The pipeline is incremental. Instead of rebuilding everything, it appends small batches every few minutes using the API. That keeps it fresh while staying cheap to run. The data is also partitioned by time, so queries do not need to scan the entire dataset (and I use very simple tech, just a Go binary running in a "screen" session, using only a few MB of RAM for the whole pipeline).
I built my own pipeline with a slightly different setup. I use Go to download and process the data, and update it every 5 minutes using the HN API, trying to stay within fair use. It is also easy to tweak if someone wants faster or slower updates.
One part I really like is the "dynamic" README on Hugging Face. It is generated automatically by the code and keeps updating as new commits come in, so you can just open it and quickly see the current state.
The code is still a bit messy right now (I open sourced it together with around 3.6M lines across 100+ other tools, hidden in a corner of GitHub, anyone interested can play Sherlock Holmes and find it :) ), but I will clean it up, and open source as clearer new repository and write a proper blog post explaining how it works.
--- You should drive the car to the car wash, but you should walk yourself.
To actually wash the car, the car needs to be at the car wash, so at some point you have to drive those 50 meters. A sensible approach is:
- Drive the car the 50 meters to the wash bay. - Park or queue as required, then get out and do the wash. - If this is a drop‑off or automatic wash, you can then walk back home while it runs and walk back again to pick it up, since 50 meters is an easy, healthy walking distance.
DevSecOps Engineer United States Army Special Operations Command · Full-time
Jun 2022 - Jul 2025 · 3 yrs 2 mos
Honestly, it is a little scary to see someone with a serious DevSecOps background ship an AI project that looks this sloppy and unreviewed. It makes you question how much rigor and code quality made it into their earlier "mission critical" engineering work.
Each chapter focuses on one HTTP concept and shows the same runnable example across net/http, Chi, Gin, Echo, Fiber, and Mizu. The key idea is that the README files are the single source of truth. All example code is generated from them and verified to compile and run (some may contain errors, let me build proper CI/CD later).
It is meant as a practical reference and a way to understand where frameworks differ in responsibility and wiring, not just syntax.
Book view here: https://github.com/go-mizu/go-fw/blob/main/BOOK.md
I've kept running into the same problems in popular Go frameworks: hidden context mutation, magic middleware ordering, reflection-heavy binding, and APIs that slowly drift away from the standard library. The Gin ecosystem in particular has accumulated a lot of technical debt and footguns, which this post summarizes well: https://eblog.fly.dev/ginbad.html
Mizu is deliberately boring by design:
- Built directly on Go 1.22 http.ServeMux
- Explicit middleware chains with clear scoping
- No reflection, no codegen, no global state
- A real request context type that still interoperates with net/http
- First class graceful shutdown and error handling
> For the love of clean code history, let's remember we're not monkeys banging on keyboards; we're educated, civilized human beings.
You rarely replay them to reconstruct business state; you just pump them into analytics or enrichment pipelines.
A single `events` table falls apart as the system grows, and untyped JSONB data in `event_data` column just moves the mess into code. Event payloads drift, handlers fill with branching logic, and replaying or migrating old events becomes slow and risky. The pattern promises clarity but eventually turns into a pile of conditionals trying to decode years of inconsistent data.
A simpler and more resilient approach is using the database features already built for this. Stored procedures can record both business data and audit records in a controlled way. CDC provides a clean stream for the tables that actually need downstream consumers. And even carefully designed triggers give you consistent invariants and auditability without maintaining a separate projection system that can lag or break.
Event sourcing works when the domain truly centers on events, but for most systems these database driven tools stay cleaner, cheaper, and far more predictable over time.
With only half serious intent, I think only the real wizard types, like Jarred Sumner (Bun) and Mitchell Hashimoto (Ghostty), who understand both low level systems and higher level languages, should be writing big tools in Zig. The tough part in the next few years will not be building things, it will be keeping them alive if the authors step away or the ecosystems move in a different direction.
need to bundle a lot of C libraries, some using dynamic linking and some using static linking, and I need to deploy them on different operating systems, including some that are difficult to work with like RHEL. Right now the builds are slow because I use a separate Dockerfile for each platform and then copy the binary back to the host. With Zig CC I could build binaries for different platforms and architectures without using Docker.
Enough rant, now back on some reasons for why choosing Zig:
- Cross platform tools with tiny binaries (Zig's built in cross compilation avoids the complex setup needed with C)
- System utilities or daemons (explicit error handling instead of silent patterns common in C)
- Embedded or bare metal work (predictable rules and fewer footguns than raw C)
- Interfacing with existing C libraries (direct header import without manual binding code)
- Build and deployment tooling (single build system that replaces Make and extra scripts)
His PhD thesis explains the thinking behind Erlang, especially how it handles failures, message passing, and concurrency. It was last updated in 2003, 22 years ago, time really flies!
Sorry for all the negativity. I added some clarifications, but it might have been a bit too late.
Let me clarify my goal here: by writing this article I hope to hear solid counterpoints to the first one, which I didn't find very convincing, and to start a more constructive discussion.
I'm also curious, how hard is it to a new target in Zig? For example, FreeBSD and NetBSD are already supported, but OpenBSD isn't yet. What are the main challenges blocking that?
Some parts of this article were refined with help from LLMs to improve clarity and technical accuracy. These are just personal notes, but I would really appreciate feedback: feel free to share your thoughts, open an issue, or send a pull request!
If you prefer to read only fully human-written articles, feel free to skip this one.