HNHacker News
TopNewBestAskShowJobs

tnelsond4

106 karma · joined January 5, 2026

tnelsond+spam@gmail.com

Github.com/tnelsond

submissionscomments
tnelsond4··on Do you even need a database?
Well, I went and looked at it again and it seems like SQLite databases although larger than my custom format do compress to a 10% smaller size provided there's no indexes. But my custom format uses indexes so it's not really a good comparison. I guess I should see what it's like with indexes.

But at the end of the day SQLite just doesn't meet the needs of my project due to it's size and lack of free decompression support. Passing a decompression stage before giving it to SQLite is unwieldy and requires a separate decompression module and incurs a copy operation. I don't know enough about SQLite to attempt to compile and integrate the zstd decompressor into it.

tnelsond4··on Do you even need a database?
Usually I'm dealing with about 20mb of compressed data, almost 100mb uncompressed. Even with only a couple mb of data SQLite still has a startup time of a couple hundred milliseconds on my phone. But that's a couple hundred milliseconds when loading a database that's already decompressed. When loading 100mb SQLite usually took a second or so which I didn't really like for a pwa.

It took me quite a few attempts to get something faster than SQLite. My new format loads instantly because I'm just casting the data to a struct. The only thing that takes time is decompressing, but that's still faster than loading the uncompressed via SQLite. My phone loads 100mb from 20mb compressed in about 400ms.

But writing my own format gives other benefits like being able to extract all the HTML tags and capital letters beforehand for fast and sensible search and reconstructing it on render. It's also just way easier for me to edit tsvs with markers for what parts are indexed and have that transformed into an indexed format with 3 indexes.

Also, with SQLite I was just running one module, but with my new format I'm running about 20 instances of it because it keeps the data nicer, more manageable and makes everything very parallel. Though I keep the number of web workers to 2 because it doesn't seem to benefit much to increase it more.

https://github.com/tnelsond/peakslab

tnelsond4··on Do you even need a database?
Yeah, I probably should since I want to add glob or regex support later.

It's a little out of the scope of stringzilla but I should still be able to use it for the heavy lifting

tnelsond4··on Do you even need a database?
I love SQLite, I love the idea, I love having something mature and lightweight, but like the author I discovered it's overkill or dare I say insufficient for certain use cases.

I was building a client side dictionary app with search functionality and thought that using sqlite's wasm port would be the perfect solution.

I used SQLite for a couple years and it started to wear on me. The database files were bigger than they should be, they didn't compress super well, the loading time was a bit slow. Linear search wasn't very fast either. Editing SQLite files and doing joins and stuff was slow and frustrating as well. I wanted something simpler.

I didn't need atomics, I didn't need writes, so I just handrolled a set of indexes from the source tsv files and compressed those with zstd and decompressed them via wasm on every load. Decompressing and loading was now faster than straight loading via SQLite and since my module was 52kb of wasm instead of 800kb for SQLite I could load as many instances of the module as I wanted with no regrets. I use stringzilla for linear scans and it's ridiculously fast.

SQLite is great, but it's not the solution for every problem.

tnelsond4··on Comparing Compression Tools
I did some tests for my web app and I came to the conclusion that zstd -19 gave me the best results for small file sizes and fast decompression of text data.

It's amazing that you can include a 50kb zstd decompression wasm module and really speed everything up. In comparison gzip via compression streams were pretty slow and not great compression. I would've gone with brotli decompression streams but those don't seem to be widely supported yet.

If you don't like wasm, there's even an 8kb pure JavaScript decompressor that's a bit slower. But since I like to write in C it's just better all around to write my app in c and have the decompressor compiled directly in.

tnelsond4··on Programming Used to Be Free
Even back in the day you had to buy programming books and courses if you wanted to learn how to make the best code. That wasn't free. It's really not all that different from LLMs, you can code without them, but they're a good resource to help you when you're stuck. There's a billion free LLMs you can use, Grok, duck.ai, etc. you don't need money or a subscription to vibe code.
tnelsond4··on How Poor Am I? – See Where You Stand in Global Wealth
It's fun to play with the numbers though I think these kind of things always rely on the fallacy of scarcity.

Elon Musk's wealth affects me in no way. He didn't get money by stealing from me, he got money by creating things. Likewise we all can create things regardless of Elon Musk. If I have a small farm I can feed my family regardless of the economy and regardless of how much money other people have. Wealth is not a limited resource, it's created through creativity and labor which are infinite.

Though being a wagie, someone who works a wage for someone else is ultimately a losing prospect.

tnelsond4··on 40x Faster Binary Search
Rambus goes VROOM Vroom.

Would all of this still be applicable for string comparisons? Would I have to hash my strings first?

tnelsond4··on Ask HN: Best books on building a programming language
I tried reading the dragon book in my teens, never really understood it, it was too much theory for me.

I'd recommend starting with implementing a forth since it's the easiest language to write an interpreter/compiler for. From there you'll have enough experience to go for something bigger.

Making your own bytecode is really fun.

Ultimately you'll probably want your compiler to target llvm bytecode so that it works on every target automatically.

tnelsond4··on Choosing a Programming Language (2016)
I recommend that everyone learn some "bad" programming languages to unlearn some useless paradigms and hype.

You should learn Java so that you can get sick of OOP and over engineered boilerplate. (I heard they're fixing this though)

You should learn Python so that you can get sick of slow performing code.

You should learn C++ so that you realize you can't keep up with an ever evolving language with centuries of legacy code.

You should learn K&R C or rust so that all you end up doing is fighting the limitations of the language.

Then you learn Forth so that you can spend all your time making your own compiler and interpreter instead of actually writing applications. But you'll be happy and that's what counts.

tnelsond4··on In defense of GitHub's poor uptime
I really don't know what their excuse is. If they were actually developing and improving their platform that'd be one thing, but they still haven't implemented custom headers in GitHub pages or custom mime types. I had to tack a .html extension to my SQLite database in order for them to send it compressed to the client. That's the difference between a 20mb and a 100mb file.
tnelsond4··on "Vibe Coding" is derogatory to the shift towards Natural Language Programming
We can change the terms to be less derogatory, but eventually the term will become derogatory because vibe coding is always going to be looked down on.

It's just like our nice terms for mentally handicapped people, eventually all the terms we use for them become offensive because of their association with mentally handicapped people. That's how language works, you can't change the stigmas by changing the language.

tnelsond4··on The Reason People Aren't Having Kids
In Cambodia they used to have 6-10 kids each, then 3-5, and now every Cambodian just wants 2 children, 1 boy, 1 girl. The reasons? Mostly economic I guess. Even though school is free you have to have enough money to buy your kids junk food to eat while they're there.

On top of that most people are in debt at high interest rates because they built a fancy concrete house then ran away to Thailand to try to make money to pay off that debt. Their children are then in the care of the grandparents who oft times are very old.

It used to be that having a lot of kids was insurance that when you got old someone would be around to take care of you, but often I see these old people stressed and taking care of their grandchildren and worrying about their children's debt instead of being cared after.

tnelsond4··on "It's rare that developers love their database that much"
Naw, developers will always be enthusiastic about stuff even if it's crap. I used to really like SQLite and think that I should use it for everything, but after using it for a couple years I realized it's just not the right tool for most things. It doesn't help that I usually mess up my joins and lead to undetected data loss.

I really wanted to use SQLite client side for the data in my dictionary web app, but I ended up fighting the structure too much. Ultimately I wrote my own binary format with indexes from scratch that work how I like them.

I've never used Postgres though.

I've been enthusiastic about a lot of things, C, libsdl, Linux, ffmpeg, vim, and as a developer this obsession is the norm rather than the exception.

> Coding is the closest thing humans have to magic. -Luke Smith

tnelsond4··on A compelling title that is cryptic enough to get you to take action on it
Comment that starts a completely unrelated rabbit trail.
tnelsond4··on Safer Casting in C – With Zero Runtime Cost – Making Casts Visible, Auditable
GCC -Wall what tells me about most implicit casts going on (which I mostly ignore). But not all of them.

I think these cast problems are ultimately something that the compiler can and should catch. And a stronger type enforcement system would be nice, but also I use C so i don't have to deal with rust's nonsense

tnelsond4··on What do you think will be the next social media?
It'll be some platform that hooks people by not having ads, then once it reaches critical mass your feed will be nothing but sponsored posts.
tnelsond4··on Show HN: Hindsight Simulator – Go back in time and get rich
Very fun. Almost as much fun as the actual stock market.
tnelsond4··on Show HN: Server-rendered multiplayer games with Lua (no client code)
Since you're doing this is rust, try experimenting to see what would happen if you did zstd compression using a dictionary on the data you're sending back and forth, it might give you a performance benefit.
← PreviousPage 4 of 4