HNHacker News
TopNewBestAskShowJobs

mattbillenstein

1,783 karma · joined April 5, 2011

matt (at) vazor (dot) com
submissionscomments
mattbillenstein··on The Architecture Behind a One-Person Tech Startup (2021)
Guy lost me at k8s and terraform - just put your thing on a single VM and yolo.
mattbillenstein··on Big data is dead (2023)
I can appreciate the vertical scaling solution, but to be honest, this is the wrong solution for almost all use cases - consumers of the data don't want awk, and even if they did, spooling over 6TB for every kinda of query without partitioning or column storage is gonna be slow on a single cpu - always.

I've generally liked BigQuery for this type of stuff - the console interface is good enough for ad-hoc stuff, you can connect a plethora of other tooling to it (Metabase, Tableau, etc). And if partitioned correctly, it shouldn't be too expensive - add in rollup tables if that becomes a problem.

mattbillenstein··on Replacing pyinstaller with 100 lines of code
Nuitka does something like this too, but in my experience does not handle signals as well as PyInstaller.

Do you do extract, fork process, cleanup? Does it handle signals sent to the extract process as if they were sent to the actual program?

mattbillenstein··on Show HN: An SQS Alternative on Postgres
Ha, I wrote one too - https://github.com/mattbillenstein/pg-queue/blob/main/pg-que...

Roughly follows the semantics of beanstalkd which I used once upon a time and quite liked.

mattbillenstein··on Why Choose ULIDs over Traditional UUIDs or IDs for Database Identification?
The encoding is completely different? And given that encoding I don't go to any great means to store them as a special type in the db - just text. uuid being hexidecimal, databases often want to store them as binary since that's half the bytes...
mattbillenstein··on Why Choose ULIDs over Traditional UUIDs or IDs for Database Identification?
I've been doing something that includes a type character as the first character - followed by 11 random base58 digits. This allows > 2^64 possible ids and a type in ~12 bytes. I occasionally wish they were ordered, but most of my tables usually have a 'created_at' field anyway should ordering matter.

I feel like uuid / ulid are just overkill for most situations - and they're long and kinda ugly imo.

mattbillenstein··on Pyinfra: Automate Infrastructure Using Python
Yeah, I'm talking more about RPC - the server sends a command to the agent - the agent does a thing and returns a response. There's no external sync of the command and given a long-lived connection - client/server what you will - this can all be completed in milliseconds with no new-connection overhead.
mattbillenstein··on Pyinfra: Automate Infrastructure Using Python
Was there any thought to perhaps do a version with an agent? I really like how fast Saltstack can be as compared to Ansible.

I've been using my own homegrown project that does just this - Python roles, server/client, Mako templates: https://github.com/mattbillenstein/salty

It's very very fast to do deploys on long-lived infrastructure, but it hasn't been optimized for large clusters yet; I expect the server process will be a bottleneck with many clients, but still probably faster than Ansible for most setups.

mattbillenstein··on Tesla driver arrested for homicide after running over motorcyclist on Autopilot
It makes a lot of sense if you're trying to churn out more profit per unit - less costs - they're at the mercy of a sour market atm on the other side of it.
mattbillenstein··on Tesla driver arrested for homicide after running over motorcyclist on Autopilot
Elon's refusal to adopt lidar - statistically probably fine, anecdotally it could turn out very badly for any one person which is a hard thing to swallow if it's you...
mattbillenstein··on Uncovering the tiny doodles left by engineers on old microchips
Always fun to see these - I did one for a class project in college we actually had fabricated - a few thousand gate 16bit multiplier in just a couple layers of metal; I believe 1.2um process.

https://vazor.com/drop/mulman.jpg https://vazor.com/drop/bearcat.jpg

mattbillenstein··on Oregon passes right-to-repair law Apple lobbied to kill
Recently bought a Framework laptop - their mission is easily repairable diy hardware with good software (Linux!) support.

Still using an iPhone though - it is a bit crazy how expensive these have gotten and how repairs can be so expensive.

mattbillenstein··on Ask HN: Why macOS?
It's probably still the best complete system - hardware + software that's fundamentally unix under the hood, but the user-space tools seem bad compared to what you get on any modern GNU/Linux system.

I'm mostly using Ubuntu on laptops, but there are small annoyances usually, things don't work as well out of the box given the variety of hardware out there.

mattbillenstein··on Ask HN: Google Analytics sucks What do you use?
What sort of abstract reports?

I think GA just doesn't have enough features for fine-grained tracking - still useful if you're participating in their ad ecosystem.

mattbillenstein··on Ask HN: Google Analytics sucks What do you use?
Small startup, but I find https://getanalytics.io/ + Google BigQuery + Metabase a nice all-around solution.
mattbillenstein··on In Defense of Simple Architectures (2022)
LEPP stack is very effective imho - Linux, Nginx, Python, Postgres.

I've solved a lot of the issues he brings up - gevent, no orm - well-known transaction boundaries, etc. It's pretty pleasant to work in.

mattbillenstein··on Postgres as queue
I wrote one of these not long ago: https://github.com/mattbillenstein/pg-queue
mattbillenstein··on Boring Python: dependency management (2022)
I kinda see myself wanting the pip-tools flow, requirements.in -> pip compile -> requirements.txt thing perhaps integrated into pip itself. Seems like the simplest of the solutions out there for good dependency declarations and pinning? I may well be wrong.
mattbillenstein··on Boring Python: dependency management (2022)
Yeah, this too.

I have been impressed that they're making progress though - wheels solve some problems conda used to solve; so doing ML stuff that is based on conda I can usually just use regular pip packages now which is very nice.

mattbillenstein··on Boring Python: dependency management (2022)
The python core team endorses pip, but pip solves something like 80% of the problem - we need them to endorse the other 20% of the solution.

I've been using pip-tools for some time, I think it solves the other 20% of the problem for me in a simple way that I like. Poetry et al seem to be trying to do too much - ymmv.

The iterations on packaging that don't really seem to ever get it right are, I think, frustrating to the community where the core likes to advertise a "Zen of Python" "one way to do things" mantra, but can never really get 100% of the packaging problem sorted out in a clean way in spite of several communities seeming to figure it out.

mattbillenstein··on Ted Cruz wants lawmakers to get security escorts through America's airports
It's just soooo cynical that the guy stoking the crazies and trying to burn it all down wants personal protection. Absolute fucking douchlord.
mattbillenstein··on Jonathan Blow on the lack of deep knowledge amongst modern software engineers [video]
I like Jon's takes and thoughts generally speaking, but what he's pointing out here is just always true in every discipline. There are experts who have deep knowledge who do the hard stuff and there are practitioners who don't who do the easy stuff.

I think maybe the problem is that the world needs soooooo much software and the bar just seems to be getting lower in terms of the quality of stuff people will ship and expect to get paid for.

mattbillenstein··on Pumped hydro is the only real gravity storage solution
Not an expert, but I have read somewhere probably the most effective use of this is to pump water back up behind dams - imagine you could pump water from the bottom of hoover dam using wind power or whatever. All the generation equipment is already there, so you sorta get that for free. Additionally with lake mead shrinking this might enable more hydro power from existing infrastructure even in the face of climate change.

What this means for water-rights downstream, I have no idea - probably not good.

mattbillenstein··on The unreasonable effectiveness of lxc containers
Does anyone else use lxc? With networking bridged to my local network, I find it a great way to spin up small and disposable dev environments for one-off things and testing.

I've been using this little bash script for awhile and decided to throw it up on github - incredibly handy to have a bare ubuntu server install in 30s whenever I need.

mattbillenstein··on Pumped hydro is the only real gravity storage solution
I've only read a little about this, but from what I understand, it's hard to find places that are actually good for this - and the volume of water you need to produce a significant amount of energy is quite large for given place where you can get enough height.

My intuition is it's probably better to just over-deploy wind and other renewables - pumps and turbines are not exactly free or easy to maintain either.

mattbillenstein··on Using GNU Stow to manage your dotfiles (2012)
No need to get complicated or have dependencies - I do this with a git repo checked out and a bash script that creates all the symlinks in $HOME/
mattbillenstein··on In Defense of Simple Architectures
Good stuff - too many people focus on the wizz-bang of new tech they forget how effective a well-designed monolith can be across a lot of very important dimensions.

I've been doing LEPP stack (Linux / Nginx, Python, Postgres) stuff for awhile now without an ORM; which makes you really consider how you're handling transactions ahead of time. So requests have become idempotent and transactional - the default of many frameworks to run in auto-commit really defeats the purpose of having a good transactional database imo - ymmv.

Also re concurrency, I've been using gevent for a long time - it's a nice framework and lets you write blocking code while being async behind the scenes. And it's nice for writing little backend http or tcp services. But, I think for what I do I could mostly just use OS threads - thread per request in gunicorn - and that'd work just fine. Request context is stored in thread-locals anyway, so there isn't really shared state across requests to worry about the GIL much. And I'd run a process per vcore anyway behind gunicorn.

mattbillenstein··on [dead]
Let it burn - the whole premise of the purchase was wrong.
mattbillenstein··on 7 watts idle – building a low powered server/NAS on Intel 12th/13th gen
Everyone's needs are different, but over time with the loss of a couple of disks, I started to hate running RAID5 or 6 with HDDs. It became an exercise in how fast could I replace a disk before the next one died and would the rebuild actually work or not - although, it always did. Also, the hot-swap case/cage I had with all the SATA connectors and power connectors seemed kinda flaky - it was very cheap.

So a couple years ago, I downsized to a single 2TB SSD in a smallish ATX case - and another in a completely different machine that gets rsync'd every 4 hours. My nas is now just my last desktop's hardware with two SSDs - one boot, one larger storage running plex, smbd, backups of misc stuff on cron, duplicity backup to the cloud, etc on Ubuntu. If I didn't have this extra hardware, I'd probably just run two medium powered NUCs with a nvme boot disk and a bigger SSD for storage.

It's all very very simple and I have it setup so I can run some LXC containers should I want to do some dev work there, but I usually have other hardware for that anyway.

mattbillenstein··on Any Python ORMs worthy of production?
An ORM in the style of Django leads to patterns where you have a lot of code on the models - and the business logic becomes very tightly intertwined with the DB schema such that it's very hard to ever refactor it. Also, God objects, circular dependencies, etc.

ORMs in general I think are not a great abstraction - you get lazy-loading patterns, performance problems due to this which leads to manually doing eager-loading which leads to even more twisted up queries being sent to the db and hard to debug problems. And most of them rely on running in auto-commit, so you can't have idempotent transactions. YMMV

Generally, I've been writing very thin CRUD db-access modules that use SQL and I run every request in a single transaction that either commits or does a rollback on error. I don't do eagerloading or anything fancy, but if I need to get a list of related items, I just do a bulk query. So there are some small N number of queries per request, but still generally things are fast. I find this type of system easier to debug and optimize - if I see a slow query in the db log, I can just grep my codebase for it which is pretty much impossible with an ORM.

Re migrations, I've used Flyway before, I like the model, but lately I've been using a single python script which more or less does what Flyway does [1]. This supports migrations in SQL or Python, but I almost exclusively just use SQL now as Python migrations can have extra code dependencies not captured in the actual script. Salt to taste.

I think if you do decide to use an ORM, revisit this thread in a few years time and think about if it's saving you time in the long-run. I find the time it saves you in the prototype you end up paying back in spades in maintenance and building new features - again, YMMV.

1. https://gist.github.com/mattbillenstein/270a4d44cbdcb181ac2e...

← PreviousPage 5 of 31Next →