1,783 karma · joined April 5, 2011
I've generally liked BigQuery for this type of stuff - the console interface is good enough for ad-hoc stuff, you can connect a plethora of other tooling to it (Metabase, Tableau, etc). And if partitioned correctly, it shouldn't be too expensive - add in rollup tables if that becomes a problem.
Do you do extract, fork process, cleanup? Does it handle signals sent to the extract process as if they were sent to the actual program?
Roughly follows the semantics of beanstalkd which I used once upon a time and quite liked.
I feel like uuid / ulid are just overkill for most situations - and they're long and kinda ugly imo.
I've been using my own homegrown project that does just this - Python roles, server/client, Mako templates: https://github.com/mattbillenstein/salty
It's very very fast to do deploys on long-lived infrastructure, but it hasn't been optimized for large clusters yet; I expect the server process will be a bottleneck with many clients, but still probably faster than Ansible for most setups.
https://vazor.com/drop/mulman.jpg https://vazor.com/drop/bearcat.jpg
Still using an iPhone though - it is a bit crazy how expensive these have gotten and how repairs can be so expensive.
I'm mostly using Ubuntu on laptops, but there are small annoyances usually, things don't work as well out of the box given the variety of hardware out there.
I think GA just doesn't have enough features for fine-grained tracking - still useful if you're participating in their ad ecosystem.
I've solved a lot of the issues he brings up - gevent, no orm - well-known transaction boundaries, etc. It's pretty pleasant to work in.
I have been impressed that they're making progress though - wheels solve some problems conda used to solve; so doing ML stuff that is based on conda I can usually just use regular pip packages now which is very nice.
I've been using pip-tools for some time, I think it solves the other 20% of the problem for me in a simple way that I like. Poetry et al seem to be trying to do too much - ymmv.
The iterations on packaging that don't really seem to ever get it right are, I think, frustrating to the community where the core likes to advertise a "Zen of Python" "one way to do things" mantra, but can never really get 100% of the packaging problem sorted out in a clean way in spite of several communities seeming to figure it out.
I think maybe the problem is that the world needs soooooo much software and the bar just seems to be getting lower in terms of the quality of stuff people will ship and expect to get paid for.
What this means for water-rights downstream, I have no idea - probably not good.
I've been using this little bash script for awhile and decided to throw it up on github - incredibly handy to have a bare ubuntu server install in 30s whenever I need.
My intuition is it's probably better to just over-deploy wind and other renewables - pumps and turbines are not exactly free or easy to maintain either.
I've been doing LEPP stack (Linux / Nginx, Python, Postgres) stuff for awhile now without an ORM; which makes you really consider how you're handling transactions ahead of time. So requests have become idempotent and transactional - the default of many frameworks to run in auto-commit really defeats the purpose of having a good transactional database imo - ymmv.
Also re concurrency, I've been using gevent for a long time - it's a nice framework and lets you write blocking code while being async behind the scenes. And it's nice for writing little backend http or tcp services. But, I think for what I do I could mostly just use OS threads - thread per request in gunicorn - and that'd work just fine. Request context is stored in thread-locals anyway, so there isn't really shared state across requests to worry about the GIL much. And I'd run a process per vcore anyway behind gunicorn.
So a couple years ago, I downsized to a single 2TB SSD in a smallish ATX case - and another in a completely different machine that gets rsync'd every 4 hours. My nas is now just my last desktop's hardware with two SSDs - one boot, one larger storage running plex, smbd, backups of misc stuff on cron, duplicity backup to the cloud, etc on Ubuntu. If I didn't have this extra hardware, I'd probably just run two medium powered NUCs with a nvme boot disk and a bigger SSD for storage.
It's all very very simple and I have it setup so I can run some LXC containers should I want to do some dev work there, but I usually have other hardware for that anyway.
ORMs in general I think are not a great abstraction - you get lazy-loading patterns, performance problems due to this which leads to manually doing eager-loading which leads to even more twisted up queries being sent to the db and hard to debug problems. And most of them rely on running in auto-commit, so you can't have idempotent transactions. YMMV
Generally, I've been writing very thin CRUD db-access modules that use SQL and I run every request in a single transaction that either commits or does a rollback on error. I don't do eagerloading or anything fancy, but if I need to get a list of related items, I just do a bulk query. So there are some small N number of queries per request, but still generally things are fast. I find this type of system easier to debug and optimize - if I see a slow query in the db log, I can just grep my codebase for it which is pretty much impossible with an ORM.
Re migrations, I've used Flyway before, I like the model, but lately I've been using a single python script which more or less does what Flyway does [1]. This supports migrations in SQL or Python, but I almost exclusively just use SQL now as Python migrations can have extra code dependencies not captured in the actual script. Salt to taste.
I think if you do decide to use an ORM, revisit this thread in a few years time and think about if it's saving you time in the long-run. I find the time it saves you in the prototype you end up paying back in spades in maintenance and building new features - again, YMMV.
1. https://gist.github.com/mattbillenstein/270a4d44cbdcb181ac2e...