451 karma · joined October 9, 2014
There is tons of great gc research/experiments/learnings out there that can have real benefits, maybe the JVM with its zillions of knobs is a step too far but find a middle ground seems like it would help people.
Is an infinite loop is the line? Updating a package to display random text without an infinite loop is crossing the line too? Who decides what _my_ OSS library is allowed to do? Can I update it just to add a banner asking for donations? How is that functionally different to random text other than intent?
https://www.2ndquadrant.com/en/blog/on-rocks-and-sand/
The linting story for postgres schemas isn't as good as it is in golang tho, so harder to automatically detect/solve for.
The classic problem with letting your ORM manage migration is if you are updating your database entity model as part of a wider change, in your ORM and you rename a column, does it know its a rename? or is it going to DROP COLUMN + ADD COLUMN? and lead to data-loss? To cover these cases and all the infinite other permutation you end up needing to give the ORM more and more state about the past, making it just a more complex migration. Making the migration explicit SQL and decoupled from the ORM forces thought, a human is unlikely to write DROP COLUMN..
.. on and on, 40MB..30MB..20MB.. hundreds of them just adds up! 147 dependencies over 1M. pretty incredible/wasteful really when you look at it.
A Git bot that parses changes to yarn lock and comments with size/loc/files of all the deps etc, kinda like how coverage bots work. get all the observability benefits without the checkin cost, also wouldn’t have to split npm changed and code changes..
Also image pull performance is generally pretty poor from most registries, you will get much higher speeds out of a straight s3 download then pulling an image via docker or containerd, even tho ecr for ex is backed by s3 .
I haven’t actually used ootb gitlab caching in quite a long time because of these limitations, but would be nice for it to work great ootb!
The default way to cache things is zip, which no matter the compression isn't a great fit, you really want a format that is stream-able like tar. When its stream-able you can download the tar and unpack it at the same time, not waiting for the full download before beginning to unpack. Obviously you would compress the tar (normally) with lz4 or zstd or similar, using this approach you should generally see a reduction in total time to get and unpack the cache. node_modules even more so since it generally has zillions of small files, so the unpack time can be quite high.
Be nice if gitlab supported this approach OOTB.
Edit: An even better way if your environment supports it is to use a mountable image (esp for node_modules) for your caches, this basically removes the entire unpack phase of making the cache available, instead of unpacking you just mount the image and off you go.
In macOS this looks like a sparseimage via attached hdituil, or in linux a squashfs image with a writable overlay mount (you need a privileged container for this if in a container). Since the OOTB gitlab.com runners are a root-user linux VM (i think?) this approach should work quite well. SquashFS images are a great fit for node_modules especially as it moves the creation of the zillions of tiny files to cache-creation time rather than cache-use time. If you share the cache images via a hostPath mount or similar for existing images you can effectively make caches available in 0 seconds (just mount an already downloaded image and done).
and then almost immediately went in 5 different directions around how to program around it.. callback hell, async/await, generators, promises etc..
This change would make the base primitives non blocking (really cheap Blocking which is better) by default, so now you can use any~ library and it should just work. Waaay better.
The idea of treating db nodes less like pets seems great, there must be a few tricks to it tho..
I guess there are people today who don’t fit that mold and approachability becomes important, shouldn’t be surprising.
- JS dependencies are a problem because of the sheer number, which is really because of all the authors of the dependencies and how they update, backwards compat or not etc. if you had 10 big things by 10 authors they would coordinate better than 1500.
- The Lazarus example is interesting but “trusted components” also means walled gardens, who gets to choose? Is there competition? How is that decided in vs out? Logical conclusion to much of this ends up with a central community package manager like maven central..
my progression was trying to find a soln for immutable key/values (immutable enabling some different tradeoffs then others might have) where most of my values are bigints so only a couple of bytes to store.
I worked through approaches: 1. obvious skinny table with duplication 2. EAV optimisation to reduce duplication
The problem with both of these is that having 1 key/value per row with a small value (8 bytes~) meant 23-27 byte row overhead, which obviously increases your storage cost dramatically.
So i looked at how to have more values on each row, so maybe 23-27 bytes overhead for 500 values..
3. EAV with values grouped into an array per row, similar to EAV but instead of 1 value per row, map N values onto 1 row using an array column. So N values are all in one array on a row, and accessed by index. This has the lowest storage overhead, because of row compression you can get on average an 8 byte bigint stored for only 6 bytes per value (so no overhead, actual savings on the raw values..), but value access via array index isn't very efficient especially for sparse arrays, or sparse queries across millions of rows. And tom says maybe not: https://marc.info/?l=pgsql-performance&m=131229661705005&w=2
Good thing about 3 is that you can abstract the value -> row array mapping in SQL, so app doesn't need to care about it, but array access is slower than i'd like, so how to avoid that? Maybe map N values onto generic columns instead? No array access penalty, and less framework fighting..
4. So like 3 but instead of N values in an array per row, have values in N generic columns per row (value_1, value_2 etc). It is too hard/bad ergonomics to do this in SQL/pgplsql, so now the app has to be aware of what value is in which generic column name, which was ok for us and it has much better query performance. In terms of storage cost, there is a relationship between number of columns/values per row and how much data is required to be paged in to retrieve a value (performance) and row overhead and compression efficacy. We landed on 100 values per row I think, this meant bigint was 8-10 bytes to store instead of the more efficient for approach 3 is tunable tho upto the column per row limit of ~1600.
Can dig up proper numbers later if useful but thats the general idea.
I wonder what approach this project takes, I’ll have to poke around!
Nevermind that that might not be your name or that in the future having the untruncated name might be useful.
Definitely a big problem for then using that data to form the base of an identity system like trusted ID compliance.