I don't have any perf numbers I can share but I can say we see ~30% compression with eStargz which is already a small win atleast heh.
10,011 karma · joined March 14, 2013
Principal Software Engineer @ NVIDIA
Opinions are mine and not that of my employer, etc.
I don't have any perf numbers I can share but I can say we see ~30% compression with eStargz which is already a small win atleast heh.
OCI has some benefits over other systems, namely that tiered caching/pull-through is already pretty battle-tested as is signing etc, beating more naive distribution methods for reliability, performance and trust.
If combined with eStargz or zstd::chunked it's also pretty nice for distributed systems as long as you can slice things up into files in such a way that not every machine needs to pull the full model weights.
Failing that there are P2P distribution mechanisms for OCI (Dragonfly etc) that can lessen the burden without resorting to DIY on Bittorrent or similar.
With low altitude being an aggravating factor he was always 100% correct in ejecting and whatever the plane did afterwards is largely irrelevant.
Would be nice if the EU could step in an make DisplayPort the required connection and protocol like they did for the USB-C port for charging.
My personal favourite solution is Bazel specifically because it can be so isolated from those layers.
No need for Docker (or Docker in Docker as many of these solutions end up requiring) or other exotic stuff, can produce OCI image artifacts with `rules_oci` directly.
By requiring so little of the runner you really don't care for runner features, you can then restrict your CI/CD runner selection to just reliability, cost, performance and ease of integration.
China will laugh all the way to the bank.
Atleast 2 of the fast growing companies I have been a part of have had serious layoffs that were very successful in cutting dead weight that had accumulated because of fast and loose hiring and poor middle management. So it can be done well.
By and large though they are a sign to jump from the soon to be sinking ship... so it's very important to know what kind it is and act accordingly.
I was a doubter a bit as to how it would work at a bigger org but so far rock solid, easy to setup and great user experience.
Think about it. They support Ukraine by sending them surplus or aged weapon systems while modernising and funnelling money to their military industrial complex (with public support!). In return Ukraine uses those weapons systems to keep Russia tied up in a war, exposing Russia's most modern weapon systems and providing a huge amount of information on how to fight a drone war.
This is a great investment independent of what happens in or to Ukraine, basically they turn weapons they needed to replace or get rid of anyway into intel, new weapons, political brownie points at home and a weakened adversary.
China doesn't mind it that much either, though they enjoy it less because they had invested fairly heavily into Ukrainian infrastructure that is being destroyed. Keeping Russia isolated makes it easy for them to extra cheap oil & gas, push for pipelines that flow to China instead of towards Europe and also offload a bunch of their industrial surplus now that their requirements are winding down and the rest of the world is trying to tariff them for dumping.
So basically the problem is 1. is good for almost everyone except Ukraine and Russia. Trump for whatever reason can't understand that and just wants to throw a tantrum as is befitting for someone of his intelligence.
You can try to use "standard" options like MinIO/Ceph(RADOS)/SeaweedFS but you will very quickly learn those systems aren't remotely fast enough for these usecases.
AI training is what this is used for, not inference (which has absolutely no need for any filesystem at all). What makes the workload somewhat special is that it's entirely random read and not cacheable at all as most reads are one and done.
Would Lustre be perfectly fine at 6TiB/s? Yes. Is it a huge pain in the ass to operate and make remotely highly available? Also yes. If this thing is capable of the throughput but easier to operate and generally more modern and less baroque it's probably an improvement. TLDR is Lustre is fast but that is literally it's only redeeming quality. I have lost far too many hours of my life to the Lustre gods.
The only competitors in the parallel FS space that are useful for this are Lustre and Weka.
Otherwise if you don't need a single namespace a bunch of fat AF NFSv4 servers w/NFS over RDMA will also get you to 6TiB/s.
The "surefire" way though is still Lustre, it's the big daddy of distributed parallel filesystems still but it's an absolute beast to setup and operate.
That said the kernel is the bigger picture, if he wants it to be successful and remain in mainline he needs to adapt to how Linus expects maintainers to operate. That probably means delaying getting fixes to mainline users until he can show they have been adequately tested. Until that trust is built bcachefs is always a few bad cycles away from being dropped from mainline.
Personal electronics (unless you are Apple basically) is a pretty fickle business and fortunes can change very rapidly. Aviation? Not so much, unless you chuck a Boeing.
GHA/Gitlab CI/Buildkite/whatever else floats your boat then just builds a bunch of Bazel targets, naively, in-order etc. Just lean on Bazel fine-grained caching until that isn't enough anymore and stir in remote build execution for more parallelism when you need it.
This works up until ~10M+ lines of code or ~10ish reasonably large services. After that you need to do a bit more work to only build graph of targets that have been changed by the diff. That will get you far enough that you will have a whole team that works on these problems.
Allowing the CI tools to do any orchestration or dictate how your projects are built is insanity. Expressing dependencies etc in YAML is is the path to darkness and is only really justifiable for very small projects.
It feels more like an OSS tool for managing university campus scale infra, which is completely fine if that is the problem you have but for commercial scale infrastructure unfortunately there isn't a good OOTB DCIM option right now.
If you are wronged by a machine you are always going to be emotional about it, i.e having your email account unceremoniously terminated by Google without recourse by an essentially automated process.
The thing to solve for is the lack of recourse. Very very difficult in American society given how skewed the relationship is between corporations and people (or the government for that matter).
If instead when a Waymo demonstratively caused an accident the payout was 10-100x higher than if a human was involved I suspect many peoples qualms would go away.
While I think just the "knowledge" (assuming Americans would accept facts...) that the driver is X% better than the best drivers on the roads would provide little comfort to most.
I think in societies outside of America these may be more successful actually. i.e most parts of Asia.
They should be compared to taxi/Uber/Lyft drivers that is what is being replaced by this and as such where the net harm increase or reduction will occur.
Everywhere I have worked where we have run clusters in the 100s to 1000s of nodes we have rarely had a team larger than 4-5 of true k8s folks and even then it's been a split between folks that are very hardware provisioning/network/etc focused and more higher level k8s folk which also take on a large portion of CI/CD work also.
At smaller scale (in the $1M/yr ballpark) I have done all the k8s bare metal ops myself along with all CI/CD and been responsible for a ton of the backend programming too. This is feasible because with distros like Talos etc it doesn't take a lot of manpower once it's setup and upgrades aren't too painful at small scale if you aren't running stateful services.
So tbh no, you just need ideally 2 folks at around ~200k/yr each that are competent and have done it before. The rest of the folks on the on-call rotation are just the rest of your engineers (and if you are at $1m/yr cloud spend you have more than 10 of those).
The only other time I have received better support was from Aussie ISPs. Back in the day when you called Internode the guy who answered the phone was a bona-fide network engineer and would go as far as getting a shell on the DSLAM to check out what is going on. To me that is peak support, live debugging of the problem!
Similarly I called into Aussie Broadband to do my first NBN setup, explained I did "BYO" modem because I was going to initiate the PPPoE session with my Linux router and they said no problem. She even offered to send me a cookie cutter pppd config along with the info to set it up myself. Easily the some of the most knowledgeable and "can do" attitude for first layer support I have encountered.
Needless to say when I encounter damn good support I stay even when it costs more.
I do think 100k/mo is the tipping point actually, that is $1.2M/yr.
It costs around $400k/yr in engineering salaries to reasonably support a sophisticated bare metal deployment (though such people can generally do that AND provide a lot of value elsewhere in the business, so really it's actual cost is lower than this) and about $100k/yr in DC commitments, HW amortisation, and BW roughly. So you save around $700k a year which is great but the benefit becomes much greater when your equiv cloud spend is even bigger than that.