Even 25 years ago being “too online” - back then it was icq, livejournal, and forums - was already destructive as hell and always led to some pretty cringe outcomes. At least these days some people make a career out of it, back then it was purely negative
First of all, web/api being down is not the worst case scenario - i didn't even notice this till late in the day. Secondly, even 1M qps at the edge is tiny amount of compute, what you have here is not a scaling problem it's a misconfiguration problem. Don't make it worse by adding more load on the system with your preemption churn and maybe taking out more important services causing more severe/longer outage - just fix it in the edge.
Yes, just absolutely crazy way of doing load shedding.
> why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway?
Like what workloads? Application backends and databases? Did you ever think that your past three employers maybe had a valid point?
What TP is talking about has nothing to do with workload preemption and is more of a variation of loadshedding (e.g overload management in Envoy). When I was part of the team that ran Google's clusters we had very few priority classes - basically just one for system and majority of serving workload ran on another priority and the rest was for batch. Pretty sure SRE book recommends just that.
backoff is only applied to individual pods/containers not across pods. the point is at scale it's easy to get into a situation where it's not possible to recover without (usually manual) full service drain
1. was already mentioned in sibling - cascade failures
2. you'll have massive number of restarts for various flake reasons and missing things that got papered over with restarts until you hit 1 and everything is broken. another popular version of this is "just restart when memory leaks too much"
article description is low on details but i don't think what you and OP are describing can be categorized as overfitting in the statistical sense and more like "reward hacking" by the model. like you said more likely than not the agent had access to bench source code and just fitted solution to that.
on older PC bios would perform a memory test - you may remember those - and that's where it would mark a bunch of pages dead/unusuable and it was not uncommon at all.
this reminds me how in the late aughts russian government tried to create some national, lib-free wikipedia version so what they did was rip entirety of wikipedia, deleted some articles and then search-replaced "wiki" with their name in the entire corpus. hilarity ensued within articles about vikings (most slavic languages only have single variant for v/w, except polish) and more.
RS and XOR EC are definitely useful features for an object store that's storing from like tens of petabytes to exabytes worth of data but for a remote block device with a typical size up to a couple dozen terabytes I'm not so sure. Thanks for the link I will check it out
Edit: skimmed it, classic AI "analysis" - totally ignores all the crap io_uring went through
Well it's SPoF for a shard not the whole thing so in practice that makes this considerably better. Secondly, it's not a datapath SPoF so presumably all tailscaled processes would continue running and exchanging traffic no problem - you just won't be able to launch any new ones.
With opus 4.7-ish to fable 5, immediately after release - before they locked it down, it was shockingly easy to find crash bugs in a lot of very heavily used DBs and other software.
First of all it's not just SV - all of US sales is like this. Second, if you're talking about Europe - base/variable comps split may not be 50/50 but it's often the same OTE structure with some modifications due to local legalese
That was exactly my process on this article and the rest. Often i actually scrap claude/chatgpt edit because it sounds so fake and rewrite it myself. And like i said elsewhere - diagrams are pure ai because i don’t personally care for react especially for throwaway code. But hey thanks for patronizing tone, as a former googler myself i wouldnt expect less
I actually do get public cloud as well as running your own racks. There is room for both. Your startup is growing 100% every quarter but you're not sure if you have real PMF yet? Yeah probably stick with cloud. Zombiecorn growing 10-20% yoy or not growing and you're paying the cloud $10M/y for 6yo CPUs? - pretty foolish to continue doing that.
This is a company blog. I run some paragraphs through ai to fix my dogshit sentence structure since english is not my first language (come to think of it, my russian has always been pretty lame too). Get over it
Did anyone really expect OCI deal not to become, shall we say, less generous as they slowly creep up their market share? Anyone banking on OCI being cheaper provider is playing a game of musical chairs.
Im not entirely sure what the motivation was for replacing nghttp2 with oghttp2 - nghttp2 is probably the most used codec for http/2 - so probably not compliance. I think it was mostly about code reuse since oghttp2 is part of QUICHE which is also used in Envoy for QUIC.