cweld510

I don’t think this is because of AI. Rather, it seems a continuation of a shift which has already been occurring for my entire career, which is that the tech industry continually sheds people whose skills are entirely practical and are tied to a specific era or regime of technology. For instance, at one point, there were webmasters, but now those jobs don’t exist anymore. Sysadmins have gone through a similar struggle with the advent of the cloud. Once there were sysadmin jobs, and now there are no more. Today it is happening to a certain kind of full-stack engineers specializing in technology of the last 15 years. In the future it will probably happen to YAML engineers who specialize in Kubernetes and GitHub Actions.

Consistently the most durable roles seem to be those which require theoretical understanding of the fundamentals —- UI/UX, systems, algorithms, etc. It’s unfortunate that not everyone gets a chance to learn these things.

You're right -- we do relax the integrality constraint, gaining performance at the expense of some precision, and we're generally able to paper over the difference at scheduling time. We've investigated integer linear programming for some use cases, but for solves to run quickly, we have to constrain the inputs significantly.

Great to see this post here -- really enjoyed writing it! I think it's really cool how an algorithm from an operational research context can play a critical role in a high-availability large-scale cloud service.

Sure, but without an LLM, measuring customer retention might require sending a request over to your data scientist because they know how to make dashboards, then they have to balance it with their other work, so who knows when it gets done. You can do this sort of thing faster with an LLM, and the communication cost will be less. So even if you choose the wrong statistic, you can get it built sooner, and find out sooner that it's wrong, and hopefully course-correct sooner as well.

Rust is more a response to C++ than to C. Both C++ and rust are big and complicated languages that are good for large projects that are performance-sensitive. Both have very strong static typing and can be verbose as a result.

C feels substantially different than Rust. It’s much smaller and less complicated. It’s technically statically typed, but also not in that it doesn’t really have robust non-primitive types. It’s a very flexible language and really good for problems where you really do have to read and write to random memory locations, rearrange registers, use raw function pointers, that sort of thing. Writing C to me feels a lot closer to Python sometimes than to Rust or C++. Writing algorithms can be easier because there is less to get in your way. In this way, there’s still a clear place for C. Projects that are small but need to be clever are maybe easier done in C than Rust. Rust is getting used more for big systems projects like VMs (firecracker), low level backends, and that sort of thing. But if I was going to write an interpreter I’d probably do it in C. Now, I’d do it in Zig.

The useful thing about proofs is that they are written in English (or another language), not formal logic. In general they can be mapped to formal logic, though. This means that people can digest them on an intuitive level. The actual goal of a proof is to create new knowledge (via the proof) which can be distributed across the mathematical community. If proofs exist but are not easily comprehensible, then they don’t accomplish this goal.

It makes sense from a financial perspective for a small company that’s trying to grow. The most precious resource at such a company is engineering time. It’s very limited - you only have X number of engineers, it’s hard to hire good engineers, and they’ll only work so much. Engineers create the thing that makes money and they are in the critical path for nearly every single product experiment that a growing startup needs to perform to figure out its PMF. So nearly all of the small startup’s engineering time should be spent on things that differentiate its product, not stuff like serving frontend code, which is a solved problem. Traditional VC-funded startup economics assumes exponential growth after PMF, which means that engineering work required to reach PMF has potentially compounding returns.

Engineers know this too, which is why they choose Vercel over rolling their own infra. Frontend engineers get paid for building nice UXs that people will pay for, not mechanically implementing routing policies. So they’ll outsource that work to Vercel. It’s similar for backend engineers. What compounds this effect is that, at a small startup, you are on call in perpetuity. So you need to aggressively farm out support work for nonessential systems to service providers or else you risk being overwhelmed.

It’s not a matter of excusing or not excusing it. Incidents like this one happen for a reason, though, and the real solution is almost never “just do better.”

Presumably crowdstrike employs some smart engineers. I think it’s reasonable to assume that those engineers know what CI/CD is, they understand its utility, and they’ve used it in the past, hopefully even at Crowdstrike. Assuming that this is the case, then how does a bug like this make it into production? Why aren’t they doing the things that would have prevented this? If they cut corners, why? It’s not useful or productive to throw around accusations or demands for specific improvements without answering questions like these.

I work on an event-based architecture that I think is successful, but that’s because our core primitives are event-based, so there is no impedance mismatch in the way that there can be if you migrate from a request-response architecture to an evented one. Specifically, we aren’t trying to deal with databases and HTTP (both of which are largely synchronous primitives). Instead, I work on a platform for somewhat arbitrary code execution; and the code we are executing depends on our code rather than vice versa. In general, the code we execute on the platform can run for an indeterminate amount of time, and it generally has control and calls back into our code rather than our code calling into it. So our control flow is naturally callback-based rather than request/response; as a result, our system is fundamentally event-based.

You can absolutely dump the traces somewhere and analyze them yourself. The problem is that this falls apart with scale. You are maybe serving thousands of requests per second. Your service has a ton of instances. Capturing all trace data for all requests from all services is just difficult. Where do you store all of it? How do you quickly find what you need? It gets very annoying very fast. When you pay a vendor, you pay them to deal with this.

I think S3 is too slow to put in your control plane’s critical path; better to use something like Redis.