mjb

I spoke about this exact thing at a conference (HPTS’19) a while back. This can work, but introduces modal behaviors into systems that make reasoning about availability very difficult and tends to cause meta stable behaviors and long outages.

The feedback loop is replicas slow -> traffic increases to primary -> primary slows -> replicas slow, etc. The only way out of this loop is to shed traffic.

When does AP help?

It helps in the case where clients are (a) able to contact a minority partition, and (b) can tolerate eventual consistency, and (c) can’t contact the majority partition. These cases are quite rare in modern internet-connected applications.

Consider a 3AZ cloud deployment with remote clients on the internet, and one AZ partitioned off. Most often, clients from the outside will either be able to contact the remaining majority (the two healthy AZs), or will be able to contact nobody. Rarely, clients from the outside will have a path into the minority partition but not the majority partition, but I don’t think I’ve seen that happen in nearly two decades of watching systems like this.

What about internal clients in the partitioned off DC? Yes, the trade-off is that they won’t be able to make isolated progress. If they’re web servers or whatever, that’s moot because they’re partitioned off and there’s no work to do. Same if they’re a training cluster, or other highly connected workloads. There are workloads that can tolerate a ton of asynchrony where being able to continue while disconnected is interesting, but they’re the exception rather than the rule.

Weak consistency is much more interesting as a mechanism for reducing latency (as DynamoDB does, for example) or increasing scalability (as the typical RDBMS ‘read replicas’ pattern does).

Practically, the difference in availability for typical internet connected application is very small. Partitions do happen, but in most cases its possible to route user traffic around them, given the paths that traffic tends to take into large-scale data center clusters (redundant, typically not the same paths as the cross-DC traffic). The remaining cases do exist, but are exceedingly rare in practice.

Note that I’m not saying that partitions don’t happen. They do! But in typical internet connected applications the cases where a significant proportion of clients is partitioned into the same partition as a minority of the database (i.e. the cases where AP actually improves availability) are very rare in practice.

For client devices and IoT, partitions off from the main internet are rare, and there local copies of data are a necessity.

Yes, you can do stuff like that. You might enjoy the CRAQ paper by Terrace et al, which does something similar to what you are saying (in a very different setting, chain replication rather than DBs).

(Op here) No deadlocks needed! There’s nothing about providing strong consistency (or even strong isolation) that requires deadlocks to be a thing. DSQL, for example, doesn’t have them*.

Event sourcing architectures can be great, but they also tend to be fairly complex (a lot of moving parts). The bigger practical problem is that they make it quite hard to offer clients ‘outside the architecture’ meaningful read-time guarantees stronger than a consistent prefix. That makes clients’ lives hard for the reasons I argue in the blog post.

I really like event-based architectures for things like observability, metering, reporting, and so on where clients can be very tolerant to seeing bounded stale data. For control planes, website backends, etc, I think strongly consistent DB architectures tend to be both simpler and offer a better customer experience.

* Ok, there’s one edge case in the cross-shard commit protocol where two committers can deadlock, which needs to be resolved by aborting one of them (the moral equivalent of WAIT-DIE). This never happens with single-shard transactions, and can’t be triggered by any SQL patterns.

(OP here).

The point of that section, which maybe isn’t obvious enough, is to reflect on how eventually-consistent read replicas limit the options of the database system builder (rather than the application builder). If I’m building the transaction layer of a database, I want to have a bunch of options for where to send me reads, so I don’t have the send the whole read part of every RMW workloads to the single leader.

(OP here). I don’t love leaking this kind of thing through the API. I think that, for most client/server shaped systems at least, we can offer guarantees like linearizability to all clients with few hard real-world trade-offs. That does require a very careful approach to designing the database, and especially to read scale-out (as you say) but it’s real and doable.

By pushing things like read-scale-out into the core database, and away from replicas and caches, we get to have stronger client and application guarantees with less architectural complexity. A great combination.

That’s a fair point. To be fair to the academic definitions, “eventually consistent” is a quiescent state in most definitions, and there are more specific ones (like “bounded staleness”, or “monotonic prefix”) that are meaningful to clients of the system.

But I agree with you in general - the dynamic nature of systems means, in my mind, that you need to use client-side guarantees, rather than state guarantees, to reason about this stuff in general. State guarantees are nicer to prove and work with formally (see Adya, for example) while client side guarantees are trickier and feel less fulfilling formally (see Crooks et al “Seeing is Believing”, or Herlihy and Wing).

Read-your-writes is a client guarantee, that requires stickiness (i.e. a definition of “your”) to be meaningful. It’s not a level of consistency I love, because it raises all kinds of edge-case questions. For example, if I have to reconnect, am I still the same “your”? This isn’t even the some rare edge case! If I’m automating around a CLI, for example, how is the server meant to know that the next CLI invocation from the same script (a different process) is the same “your”? Sure, I can fix that with some kind of token, but then I’ve made the API more complicated.

Linearizability, as a global guarantee, is much nicer because it avoids all those edge cases.

The point is that, in a disaggregated system, the transaction processor has less flexibility about how to route parts of the same transaction (that section is a point about internal implementation details of transaction systems).

There is no reason a database can’t be both strongly consistent (linearizable, or equivalent) and available to clients on the majority side of a partition. This is, by far, the common case of real-world partitions in deployments with 3 data centers. One is disconnected or fails. The other two can continue, offering both strong consistency and availability to clients on their side of the partition.

The Gilbert and Lynch definition of CAP calls this state ‘unavailable’, in that it’s not available to all clients. Practically, though, it’s still available for two thirds of clients (or more, if we can reroute clients from the outside), which seems meaningfully ‘available’ to me!

If you don’t believe me, check out Phil Bernstein’s paper (Bernstein and Das) about this. Or read the Gilbert and Lynch proof carefully.

This doesn't work as cleanly for SQL-style transactions where there are tons of RW transactions, sadly.

Best of all, /dev/null is also serializable (but not strict serializable) under many academic and textbook definitions.

Specifically, these definitions require that transactions appear to execute in some serial order, and place no constraints on that serial order. So the database can issue all reads at time zero, returning empty results, and all writes at the time they happen (because who the hell cares?).

The lesson? Demand real-time guarantees.

That's much better results than mine!

I notice a similar 'holographic' effect when coloring titanium a couple weeks back, and experimented with getting them dialed in along the same lines as this video. I didn't have nearly as much success, despite the underlying physics being similar. My guess is that the much lower thermal conductivity of titanium causes a lot more smudging than on stainless, which makes the grating effect less pronounced.

One interesting thing I noted with Ti is that satin finished Ti (media blasted with 500 grit glass media) won't take a color from electrocoloring, but will from MOPA laser coloring. Not nearly as nice as polished Ti, but still there. Given that they are such similar processes (growing a set thickness oxide layer), its somewhat surprising to see different results.

I guess I'm going to have to experiment on some polished 304.

I agree. Modern code models tend to do a great job advising in SQL, especially if you include the table definition and EXPLAIN output in the context. Alternatively, I've found that an EXPLAIN MCP tool works well.

Yes. The spatial locality benefits drop off quite quickly. A hashed uuidv7-like scheme with a rotating salt, for example, would keep short term locality and it's performance benefits while not having long term locality and it's downsides.

Always nice to see folks talking about VM snapshots - they're an extremely powerful tool for building systems of all kinds. At AWS, we use snapshots in Lambda Snapstart (along with cloning, and snapshots are distributed across multiple workers), and in Aurora DSQL (where we clone and restore a snapshot of Postgres on every database connection), in AgentCore Runtime, and a number of other places.

> But Firecracker comes with a few limitations, specifically around PCI passthrough and GPU virtualization, which prevented Firecracker from working with GPU Instances

Worth mentioning that Firecracker supports PCI passthrough as of 1.13.0. But that doesn't diminish the value of Cloud Hypervisor - it's really good to have multiple options in this space with different design goals (including QEMU, which has the most features).

> We use the sk_buff.mark field — a kernel-level metadata flag on packets - to tag health check traffic.

Clever!

> Light Sleep, which reduces cold starts to around 200ms for CPU workloads.

If you're restoring on the same box, I suspect 200ms is significantly above the best you can do (unless your images are huge). Do you know what you're spending those 200ms doing? Is it just creating the VMM process and setting up kvm? Device and networking setup? I assume you're mmapping the snapshot of memory and loading it on demand, but wouldn't expect anywhere near 200ms of page faults to handle a simple request.

DSQL does use distributed transactions to offer active-active alongside strong consistency and isolation guarantees. Here's how: https://brooker.co.za/blog/2024/12/05/inside-dsql-writes.htm...

There are a large number of "toolchanger" printers out there, include Prusa's XL. They're pretty great: multiple nozzle sizes and types, multi-material and multi-color without a lot of waste, and even the ability to use different tools (like lasers and subtractive manufacturing). In the consumer space the technology is still fairly expensive, and the complexity reduces reliability, but I suspect is going to become more and more of the norm over time.

Most modern printers also support simpler multi-material setups which change the filament in a single tool automatically. Waste is fairly high (because of the need to purge), and speed fairly low (because of the need to purge), but the technology is mature and cheap.

Titanium fires sure are scary. But there's a good amount of chicken and egg here: expensive material limits demand, which limits progress on manufacturing techniques, which keeps part prices high. I would expect that significant manufacturing method progress would be made if there was a step change in the price of titanium stock.

And I wouldn't overstate the machining difficulty. Sure, it's a pain in the rear, and expensive, but can be done on regular machines with the right tools, techniques, and processes. I've made a couple of titanium parts myself.

Runtime monitoring of system behavior against the spec is one way to close the gap. We wrote about our experiences with one tool for that (PObserve) here: https://cacm.acm.org/practice/systems-correctness-practices-...

Doing distributed systems work in Lean is possible, but right now is much harder than something like TLA+ or P. It's possible that a richer library of systems primitives in Lean ('mathlib for systems') could make it easier. Lean is a very useful tool, but right now isn't where I'd start for systems work (unless I was doing something specific, like trying to formalize FLP for a paper).

Hey Will. I'm a huge fan of the work you all are doing, and of FoundationDB, but I don't believe it's accurate that DST was invented at FoundationDB (or, maybe it was, but was also used in other places around the same time or before).

For example, the first implementations of AWS's internal lock service (Alf) used DST as a key part of the testing strategy, sometime around 2009. Al Vermeulen was influential in introducing it at AWS, and I believe it built on some things he'd worked on before.

Still, Anithesis is super cool, and I really admire how you all are changing the conversation around systems correctness. So this is a minor point.

Runtime monitoring with something like PObserve is a semi-formal approach. Not just regular alarming and metrics.

I (one of the authors) did some distributed systems work with Promela about a decade ago, but it never felt like the right fit in the domain. It's got some cool ideas, and may be worth revisiting at some point.

Correct: https://docs.aws.amazon.com/aurora-dsql/latest/userguide/wor...

Thanks. The team's working on both. For views, do you need updatable views, or are read-only views sufficient?

> Maybe one can transform slow code from high level languages to low level language via LLMs in future.

This is one of the areas I'm most excited for LLM developer tooling. Choosing a language, database, or framework is a really expensive up-front decision for a lot of teams, made when they have the least information about what they're building, and very expensive to take back.

If LLM-powered tools could take 10-100x off the cost of these migrations, it would significantly reduce the risk of early decisions, and make it a ton easier to make software more reliable and cheaper to run.

It's very believable to me that, even with today's model capabilities, that 10-100x is achievable.

Which features would you like to see the team build first? Which limits would you like to see lifted first?

Most of the limitations you can see in the documentation are things we haven't gotten to building yet, and it's super helpful to know what folks need so we can prioritize the backlog.

This makes sense, especially with the move of OSDI to being annual, and NSDI accepting more and more general systems-y work (e.g. we published the Firecracker paper at NSDI, which wouldn't have made sense even 5 years earlier). ATC was left in a difficult niche, but still a valuable venue for "hackier" systems work, industry systems papers of the less quantitative kind, and a few others.

I'd love to see OSDI evolve to accept more of this work, and look at the ATC work that has stood the test of time and accept more work like that. Maybe SOSP and Eurosys too. I bet USENIX is going to figure this out - they're generally a smart and well-run organization.

Fun fact: we won best industry paper at ATC'23 for "On-demand container loading in AWS Lambda" (https://www.usenix.org/conference/atc23/presentation/brooker). I was super happy with it as a paper and got a ton of good feedback on it. About six months earlier, it'd been desk-rejected by the chair of another systems conference who asked that we don't submit such low-quality work to them.