Admittedly this is armchair architecture talk, but it seems like either consul or Roblox's use of Consul is falling into a CAP-trap: they are using a CP system when what they need is an eventually-consistent AP system. Granted, the use of consul seems heterogenous, but it seems like the main root cause was service discovery. And service discovery loves stale data.

Service discovery largely doesn't change that often. Especially in an outage where a lot of things that churn service discovery are disabled (e.g. deploys), returning stale responses should work fine. There's a reason DNS works this way - it prioritizes having any response, even if stale, since most DNS entries don't change that frequently. That said, DNS is not a great service discovery mechanism for other reasons. Not sure if there's an off-the-shelf solution that relies more on fast invalidation rather than distributed consistent stores.