Hacker News Reader

Comment by bastawhiz | Hacker News Reader

Even if the response changes only very infrequently, permissions are not what you want to be serving stale responses for. Can you confidently invalidate the right caches in a system serving 400M QPS? I would bet that relatively little is cached in IAM today.

View on HN

Not very stale, but there is definitely a period where the policy is unstable, e.g. if you allow some action, for a relatively short while (usually less than a minute) you can see stale responses where the action is blocked, sometimes interleaved with allowed access.

It depends on what exactly they're providing. Are they providing bounded staleness for certain types of reads? And is that the case for all reads, or do they also cover the new enemy problem? I.e. do they prevent people who previously had access to a resource from viewing changes to that resource?

iam is eventually consistent...because its caching so much. we can see changes easily take a minute to propagate

Replication delay and stale caches are two different problems with vastly different causes

Can you elaborate more? Is it not possible to have replication delay due to stale caches?

I'm not aware of a data store which populates its replicas with data from a cache. Surely there are caches within any given data store, but the whole point of a replica is to have the data replicated (for scaling reads, redundancy in case your leader fails).

If you're using caches, you're deliberately sending old data to the replicas. I can't think of why you'd do that. I guess you could have replicas of your caches, but I'm referring specifically to data store replicas.

Is a replica not a cache of sorts?

Not really. Replicas are constantly replicating live data from your leader. All of it, in its raw form. A query against a replica isn't inherently faster than a query against the leader, the data is likely just a bit old.

A cache stores the output of an operation to avoid needing to perform the same operation again. You deliberately put data into a cache to avoid needing to create that data a second time—it won't magically appear there.

You don't always need to shoot down caches for consistency. See the Zanzibar paper for how one system maintains consistency without cache invalidation.

> Even if the response changes only very infrequently, permissions are not what you want to be serving stale responses for.

But perhaps the permission updates are slow. Like, you change a permission, wait 5 seconds, and the system tells you that they are updated across all servers.

The delay is either in replicating the data to all of the sources of truth, or invalidating the cache. Invalidate the cache too soon and you're recaching stale data. Caches only increase the delay more.

Reply on news.ycombinator.com