They claim they test backups quarterly yet they don't have a procedure in place to restore the operation. We all know your backup is not tested until you restored everything successfully.
This is not an engineering mistake, it is a flat out lie.
But,
You can imagine problems restoring one
individual tenant's data to an otherwise
active database with many tenants
any cross-tenant primary keys
That's such a recipe for disaster; giving each client their own database seems like the easiest win in the world.
But I'm not intimate with Atlassian products. Maybe they have some products where that's not practical for some reason.
1) Its super common even in multitenant systems to have a common database with configuration information (for example) which serves all tenants, and tenant-specific databases used alongside that to host their private data.
2) Back when sharding started to be a popular scaling pattern, tenants were not always split up by the tenant boundary but by some other reliable key. Obviously this isn't true multitenancy and I think most DBAs would discourage the pattern today. However, given the age of the products at Atlassian (and assuming a fast-and-loose engineering culture, which has been alluded to elsewhere) its entirely possible that parts of these products as well as the entire product itself may use this kind of sharding.
Bottom line, we can only hypothesize unless and until someone from Atlassian actually details their architecture (which may have happened? I dunno, I haven't been paying that much attention to it…)
1) Its super common even in multitenant
systems to have a common database with
configuration information (for example)
which serves all tenants, and tenant-specific
databases used alongside that to host their
private data.
One wouldn't imagine that the shared config database would have a dependency on any of the individual client databases and that they could therefore be moved/dropped/restored at will, independently of the shared config database.
2) Back when sharding started to be a popular
scaling pattern, tenants were not always split
up by the tenant boundary but by some other
reliable key.
Nothing, and I mean absolutely nothing, that Atlassian has to offer is rocket surgery-kind of hard... yet, here we are... not being particularly surprised at all.
But the idea of doing a full restore to a new environment, then only enabling accounts for the impacted tenants, is a good one.
They're in control of the architecture: rollback, backup, and recovery should all be considerations
On a single Postgres instance you can (at least theoretically) have 4 billion databases per instance.
In that case, the tradeoff between isolation and ease of development is made. That said, having a schema per user (even if in the same physical database) seems like a nice approach, if you can stomach the overhead and added ops complexity.
And I don't think even pgbouncer would help.