Hacker News Reader

Comment by timenova | Hacker News Reader

I have a genuine question about this. How are we supposed to isolate user data stored in, say Postgres, by region, and still do multi-region replication, and restoring correctly? And how is the application supposed to find which region a user's record is stored?

For example, if the user's data is only supposed to be kept in the EU, and if the user tries to login connected to a US node, should the app forward the request to the EU node (and how would it know?), which then accesses the EU database? If all nodes are connected to all regional databases, does it even make a difference?

Basically, I am very confused about regional separation of data while still running a single application.

View on HN

The basic Problem is that if some CCP goon turns up at your Chinese branch and requests all data on your US/EU users and your local branch is able to do that you failed. Same if some goon from the US government wants to access all data on your EU users. Or some German guy looking for US data in your German data center.

You have to take steps to prevent this.

Difference is that your US and German branches have access to a justice system that can stop the government.

In China, the justice system is part of the CCP and if you don’t comply, your company is gone.

The whole US / EU safe harbour provisions issue is because the EU is not satisfied that in the United States you don’t in fact have a justice system that can stop the government… that there are insufficient powers to stop certain parts of the government like the the NSA, CIA, DEA, FBI counter terrorism groups etc… who are granted extraordinary powers to do questionable shit like sucking up as much internet data as possible without a warrant in the name off “efficiency” using such flimsy reasoning as “it’s only a search for legal purposes when we run a database query on a US citizens data later” and “we’re legally allowed to do anything we like with the data of anyone who isn’t a US citizen… so these EU guys can fuck off” (or the appropriate US lawyer speak equivalent)

The CJEU explicitly ruled that the FISA courts are not independent, I.e. kangaroo courts. I think it’s a fair assessment.

> Difference is that your US and German branches have access to a justice system that can stop the government.

Germany sure, but The US? Do you not remember the Snowden leaks at all?

Almost every company I've worked with the past several years has had a very strict No-US-Servers-Ever-policy, because people don't trusts the US government to not snoop around.

It should be “No US company, period” because the Cloud Act empowers the US government to compel, say Amazon Web Services to disclose information they hold on a server hosted in Germany even if this violates EU law.

https://en.wikipedia.org/wiki/National_security_letter

My interpretation of what GP is trying to say that the way forward is to intentionally limit your own capabilities such that it is simply impossible to comply with government overreach. For an example see Signal and their subpoenas.

> a justice system that can stop the government.

There might be a law that allows government to obtain the data if it is a matter of national security. And what is considered such a matter is up to government to decide.

> have access to a justice system that can stop the government.

Haha, good one.

Encrypting data at rest and in transit will certainly reduce the risk. If a branch is deemed at risk, its employees would not be given access to any databases. Or don't have branches in regions where you have the risk of a government threatening your employees for other countries' users' data.

I am the believer of simple statement - if you don't control physical servers then your data is not secured, no matter what you do to secure it. Once vault is unlocked and you don't control physical hardware your game is over.

In a very small number of cases, regional storage of data becomes regional restrictions. You see that already in how it's unlikely you will be able to access your consumer account from both Russia and the US at the same time unless you have a travel history to one or both countries. Similarly, an app storing data for EU citizens might indeed say, no, we can't share this data outside the EU, we're not allowed. How it figures out if you're outside a region is, of course, problematic.

But that's rare. Generally the data lives on a regional server and gets accessed by a CDN in a region near where the data will be delivered. If you're allowed to sign in from that region, then you're likely authorizing transfer of data (for the duration of that session) to that same region.

As for multi-region replication, it's often the case there are multiple regions in the same country - for example, us-east-1 and us-central-1. No jurisdictional problems to host the data in two regions in the same country, after all.

As to how you look up what region the user's data is stored in, well, you can keep a list of users and regions. Worst case, force the user to remember their region, and use separate copies of the website (and DNS) to pick a region to sign in on. For example, the same app/website, but at different country domains.

I understand that one way is to block users from logging in from other regions, or giving them a region selector at login (which is terrible UX IMO).

However, if any of these approaches are done, the operational complexity will be immense, and latency for requests much higher than they could have been. A lot of message passing would be required to build such an architecture, and a lot of bandwidth would get wasted, not to mention the unnecessary costs.

A simpler approach would to store core business data regionally, but peripheral data everywhere. For example, a todo app would store all user records, project lists, and other data everywhere, but the actual todos regionally. And yes this becomes infinitely complex will collaborative cross-region applications.

I don't think there are any good solutions yet to this, and governments should invest in creating better solutions before forcing to go down this route.

Bonus question: what do you do if data needs to stay in the creating user's region by default, but users can share data across regions, including creating new data associated with a record from another region?

For example, American and EU users sharing documents and requesting comments be added. How do you manage both isolation and sharing? Do you copy such documents to both regions? Do comments live in the same region as the document, or in the region of the creating user? Do you just give up on foreign key integrity for documents, comments, and users because they could be in separate databases? If your American servers can request data out of the European region on demand (subject to business logic), are you really protecting data to the standard the law demands?

> are you really protecting data to the standard the law demands?

This assumes the law's purpose is to protect citizen's data and that it's not just a way to bully tech companies into submission by writing impossible to implement laws and selectively enforcing them.

If you need geographic redundancy beyond the European borders, what is your threat model trying to mitigate? E.g. Let's say you have servers in Norway, Germany and Spain, what would be solved by e.g. having an extra one in the US?

Roaming access can be handled through a VPN

It seems a product opportunity might exist to ensure only encrypted copies are replicated out of region with the keys remaining in region. The side effect is a region might be down if something catastrophic happens but data could be restored

Encrypted copies for some isolated data is a good solution, but for a user record when you want to login a user, will not work. Then you would need to fetch the data from another datacenter anyway, increasing costs, latency, and operational complexity.

You're assuming the laws would allow for that. I wouldn't count on it.

The simple answer is that it's no longer a single application. It's a separate copy of the app per country/region.

should the app forward the request to the EU node (and how would it know?)

You have to add a home jurisdiction field to the account.

The word that jumps out (to me) with what you describe is "Partition".

What part of your question is the concern of a bureaucrat? We can drop a couple 9s of uptime in the name of local data primacy right? (I specifically didn't say "data privacy") :)

I believe it'll lead to much higher costs, CO2 emissions, and other issues, simply because this architecture will require more message passing, wasted bandwidth, will be less reliable, and have higher operational complexity. I get that those in decision-making power aren't usually technologically adept at figuring out solutions for these issues, but they should invest in solutions first.

I would predict that data localization leads to data traveling shorter distances because most people access their local region rather than everything going to us-east-1.

Reply on news.ycombinator.com