Hacker News Reader

If the data fits in RAM, you could try loading it into an in-memory sqlite database, and run SQL queries there.

I thought about making a "shadow database" on a server since my work PC is only 8 GB ram but ultimately decided against it.

The first reason was GDPR compliance and the second is that we have an already complex architecture as it is (and we're only 2 to manage it) so I don't want to add one more brittle layer that will need debugging at the worst time.

About GDPR: is it possible that holding data in a non-storage/volatile medium could be legally distinct from non-volatile storage, especially if it's essentially syncing with an authoriative data source that's responsible for managing GDPR? Because if not, it seems to me every proxy and persistence layer runs legal risks....

To be honest I have no idea. But since our supplier still hasn't implemented the mandatory 4 months max data retention I'm not taking any risk.

No. GDPR covers any processing. Simply having the data pass through your machines / software makes you a part of the chain of processors / controllers.

They do.

Reply on news.ycombinator.com