RAM is really fucking expensive though. Even a modest data set of a couple of dozen terabytes will cost an arm and a leg if you're keeping it mostly in RAM (with redundancy).
I mean, yes, putting a bunch of drives on a local machine is cheap. But that too is a circumstance where a mmap()-using DBMS is probably inappropriate.
This is really, really false. It would make my life a lot easier if it were true.
(The first job I had was at a 10-person startup in a coworking space, barely making enough revenue to break even, and still consuming a vast, vast amount of data, because the product involved constant streams of sensor data from a vast number of tiny cheap devices. People tend to forget that not every single business is a CRUD web/mobile app whose average user accounts for $20/month revenue against at most a few hundred HTTP requests and a couple megabytes of disk.)
Even that depends on a lot of factors. Just as an example, if it takes $10 to build and deploy a sensor, and it returns one number per second, then $1 of RAM can hold 2+ years of data before archiving it.
[0] https://static.googleusercontent.com/media/research.google.c...
32. Which I would expect to be overkill compared to the precision of your average sensor.
> Also, you need to identify which user it belongs to, as well as - in our case - which device and which sensor on that device.
Which is the same for big chunks of readings, so I'm assuming a system that's able to store that metadata once per several readings.
> How do you protect against bit flips?
I dunno, I was just saying the cost of memory. You can have bit flips on any kind of server no matter how you're doing it, and they might get persisted, so I'd say that's out of scope.
> average incidence of uncorrectable errors per DIMM
That average is highly skewed by broken DIMMs though. They found a mean of a few thousand correctable errors per DIMM, but they also found that 80% of DIMMs on one platform and >96% of DIMMs on the other platform had zero correctable errors.
> Your tolerance of errors may differ, but you'd probably want to replicate the data at least once. Disk seems a good choice, for the added benefit that you can actually survive a power cut without going bust
Sure, disk for backup sounds lovely and would be extremely cheap compared to the RAM. I wasn't advocating having only the RAM copy, just saying that depending on other factors it might be reasonable for a RAM copy to be the main analysis database even for sensor data.
The post that started this thread directly says you should be using persistent files as write-only dumps/logs.