What I really want: SQLite storage backend driver for s3/gcs. No need for disks then. I haven’t been able to find such a solution though; and am not technically proficient enough in C (the Lang SQLite is written in) to do so myself.
https://github.com/dacort/athena-sqlite/blob/master/lambda-f...
Implemented the VFS side in Python, thanks to the awesome apsw library.
Furthermore, it doesn't solve the multiple writers problem, because (afaik) there's no way to lock a file on S3.
Is this the cost of network transfers changed by AWS or GCP? What if I'm hosting my app on e.g. EKS or GKE respectively?
S3 objects are also immutable. Once they’re written, they can’t be updated.
A read-only version of this might be useful, but probably wouldn’t work in-place.
Something that might be of interest is S3 SELECT support that lets you query a single (optionally compressed) CSV, JSON or Parquet file server-side at the same cost of a regular S3 GET.
https://docs.aws.amazon.com/AmazonS3/latest/dev/selecting-co...
And if you really want relational (i.e. JOINs, aggregations and sub-queries) semantics on a bucket full of CSV, JSON, Parquet, ORC or regular-expression-describable files in a cost effective way that has great performance on buckets containing 100’s of TBs of data, definitely look at Athena which is only $5/TB of data scanned during a query.
The interface will be either HTTP or Redis protocol, you create your database, and daily I will back it up on S3.
(If interested you can subscribe for updated here: https://simplesql.carrd.co/
[1]: https://redisql.com/
1. I do not want my production instance to shut down for WHATEVER reason. This is just not an acceptable risk for most businesses. The only time a DB can go down is something goes wrong.
2. As an engineer, I understand that 3 counters that are not accurate aren't a big deal. I can even look into the source and see that they really do as you say. Justifying this to a security org will be a complete nightmare, as most security orgs in enterprises are staffed with barely technical folks masquerading as "security".
So, it seems like a pretty good way to coerce enterprises to pay up while letting hobbyists continue using it. Very smart, I wish you the very best!
Make it available and simple for hobbyist and small companies, ask money to who can afford it to sustain the development.
Right now I am busy with releasing the v2, but after that I should definitely do some more marketing.
Anyhow, to give you an order of magnitude, on memory data storage we reach ~80k inserts for second. On a machine with 1vCPU and 3GB of RAM, a 15$/month box from DO.
APSW exposes the sqlite backup api so you could do them online without shutting down the database.