Hacker News Reader

Comment by kentonv | Hacker News Reader

You'd use it whenever you're building an application on Cloudflare Workers and you need to store data. D1 provides you with a SQL database. It's based on SQLite, so it's designed for relatively small datasets but can serve them very quickly from the edge.

Note there are several alternatives here, too. Workers Durable Objects[0] provide a lower-level primitive for building advanced distributed systems. But D1 is easier to use for typical use cases. For blob storage you might use R2[1]. And for large databases Workers can easily integrate with several serverless database providers.[2]

[0] https://blog.cloudflare.com/introducing-workers-durable-obje... [1] https://www.cloudflare.com/products/r2/ [2] https://blog.cloudflare.com/announcing-database-integrations...

View on HN

Is it a non-goal to be long term usable for larger databases? That would force the usage of something like turso your closest direct comparison as a possible migration strategy or relying on "Smart Placement" (which from my point of view reduces the benefit of global edge) for other serverless non-global dbs.

Personally, I'm a firm believer that most "web app" use cases are better served by many small databases (e.g. per-user or per-document) rather than a single monolithic databases. This is especially true when serving users all around the world -- per-user databases can be located near each user (both for speed and to comply with data locality laws).

What I'd like to enable here is a progression where you start out prototyping your app with a single D1 database, which is easy to use and reasonably fast. Then as you grow we provide tools to let you transition to many D1 databases sharded in a way that makes sense (e.g. per-user). Apps that want even more control can move to using full-on Durable Objects (which will soon support a SQLite database per-object).

That said, there are certainly many use cases out there where simple monoliths make sense, especially non-interactive data crunching. I'm not sure yet if D1 will ever be the right choice for those, but the Workers platform aims to provide many options.

Thanks for the insight, I greatly appreciate it! This definitely is a reasonable idea for many things and I'm looking forward to seeing something similar to the sharding mechanism in the future.

I've only started to think about this and I'm thinking the hardest part will be dealing with cross-cutting concerns (in a non-auto sharded world manually creating multiple database) and trying to find a way to keep each database isolated without extra burden compared to using a hosted Postgres.

As an aside, that lan optimized house was a gaming dream. Hope your new house is as awesome.

" Apps that want even more control can move to using full-on Durable Objects (which will soon support a SQLite database per-object)."

Can you elaborate this little bit more? Im using DO today and i have a bad time sharding my data (works, but i hate it);

So i will have the option to use the standard store or/and SQLite?

If so, i dont can keep with my DO (because i have control of everything) and use SQLite for things that is bigger than what the value store supports.

Sorry, I don't quite understand what you're asking.

In the future each DO will have a private SQLite database. The key/value store will actually be redirected to store into a special table in this database, but probably new apps will just use the database and not the KV store.

Separately from that, I would like to develop tools that make sharding Durable Objects (and D1 databases) easier. Today it's a pain to do manually. This is independent from the underlying storage model, though.

I've been working on a little project that uses D1 and have been hoping that is how D1 will evolve. A db per customer in my case. Is the colocation in a durable object so 'business logic' and sqlite can live in the same isolate for performance and security? Would that be a post 'out of beta' feature? Ive been building as a monolith thinking that the number of databases in the alpha was indicative of the future.

Cloudflare are 285 pops. Surely you dont need your db replication them all of them. A few locations per continent should suffice.

For comparison, fly.io, turso's provider, has 34 locations and well-documented reliability issues.

That is a fair point. A few centralized locations will likely be more than sufficient for most use cases.

I mean, our object storage is called R2, our first database offering is called D1, and if we were to offer a fully Distributed Database then D2 seems like a good name.

And you can follow up D2 with D2: Lord of Distributed.

Reply on news.ycombinator.com