What kind of access rates? Etc.
Very general answers are okay -- I'm trying to wrap my head around whether this is even in the right ballpark for my world.
Distinguishing current proof-of-concept vs. design-goal scale is okay too.
Thanks!
But we (cautiously) don't see any reason why the basic design shouldn't scale to very large (e.g. petabyte) datasets, and that is our eventual goal.
That said, we do think there are a lot (even maybe the majority) of use cases in the GB-TB range.
I think the git-like features (history, merging) are very helpful for internal work, but when the dataset must be published, I think in most cases only the newest snapshot should be made available. But then the question is what format should it have...?
In such situations, you could either (eventually, when it is implemented) prune old data, or aggregate the changes into bigger blocks.