Hacker News Reader

Document stores trade off some performance and robustness to increase ease of development. When that becomes a problem you can switch to more structured tables. One possibility is to structure frequently accessed fields and stuff the remainder into a json column.

View on HN

This just forces complexity onto the caller - it's not unreasonable to expect the DB to "pseudo-schematize" as most keys will be repeated.

With a power law, only a few keys occur frequently. Changing the data representation implicitly would cause unpredictable performance as columns get rewritten. If you want to do more explicit storage optimisations, maybe one of the mongo-like overlays could accommodate that.

It's more than that - virtually large all datasets I've worked with (> 20B documents) had a keyspace of under 128 keys/columns per nested level of a JSON document. Meaning an optimizing DB could express most keys as single byte (or less with packing). I don't thing column reordering would be necessary unless you exhausted the first 128 keys and you wanted to optimize the base-128 keyspace assignment. Such an operation could be done during compaction or any other operation that rewrites a table.

Reply on news.ycombinator.com