Comment by weberc2 | Hacker News Reader

Comment by weberc2 | Hacker News Reader

Parent Full thread

> For one, because they need a fixed binary representation for the type, to persist it on disk. In a programming language yoi do things in memory, so you don't have that issue...

Memory and consequently programming languages also requires a fixed binary representation. How you represent data is orthogonal to its storage medium--you can write application memory to disk and read it back in, no problem (e.g., swap).

> Still, you could have had union types, or even coerce everything as string, as SQLite does, but it would be bad for performance as they'd need an alternate representation.

The problem doesn't go away by moving support out of the database and into the application; it only makes it worse insofar as the application is limited in its optimizations. At the end of the day, the real world has sum types, applications use them, and they are encoded into databases--they simply aren't encoded _well_ and the database isn't giving you any correctness guarantees as it does for product types (i.e., structs, records, etc).

>Memory and consequently programming languages also requires a fixed binary representation

Already covered that.

Programming languages can save their data as unions or structs, and take the minimal hit to switch on the type.

DB's persisting data on disk can do the same but will take a much bigger hit.

>How you represent data is orthogonal to its storage medium--you can write application memory to disk and read it back in, no problem (e.g., swap).

The costs are not orthogonal to the storage medium however.

>The problem doesn't go away by moving support out of the database and into the application

On the DBs side, it does go away. The DB only has to guarantee what it says it supports (only store one specific type in a column). So for the DB implementors, that's a great invariant for their implementation ease and performance.

>The DB only has to guarantee what it says it supports (only store one specific type in a column).

If you mean primitive types specifically, this already is far from the case, and it's great. One "type" is great, but types can be composed. Allowing types beyond primitive types has already been a blessing for me. Getting columns of type ARRAY[some other type] or MAP[type, type] is incredibly convenient. I don't feel so strongly about the JSON types that are entering into every db, but they're certainly supported and widely used.

> Already covered that.

Where? I didn’t see it.

> Programming languages can save their data as unions or structs, and take the minimal hit to switch on the type. DB's persisting data on disk can do the same but will take a much bigger hit.

Yeah, of course. Disk is more expensive across the board. Same applies for storing ints, but databases don’t punt on that. And anyway, the sum types still exist in the schema, they are just implicit, as hoc spectacles built on the fly by the user. So you’re still dealing with performance issues, but they’re worse.

> On the DBs side, it does go away. The DB only has to guarantee what it says it supports (only store one specific type in a column). So for the DB implementors, that's a great invariant for their implementation ease and performance.

This applies to every feature for every tool. You don’t have to solve the problem if you just put it on your users.

Of course, tools have charters, and the relational axiom is that users shouldn’t have to manage their own storage and retrieval layer, but rather they should declare a data model and interface with it and the RDBMS would make search and retrieval fast and correct. Sum types are necessary in data modeling, so it fits clearly and neatly into the charter.

>Where? I didn’t see it.

"For one, because they need a fixed binary representation for the type, to persist it on disk. In a programming language you do things in memory, so you don't have that issue..."

The crucial difference I point is "persist on disk" vs "do it in memory", not in the "binary representation". Both running programs and DBs have one, but one absolutely needs to be persisted on disk, whereas live program memory doesn't.

>This applies to every feature for every tool. You don’t have to solve the problem if you just put it on your users.

There's also the fact that it might not be a problem just an easy cop-out from the user.

In which case it's better to force your users into the more formal and rigid structure, and have them rethink their model, than turn the DB into an "anything goes anywhere" store.

> The crucial difference I point is "persist on disk" vs "do it in memory", not in the "binary representation". Both running programs and DBs have one, but one absolutely needs to be persisted on disk, whereas live program memory doesn't.

Right, I agree, and my point was it doesn’t matter. Disk vs memory is a red herring. The same principles apply to both and the fact that disk is slower applies as much to product types as it does to sum types. In fact, sum types are represented as product types, but the system enforces invariants about the structure.

> There's also the fact that it might not be a problem just an easy cop-out from the user.

That’s a nope from me. Tools exist to solve problems. If a tool purports to solve a problem but only does it halfway, it warrants criticism or observation.

> In which case it's better to force your users into the more formal and rigid structure, and have them rethink their model, than turn the DB into an "anything goes anywhere" store.

RDBMSs are literally forcing their users into a less formal structure. You can’t rethink your model and make them go away (they are fundamental data modeling primitives), you can only find ways to hack product types to represent them, but you have to do all the work to make them fast and you probably just have to give up on verifiable safety altogether.

And how do you get from “sum types” to “anything goes data store”? Are you sure you understand the debate?

Reply on news.ycombinator.com