The Impedance Mismatch Is Our Fault (2012)
infoq.com
infoq.com
Excerpt:
16:08
"And locks don't compose... I can't take two arbitrary pieces of code that each do locking, and then combine them together without knowing the implementation details.
Which means that the entire notion of reuse in OO is broken."
Never thought about that before, but it makes a lot of sense...
Now, in thinking about it, I'd change that sentence, slightly, to "the whole notion of reuse in OO is broken in objects which depend on databases and specific locking strategies or objects which depend on other objects that do", but the idea is still a very valid one...
To-do: Rewatch this video in the future...
Interesting that his wikipedia page doesn't seem to mention it. I'm a little sad that STM didn't seem to pan out, but maybe someone will try again someday on top of a Rust-like object ownership semantics (some sort of escape analysis for shared stated).
I suspect it's more true that the entire notion of reuse or composability is broken in code which depends on databases and specific locking strategies or code which depends on other code that does. It doesn't have anything to do with objects.
And yes, purity composes (because it requires no locks), and since FP pushes you to purify, FP helps. You don't have to do FP to have composability, but FP has great composability.
> I suspect it's more true that the entire notion of reuse or composability is broken in code which depends on databases and specific locking strategies or code which depends on other code that does. It doesn't have anything to do with objects.
Mutable objects impose locking requirements, and locks don't compose easily because lock-taking order matters (to avoid deadlocks). Yes, you can add deadlock detection and rollback and retry/abandon on deadlock, but it's a complication you have to handle because OOPs generally don't provide all of that.
But they compose in OO too, right?
> Yes, you can add deadlock detection and rollback and retry/abandon on deadlock, but it's a complication you have to handle because OOPs generally don't provide all of that.
But FP doesn't provide all of that either, does it?
If your ORM doesn't give you transactions, then you don't have transactions. But FP needs some kind of layer to talk to a database, too, doesn't it? And if that layer doesn't give you transactions, you're in the same place, aren't you?
Am I missing something? Or are you missing my point?
Somewhere, though, there will be locks, indeed. Pushing the locking to the edges of the system helps keep the core composable. There's no panacea, of course.
Also, an RDBMS can order locks because once you've decided what INSERTs/UPDATEs/DELETEs you're doing, they can be sorted and run in a different order than they were requested (after all, if the transaction is atomic, then the internal order doesn't matter).
You can absolutely build all of that (transactions, lock ordering, ...) in any OOP (or FP or other) language of your choice. But that's essentially recreating part of an RDBS -- if you're after reuse, just use an RDBMS so you don't have to recreate its functionality. Using an RDBMS w/o an ORM helps you avoid all the issues raised by TFA. Further, using an EAV design gives you many benfits as outlined in TFA and elsewhere in the commentary here and on the web.
> So, if I understand you correctly, the issue is with ORM, rather than with OOP
Well, there is certainly an issue with ORM. But OOP even absent databases and persistence, when used as originally intended (with objects abstracting state and behavior), has concurrency issues.
There is a duality between the much-hated EAV and table-oriented representations of data, especially if you are disciplined in your schema design so that the mapping between your EAV schema and a SQL table-oriented schema is natural.
I really would like to see automatic duality between EAV and relational models in RDBMSes. This would give users great query expressive power. EAV is perfect for "graph" queries, which are essentially recursive queries that chase relations without having to specify which relation types -- as long as all of them have the same shape, you can do this with EAV. That means that you can write queries like "give me all the things this user has access to", or "all the users that have access to this entity" while traversing arbitrary relations like "user belongs to group", "group belongs to group", "access grant", etc.
EAV is, essentially, the most normal form you can construct. And the more normal the form, the easier it is to apply CRDT techniques should you want to. Also, if you normalize the keys (see SQLite4[0], yes, 4, an abandoned project, but the docs are still there) then you get to trivially use any distributed ordered key/value store, and you can distribute the key ranges, and it's trivial.
Now, EAV is much hated. But that hatred is wrong. The problem is that RDBMSes aren't exposing a dual of EAV and tables. If you have an EAV schema you can create VIEWs that look like equivalent table-oriented schemas -- the reverse is also possible, but somewhat harder. Using VIEWs to bridge the gap imposes a heavy burden on the query optimizer. But SQLite4 and similar database designs essentially have EAV stores internally that they could expose with a lot less complexity than VIEWs impose.
We can haz EAV and table orientation at the same time. We should have it.
Plus, as TFA says, you can combine EAV with timeseries, which allows you to have time travel in your queries, and even transactions (e.g., run a transaction now that has an effect in the future), which makes your data naturally copy-on-write, which greatly improves read performance. Granted, you have to cleanup unnecessary history at some point, otherwise it piles up (vacuum), but that's OK.
Which is a cool model if you think about the world as observations of facts over time
It's a courtesy to explain an acronym when it is first introduced.
But that's a drastic over-simplification of time and consistent responses. Consistent responses in particular seems like a hard requirement given how much attention he's given to "why N+1 queries are wrong and you need to do them as a single operation" (which I totally agree with). And I don't think this is something that can be glossed over at all because the whole concept hinges on history being immutable... which means a major synchronization time bottleneck.
And while he does address this at 46:40, but he seems to treat inconsistent timestamps as "this is an interesting space, but if you have [inconsistent time in your db] you have a problem" as if it's a solved problem. Spanner has arguably the most sophisticated approach at scale right now, with atomic clocks all over the place, and it still uses MVCC to address the discrepancies because to do otherwise means major scaling problems.
---
There's a lot to unpack in here, delivered rapidly and concisely (yay!), and some good detailed chunks of info, but also what feels like a lot of "these are wrong and/or difficult for specific and largely-correct reasons" -> "this could be great because [hands waving furiously] it has none of those problems!".
Values are great, documents are terrible, programmability is king, I completely agree. But none of this sounds like it leads to a solution, just a "these are the tradeoffs we're making, how about we stop trading them off?". As a thought-provoker it's interesting, but it doesn't seem to directly lead to anything.
Datomic is not for Spanner-scale systems, it is an application database competitive with relational databases.
This means replicas are synchronized. Or you cannot have replicas. Whether it's a lightweight "transaction-only" replica or not doesn't really matter (I'm under the fuzzy impression that postgres's txids serve this same purpose?).