I've spent the night reading Tandem tech reports and manuals. I think the system you have described is very interesting, and it's maybe the system Jim Gray would have liked to build, but it doesn't seem to be the system they actually did build. The system they did build is a lot like Unix, except that it lacks the uniform hierarchical filesystem namespace that is the heart of Unix. (And, maybe more fundamentally, Unix is first of all tasteful; Guardian/NKS is completely lacking in taste, and everything about it is fractally ugly.)
The programs running under the Tandem operating system/Guardian/NonStop Kernel (they couldn't make up their mind about the name) did not, in general, start, do one transaction, and exit. Instead, they were long-running processes very much in the Unix mold. They even had suid (which makes no sense), a superuser, and zombie processes.
The filesystem has what looks like an ISAM library, called Enscribe structured files, built into it, and pretty decent locking. I guess that's what you're calling "a database system". Also, there are SQL files as an alternative to Enscribe files.
There is some really interesting stuff going on in interprocess communication for fault-tolerance, but it's not transactional; instead, it has to do with process naming, sequence numbers, and copying around memory space images between process pairs running on separate "CPUs", so that if one of them fails, the other one can pick up from the last checkpoint. The IPC mechanisms get involved in order to replay any responses to requests that the failed process made between its last checkpoint and the crash, which is what the sequence numbers are for. Every time you open a file, you have to declare how many messages that file might need to replay in case of failover (the "sync depth"). The IPC is kind of vaguely Symbianish, really.
I put "CPUs" in quotes because each CPU had its own memory space, communicating with the other CPUs by sending them messages, not by mutating shared memory. Shared memory is death to fault tolerance. CPUs within a "node" or "system" were numbered, like cattle; nodes were named, like pets.
There was also distribution between nodes — you could access files or devices or do IPC to processes on any other node very nearly as simply as on your own node.
They did have better IPC and locking than Unix, and additionally there's this transaction-management facility called Transaction Management Facility that "audits" selected files by snapshotting data that's about to be overwritten, and allowing you to roll back in case of a transaction abort. I don't fully understand the TMF programming interface and semantics yet, but it definitely wasn't a systemwide universal thing applied to all files, and it definitely didn't need your program to exit in order to commit.
One thing that makes this stuff super hairy is that it has to handle transactions that include activity in multiple separate processes because of how Tandem encouraged you to structure your applications in a requester-server way, with one requester possibly talking to several servers; the server needs to associate its updates with the requester's transaction so they can be rolled back if need be. And at any moment any of the computers ("CPUs") involved can fail. But Tandem didn't have to deal with the really hairy part of that hairy problem, which is that when a computer fails, you don't know if it failed or is just slow — they had a 300ms heartbeat from the beginning.
About the hardware architecture, you say, "the system had to be ported from Tandem to MIPS to Alpha to HP-RISC to Itanium to, finally, x86." As far as I can tell, the path was their own custom HP-3000-ish hardware ("TNS") → MIPS ("TNS/R") → Itanic ("TNS/E") → amd64 ("TNS/X"). I can find no trace of Alpha, PA-RISC, or i386. Also, I don't think the issue there was acquisitions; they were still independent when they switched to MIPS, but then MIPS got bought by SGI, a direct competitor and maybe not the stablest company, and anyway Itanic seemed like a good bet before everyone saw the iceberg. And certainly deserting Itanic for the amd64 lifeboats can't be blamed on acquisitions.
They claimed from the beginning that their hardware cost was comparable to competitors, but maybe they were lying. And surely their hardware cost didn't fall as fast as Google's did in the 2000s. So maybe even if they weren't lying, they might have become wrong.
Still, I'd propose as an alternative that their main problem was that they weren't IBM and, unlike e.g. Amdahl, they didn't run S/370 assembly.
What do you think? Am I missing the forest for the trees?