It's probably a good bet that we are going to discover that to rectify the complexities of QM, there is some assertion most people have fundamentally accepted and is quite uncomfortable to let go but is necessary to simplify the equations. I can't really guess what it's going to be though.
Saying "we don't know what we don't know" is both a tautology and completely useless. It's like my saying: "I believe that you committed a crime last week." You ask for more details about why I think that and what the crime was, and I reply "we just don't know." I hope you see how ludicrous that is.
Re-framing the whole system to be heliocentric allowed for the massive simplification of Kepler's Laws (and, not too much later, the further simplification of describing Kepler's laws via Newton's law of gravitational attraction).
We currently have an understanding of QM that doesn't reconcile well with macroscopic observation and requires a lot of intuitions to be broken (including, possibly, the singular nature of existence, time-forward causality, or lightspeed-constrained information locality). I can't help but wonder if there's some equivalent to "Assuming Earth is the center of the coordinate frame" that we currently do that forces these unintuitive (though working) solutions.
The solar system moved in epicycles because "it just did," until several leaps of intuition allowed us to see how it didn't. I wonder if there are similar leaps of intuition waiting on the horizon to allow us to explain quantum entanglement without nonlocal information sharing (hidden variables, we have shown, is not it).
Why should physics care about human "intuition" at the macroscopic level? That's an incredibly anthropic point of view to take on a cosmos that couldn't care less if we didn't exist.
What I see in QM right now "smells" like we're having to compromise too many intuitions to reconcile what the experiments are showing us with the way we expect the universe ought to be working. I agree that the universe is under no obligation to work the way we expect it to---problem, what we have right now in QM is a lack of agreement on how to even rework our intuitions to match our observations and the reality the theory describes. That suggests to me that---much like epicycles, or like gravity being separated from acceleration before relativity---the stories we are telling ourselves about what we see are still too complicated, and a simpler explanation that requires us to sacrifice fewer pieces of intuition has not yet been reached.
This is, to be clear, extrapolation. I have no way to know whether such simplification exists or whether we have hit the boundary where nature actually refuses to conform in such a way to our senses and minds that we can change our intuitions to follow her. It's not impossible that this is the case. But if it is, it's a break from the pattern of physics discovery up to this point in human history.
We have intuition to understand QM, people struggle with myths, not lack of intuition. Absence of conservation of energy is one of those myths, only those myths need to be sacrificed.
if they are just a story, it feels like a complicated story to get to the goal.
At the same time, I completely agree that the story of wave function collapse also smells. It creates a distinction between observer and observed reality, and that creates its own set of paradoxes (we can put Schrodinger's cats in nested boxes all day long, and some of the boxes even include experimenters who think they are observing the cat, but experimenters outside the lab don't agree with them that they are the observers!).
Looking at the multiverse as branching is where the confusion comes in. There's one Universal wave-function. Why we don't experience the wave-function directly is the mystery.
It's much more plausible that the uncomfortable human-centric thing we need to let go is the idea that our perceptions about macroscopic reality should be indicative of how things actually are.
EDIT: To stretch the metaphor a bit more. Maybe QFT is the analogous of the solar system described with epicycles.
Relativity tells us things get weird at high velocities, but our daily perception is basically correct. Quantum theory seems to suggest that solid, tangible reality isn't at all like our day-to-day experience. That's a pretty decent leap to ask of people.
So, maybe it's time for another round. ;-)
Same with Schrödinger equation: it's very simple for one particle in an empty Universe, but things go more complicated when more particles are considered.
There is no guarantee that a different approach can replace QM and describe things in a simpler way. But I suspect that attempts to look for such approaches are not useless.
I'd love to see a derivation if it exists.
I really think Sean Carroll's ideas involving MWI and emergent spacetime from hilbert space provides a great deal of clarity.