I'm not sure if I'm barking up the same tree, but my thought process has been something like this: A Minkowski diagram plots distance on the 'x' axis, and time on the 'y' axis, and the "here and now" is at the origin in the centre. The past that can affect that central point is a right triangle with the apex pointing up at the origin. The future spacetime that the "here and now" can affect is an upside down triangle with the apex starting at the origin.
This gets interesting if you think of the past as a collection of "states", such as boundary conditions, and the area of the triangle as some "function" over the states. Think: hash function with some subset of a file as the input.
Now, imagine there are two points, one for each observer. Two scientists, say. There's one at the origin, call him Alex ("A") and another scientist in his "past light cone", call him Bob ("B"). Their states are like the hash function, and the inputs are the boundary conditions, like different but overlapping subsets of a file. They'll disagree because they don't all see the same input bits.
From the perspective of Alex, Bob is in his past, and Bob has only experienced a subset of the boundary conditions. In the physical universe, this is the big bang. Alex can, in mathematical principle, but maybe not a physical one, predict everything Bob will say about any experiment Bob will do. But Alex also knows that Bob is missing some information, the greater part of Alex's past light cone not shared by Bob. So Alex can make predictions that Bob cannot possibly predict. The situation is inherently, fundamentally asymmetric!
Here's the rub though: from Bob's perspective, Alex is in his past, and it is Bob that knows the entire state of the universe from the big bang leading up to his here & now, and it is Alex that is ignorant of a "slice" of it.
So what's going on?
The reality is that there are 4 observers: {Alex Prime, Alex's Bob}, and {Bob Prime, Bob's Alex}.
We colloquially talk about them as if there are only 2 observers, but the more accurate perspective is that there are 2 observers in 2 universes, and they can never completely agree on all experiments, not even in principle. They have different inputs, different boundary conditions, and different predictions.