So, while the math is reversable, time is a one-way street by virtue of the construction of the physical universe, which is not reversable. And it's that way because that's how this magnificent universe was instantiated.
It may sound trite, but sometimes Occam is bang-on. As well, there are endless unknowns, but there are also unknowables.
All matter is >99% empty, not counting field interactions. I don't know how much of it can be understood, but some of it is likely to be unknowable.
However, say we receive a video of a billion billion such masses all starting in a single point, staying more or less still for a few seconds, and then moving out at high speed away from each other. It is obvious that this video is almost certainly playing forward in time, since the reverse, a billion billion balls all coming together to form a single object, is very very unlikely.