The direction is purely philosophical. Things are changing from one state to the other. Even if it was backwards, you'd still convince yourself that it is the forward way. Things simply change.
Causality, I guess would be my answer there.
But in your view of time, what is causality? Can it even exist?