But even then, this would imply that our forward/backward time direction is just convention, much like left/right space directions.
Yet, isn't it strange that all human cultures have developed the same notion of the forward time direction? Wouldn't we expect some islands to have developed the other convention, just like there are some places where cars drive on the left rather than right side of the road? Or, is it our human brain? Then, shouldn't we observe some animals or plants with an apparently different perception?
More importantly, however, this disagrees completely with physics, ancient as well as modern physics, because of, well, experiments. [1][2] The forward/backward time directions are more comparable to up/down space directions (in a gravitational field), which are very distinct from each other. The direction of time can be measured in so many different ways, almost none of them really connected to our personal perception.
It's the whole world around us that makes a clear difference between forward and backward time direction, not just we in our heads.
[1] Those led to the discovery of the second law of thermodynamics, which modern physics does question in its fine-grained details (as with gravitation, etc.), but not in principle, because it is so clearly visible in experiments. https://en.wikipedia.org/wiki/Second_law_of_thermodynamics
[2] See also: https://en.wikipedia.org/wiki/Arrow_of_time