Not a physicist but this echoes my feelings when people talk about time as an emergent phenomenon.
I do agree that you can go quite far with calculus, linear algebra, and probability. But I do think that you overstate the case.
¹https://webhomes.maths.ed.ac.uk/~v1ranick/papers/eisenbudhar...