OP should have said that change in state = time.

Anyone who has worked on secure consensus has had the same thought. You can't derive time from pure math. You must observe some state change.

It's a useful concept, but strictly speaking it's not true. If this universe was completely devoid of matter, time would still pass. "If a tree falls in the woods and nobody is around to hear it, does it make a sound?"