Actually, a state is when a measurable attribute of an object remains unchanged for a period of time. So saying state = time completely misses what a state is.
Anyone who has worked on secure consensus has had the same thought. You can't derive time from pure math. You must observe some state change.
It's a useful concept, but strictly speaking it's not true. If this universe was completely devoid of matter, time would still pass. "If a tree falls in the woods and nobody is around to hear it, does it make a sound?"
If the universe was devoid of matter, it would mean there would be no us. And since time is a human concept, that concept would not exist :)