I have a theory that simple sets can become difficult to understand not because of anything in their nature, but because of how they are desired to be consumed.

In other words, if you have a simple to understand thing, then it might become hard to understand if it needs to be fed into something that is itself hard to understand. (Or even if the way in which you feed it into the other thing is hard to understand).

Additionally, if you have something which is easy to understand that is fed into multiple incompatible things (either the things are easy to understand or hard to understand), then the whole system becomes harder to understand.

Finally, if the starting point that is fed into all other things is not actually easy to understand but in fact hard to understand ... things can get pretty bad.

Even if time was easy to understand by itself, then I still think it would be pretty hard to deal with because it is used for multiple incompatible domains (astronomical vs how long a day is on earth for example).