This assumes the people fixing the clock know how to tell time based on the movement of the sun.

In the story, the baker was the only one left with that ability, as others left the skill behind when the clock became the authority.

This problem has been seen when missionaries, or whoever else, try and bring tribes into more modern civilization. They have trouble adapting to modern civilization, but the skill loss during that process makes it almost impossible to go back to the jungle.

When skills are lost due to modern technology, it isn’t so trivial to get them back. Look at how much time people have spent trying to figure out how stone was moved in ancient Egypt, for example.