If you move to one time zone, dates get complicated as well. You may have meetings scheduled on separate dates but in the same business day. Birthdays: they're now the afternoon of one day and the morning of the next.

Furthermore, while travelers may not have to change their wristwatch to account for the local time, they still would have to grasp "what time the locals do X." This may be more difficult for people to reason about if they can't easily draw on their expected scheduling from "back home." Simply knowing that 8:00 to 5:00 is a common work-day will no longer be easily translated, as it may be 11:00-20:00 one place and 05:00-14:00 somewhere else.

In any case, people are creatures of habit, so they tend to fight change. So I don't foresee this ever happening, regardless of whether it would solve some problems (and debatably it may cause as many problems as it solves).