One could also break the Olympics up, and host the gymnastics in one city and the swimming events in another etc. Then it wouldn't be such an overwhelming financial/infrastructure burden for any particular host.
Although to be fair, most of the multi-sport athletes complete in closely-related sports (which would likely be held in the same location) or are from the old days when tug-of-war was an Olympic sport.
https://en.wikipedia.org/wiki/Ester_Ledeck%C3%A1
Obviously those aren't as different as swimming and handball though.
Honestly, I don't think the idea of splitting the Olympics over several sites is a bad one. The advantages (less construction at each site, more ideal conditions for each sport) might outweigh the disadvantages (less opportunity for multi-sport athletes, less interaction between athletes from different sports). I always like seeing the USA men's basketball team show up to watch the USA swimmers, but it wouldn't be the end of the world if that tradition doesn't continue.