One counterargument is that hosting the Olympics seems to be an important prestige signal for nation states. Think Beijing 2008.
I mean getting a favourable mention in the US state of the Union address[0] is also something that would be prestigious for nations, happens every year and is more under US control, so wouldn't that be better?
[0]: When American politicians mention smaller countries it becomes national news in those countries. I doubt the US news spend much time talking about a state visit from Denmark, but Danish news does.