Arguably, the appeal is how much money it brings in, and the economic benefits of people being paid to build all this and all of the tourists traveling in? Beyond the general prestige, of course.
The only way the host country makes money from the Olympics is if tourists spend enough money on lodging, food, etc. That pipe dream hasn't worked out well for several countries who've recently hosted the games. The deal is so bad for the winter games that few countries are bidding it anymore.
It's only recently that more people are realizing that the numbers don't usually add up. Even then, lots of people are so passionate about sports that they simply don't care about how much public money is poured into a black hole to host these events (or build stadiums). And that's not even getting into the corruption.
Building a gigantic temporary stadium does not.