Look at any comparable business and the talent makes a way bigger % of the money, actors in the movie industry, musicians in the music industry, professional athletes in NFL, MLB, NHL, NBA etc. These are all "jobs" that people would gladly do for low pay just like the Olympics if that was their only choice. Does that mean that the other industries are overpaying their talent? No imo. It's reasonable to expect that the people that provide the talent and a large part of the value to the industry deserve a large portion of the rewards.
Professional athletes and hollywood have strong unions to fight for fair pay(they used to be exploited for low pay), the music industry is a semimonopoly and does screw artists, but at least there is some level of competition. If you don't like one record company you can always switch labels.
The problem with the Olympics and what makes them so valuable to begin with is their rarity. They only come around once every four years. A top tier athlete may only make one Olympics in his/her lifetime and perhaps as many as ~three. Are you really going to organize and "holdout" for a whole Olympic games with your fellow athletes to maybe better the financial position for all your fellow Olympians? It's one thing to sit out a season when your expected career is 10 seasons and you may only have to sit out 1/3 of one season, it's entirely different when you may have to sit out your entire career. The athletes are being exploited because
1. IOC is a monopoly
2. Their careers are short
3. It's too risky to organize because the risk of not competing is too high. They've trained so hard to get there, to not compete is not really an option.
4. Olympics only happen once every four years.
It's a very similar situation as the NCAA except for #4. If you want to make it "not about the money" that's fine by me, make tickets to attend these events cheap or free, let any network cover the games(why does NBC get a coverage monopoly?), get rid of the endorsements and ads, make all the IOC "volunteer" administrators unpaid, but it's never going to happen. When coaches and administrators are making millions and the talent is making close to zero, the talent is clearly being exploited.