You should look into the history of the Trans Iowa race[0]. It was a single "day" event (if you don't hit certain checkpoints in time, you're out) and was usually around 330 miles (530 km)...on gravel roads of all qualities. There was a separate division for single speed bikes. It was insane.

Not to dismiss RAGBRAI, though. It is a different kind of challenge - biking 50 miles in a day, especially with frequent stops, is no big deal. Sleeping in a tent, waking up, and doing it every day for a week especially with all of the nighttime activities is a different kind of challenge.

[0] https://www.cxmagazine.com/trans-iowa-2018-gravel-race-14-fi...