IIRC correctly, unlike HDDs, (audio) CD players change their rotation speed to match the data density, because they need the data at a constant bit rate. So it wouldn’t matter whether the data starts inside or outside. I think your argument about physical robustness is very plausible.
Additionally, starting from the inside allowed for physically smaller disc formats without any special software support: https://en.m.wikipedia.org/wiki/Mini_CD
I don’t know for sure if that possibility was part of the original design. But it must have been introduced somewhat early, given that every CD player tray I’ve ever seen has that extra ridge to hold the smaller discs in place.