Colloquially they were always just referred to as Oggs, which wasn't too weird to my ears.

To my recollection what hampered uptake was simply that most software and especially most hardware did not support Oggs. So no-one encoding for distribution (i.e., P2P filesharing) used it. And once commercial streaming services came into being, they all used proprietary DRM-backed formats (WMA, and whatever iTunes used at the time, AAC?).

By the time digital music services gave up on DRM, MP3 patents were coming up on expiry, and MP3 encoder technology had advanced closing most of the gap between it and Ogg, especially when run at higher bitrates which cheaper storage and bandwidth made acceptable to use.

But now with many streaming services using Opus, all is right in the world again.