Yea, it is such weird behavior. I can count the number of times I've ever wanted to adjust the volume of the ringer, rather than just turn it on/off, on one hand.
Luckily on newer version of Android there is always an option to quickly access the media volume after you press the volume button once, but it's still unintuitive.