Yeah, you're not going to get FM quality from 4kbps, so it's exactly like I said.
It might sound better to you because you're only listening to the channels they allocate the maximum bitrate to (i.e., the most popular ones). I don't listen to pop, so the stuff I tried listening to (hard rock and metal) sounded like total shit.
Here's a test for you: rip your favorite CD into FLAC, then encode this into a bunch of different MP3 or AAC streams of different bitrates. See what they sound like at 32kbps or worse. If that bitrate actually sounds fine to you, you might want to visit an audiologist and get your hearing checked. 64kbps will probably sound OK, depending on the codec (Ogg Vorbis and probably AAC will sound noticeably better than MP3), but nothing is going to sound good at 32 or less.
In short, XM does not have enough spectrum available to stream all those channels at a high-quality bitrate, so they devised this wacky scheme to allow them to broadcast some things at higher bitrates than others. While you don't need a high bitrate for talk radio of course, they also relegate many other music channels to lower bitrates, resulting in poor audio quality. Why would I pay $12/month or whatever for this? These days, if you insist on streaming music, you're better off just using your phone with a high or unlimited data plan and listening to Spotify or similar.