That CPU use is resampling to match the sound card rate. If PA isn't doing it, your apps will be instead. Either way, it's going to take some cycles.
You were right about that, I tested it in another comment on here. I thought that resampling would happen within a kernel driver or something. But no, apparently when an app uses the alsa library, the functions that resample will happen within the app generating the sound (and thus in userland).