My own intuition tells me that everyone probably has a maximum (and maybe minimum) speed that they're comfortable with, and there's no reason to assume that a podcast host's preferred speaking rate is the same as an audience member's preferred listening rate.
Speeding up your primary focus is not at all comparable to consuming something as a secondary focus. I agree that "watching" something in the background is terrible for comprehension and retention.