For a desktop user, what's the downside to using a realtime kernel vs the standard one?
Should audio be prioritized over the touchpad "moving" the cursor?
Yes
I presume most drivers haven't been tested in RT mode, so it's possible that RT-specific driver bugs crash your system.
A small impact on throughput is expected, but it shouldn't be noticeable to the user.
What the user can and will notice is the system not being responsive to his commands, as well as audio cuts or audio latency (to prevent cuts).
Thus PREEMPT_RT is a net win.