On mainstream distros with pulseaudio like Fedora or Ubuntu, audio just works, when you don't have low-latency requirements.
When you do have low-latency requirements, things are a bit tricky. You do pretty much need to get a low-latency or realtime kernel, and you definitely want to use Jack or ALSA. Maybe this setup is a little more frustrating than Windows, where you may just need to install one driver like ASIO4ALL.
But it's flexible, and you can actually get Pulseaudio and Jack working pretty well together after installing the pulseaudio-module-jack. You can turn on jack when you need it, turn it off when you don't, and all of the pulseaudio stuff will get routed through it so that you don't lose sound from your other applications. If you want, you can route audio from pulseaudio into whatever other audio applications you're using - sometimes I like taking Youtube videos and routing the audio through weird effects in Pure Data.
I toggle jack on and off with this little script, works pretty well for me right now: https://gist.github.com/YottaSecond/f0a1b515f95b2e791755