On Mac, you get that insane latency without having to put a moment's thought into any part of your architecture. I get that insane latency even if I never bothered to learn why latency matters.
"Just works" is more than ease of use. It's productivity. It shows respect to your mental load and mental priorities. And it is worth money to anyone whose time is valuable.
The plus to CoreAudio is that any system it runs on has probably been designed so that a non-technical user can get something like the low-latency you're describing by default. The minus is that hardware probably costs at least $800, and CoreAudio doesn't support things like the RPI.
The plus to ALSA is that it runs on things like RPI, with the minus that non-technical users probably won't get round-trip latency below 5 ms without paying someone Mac-level money for hardware designed especially for Linux audio.
Edit: and it's possible to go much lower, for example look at the linux-based Bela: https://bela.io/about#why-latency-matters