PulseAudio is already replaced by PipeWire[1] at Fedora Linux:
Latency is measurably better than PulseAudio even via pipewire-pulse.
edit: I think the confusion is over my use of the word "shim". Pipewire doesn't route through PulseAudio, it replaces PulseAudio but includes an implemenation of the same API. There's no increase in latency or complexity because there's no extra steps.