I’ve done this to make music remotely on the CLI from my phone SSHd into a VPS, forwarding pulseaudio/JACK and listening to the stream in a browser in the background. fun times
I've experimented a tiny bit with the builtin pulseaudio RTP sink/source, but I've not used it extensively. I'm not sure if cutting out ffmpeg would be beneficial.
Managed to get the lag down to a usable amount for algorithmic music, but would not be suitable for playing an instrument remotely