The protocol has been designed by taking the commonalities between all different casting protocols like AirPlay and Chromecast as a base. This is what has been used to make version 1.0. I tried making it as simple as possible and low effort to implement.

https://gitlab.com/futo-org/fcast/-/wikis/Protocol-version-1

As you can see the 1.0 protocol is very lean and fits on an A4.

If you want to have an audio-only ESP32 speaker I don't see why it wouldn't be possible with the FCast protocol.