I am super impressed you were able to get the HomePods to work reliably as stereo speakers. We were never able to get it work consistently. One of the four HomePods I purchased is already bricked.
You may have to look at replacing your router. Because AirPlay works over WiFi, if the router is unable to keep up you will end up having a bad experience
That has worked flawlessly since day 1 in Jun 2020 for me.