There’s an app called Micro Snitch [1] that will alert the user when any application activates the microphone or the webcam and tell the user which application is doing it (including Apple’s apps like FaceTime). You could try it to confirm that it is indeed Signal that’s activating the microphone at all times and file a bug report with Signal. Listening in the background is a very serious issue because we don’t expect Signal to do that.
Edit: FWIW, I just remembered that I have Signal and Micro Snitch on an old MBP (running the latest macOS). I have never seen alerts for microphone activation when Signal is running (I don’t use it for calls from the computer). Nevertheless, if the coreaudiod issue is with an app using the mic, Micro Snitch may help.