If you really don't want Facebook listening to anything you say at any time, you can turn off the app's access to your microphone.
In iOS, go to the Settings panel, find Facebook, and slide off the "microphone" option.
On Android, go to "Privacy and Safety" in Settings, find the microphone section under the app permissions panel, and toggle off Facebook's access.