Disclaimer: I've only worked on lower parts of the stack so my experience is biased. It's mostly the concept of "garbage in, garbage out."
There's a simple fix for most audio problems:
- use wired headphones
- use a wired microphone
- disable all DSP in your conferencing platform (echo cancellation is the big one, AGC helps as well)
Almost all the fidelity problems in audio conferencing stacks comes down to crappy microphones, wireless codecs/latency constraints on wireless audio (almost all bluetooth headsets, it doesn't really matter how much you spend, the only model I've heard that doesn't wreck fidelity are Earpods), and DSP that is intended to remedy environmental problems and compensate for the aforementioned crappy microphones and mic conditions.
> It would seem sensible to switch to audio-only when bandwidth is limited
Audio doesn't need that much bandwidth to be perceptually lossless. The bigger problem for conferencing encoders is latency.
I'd also add that the "garbling" sound is definitely different compared to how poor fidelity manifests in analog telecoms/radio. Before you'd get nonlinear distortion (frequencies that don't exist, but are correlated, not a huge deal), low bandwidth (the human brain can interpolate missing frequency information quite well) and wideband noise (human brains are good at dealing with this).
The garbling sound you hear in crappy DSP, including encoders that are doing their best to discard information that doesn't matter and echo cancellation doing realtime adaptive filtering (which is like training a neural net as you evaluate it!) will manifest in really strange ways where small amounts of temporal/phase coherence is lost.
I'm not an expert on perceptual codecs but my guess is that overlap-add/save or other time/frequency based analysis/synthesis algorithms that can lose phase coherence between frames will wind up mutating the edges of some phonemes in human speech in a way that our brains can't interpolate what was said, and its a very alien kind of distortion compared to harmonic distortion, low bandwidth, and noise which all occur in nature.