VM has to have some communication bridge to the host, for all the features of the modem. That's still quite a lot of attack surface, once guest VM is compromised.
It's better to not expose the needless attack surface rather than just attempting to isolate the attack into a leaky VM.