I guess one limitation of this approach is that one can't login to anything from a VM that is running on a server that one has no physical access to - i.e. no way to plug the USB key in.
There are one-time codes and and there is support of remote keys over SSH (in most simple from this key just pretends to be USB keyboard and does typing of the code for you).
Most remote desktop apps have a form of USB passthrough. And I believe you can also do this in a variety of ways with ssh plus some combo of libusb.
I'd say that is a feature rather than a limitation.