* By necessity there's a pretty large attack surface, you've likely got a vendor specific bluetooth stack and coprocessor. The same goes for wired solutions with hardware USB peripherals (though atleast significantly less complex). Being sure the hardware peripheral doesn't have any major memory safety issues is a complete bear to test, and that's not even touching on sidechannel analysis.
* Running public key crypto on microcontroller (excluding specific hardware crypto support) is a little more novel than a desktop machine, the leading libraries are no where near as extensively battle tested. And there's architectural concerns about sidechannel attacks and often the lack of any memory protection units (bigger chips solve these issues, but power budget concerns are tricky).
* And given all software has bugs, how do you update the firmware? Signed firmware patches as part of driver update? Who's to say somebody hasn't already owned your bootloader.
I think the solution to all this, is to abandon the idea of additional hardware, what if your smartphone could act as a two factor hardware token over Bluetooth? The big problem is pairing really, and how to make this an easy process for users, I never figured that out. On the other hand I did come up with a scheme that would allow usb based hardware tokens to work without additional drivers / software and on all current major operating systems and browsers. I really ought to work on it, but as above I don't trust embedded hardware (embedded hardware is such an antithesis to, move fast and break things).