Surely if someone managed to factorize a 3 or 4 digits number, they would have published it as it's far enough of weaponization to be worth publishing. To be used to break cryptosystems, you need to be able to factor at least 2048-digits numbers. Even assuming the progress is linear with respect to the number of bits in the public key (this is the theoretical lower bound but assume hardware scaling is itself linear, which doesn't seem to be the case), there's a pretty big gap between 5 and 2048 and the fact that no-one has ever published any significant result (that is, not a magic trick by choosing the number in a way that makes the calculation trivial, see my link above) showing any process in that direction suggest we're not in any kind of immediate threat.
The reality is that quantum computing is still very very hard, and very very far from being able what is theoretically possible with them.
If results in quantum computing would start to "go dark", unpublished in scientific literature and only communicated to the government/ military, shouldn't he be one of the first to know or at least notice?