The reasons you listed are exactly why the lack of factorization records should not be seen as a "critical black eye to this field", because they are not a relevant measure of progress. Again, think of the parallel with LLMs: it took decades to get out of the "AI winter", because that's what non-linear technological progress looks like.

With QC, the risk (and I am not saying this is going to happen, but I'm saying that it is a non-overlookable risk) is that we end up transitioning from "QC can only factorize 15" to "RSA-2048 is broken" in such a sudden way that the industry has no time to adapt.