It's not much weirder than things humans get up to, and if spacefaring civilizations that can build such probes are common at all (even if short-lived), you might expect at least one of them to have initiated such a project and been successful enough to have built a giant "we wuz here" radioactive pyramid on the moon. Evidently, none of them did- and depending on how many there were, this is either unsurprising or deeply unlikely.
The Fermi problem is a speculative theory based on extrapolating human civilization's use of energy to stellar levels (various hypothetical civilization "types" etc). It's plausible life at the scale of earth could be common, based on the principle of our natural non-uniqueness. So the simplest explanation is the speculative extrapolation to stellar energy uses is wrong - human already are more destroying themselves with excess energy use than preparing for any next stage, just as one datum.