This is reverse survivorship bias. There were probably bolides a'plenty, it's just they took down some trees, killed some animals, and interrupted a villagers sleep. There would be no record of such an event.
My point is that human cities are "bolide sensors" and unevenly distributed in time and space, so the "measurements" are necessarily fewer than what occurs in nature.
Then there's bolides over desert, savanna, tundra, ice sheets, etc.
An air burst impact that’s large enough to take out a city won’t leave a major impact crater. And back in the day it might even be the case of the perceived cause contributing more to the decline of a city or even an empire than the physical damage.
If a meteor hit blows up your town when you have no ability to comprehend what just happened other than god/gods are angry you gonna move to a less cursed place.
It can also cause social impacts such as the toppling of a given religious or leadership class because they angered the gods.
And not even that guarantees a complete picture - what if a tsunami event erases or conceals the record of something like this? Volcanic activity? Desertification?
My point is that we can't reason about the rarity or uniqueness of these events as a main factor for accepting or refuting the hypothesis