There's a good chance the famous sky-blackening flocks of them weren't a regular occurrence, historically speaking. North America was in ecological chaos after the Columbian exchange, and booms of some species are a common occurrence in such situations.
Why were specifically the passenger pigeon hunted? Why not the pigeons we have now
The typical pigeon you see in cities are rock doves which are introduced. There probably weren’t as many when they had to compete with passenger pigeons. Mourning Doves are pretty similar to passenger pigeons and are very common too, they are still hunted to this day.