Is the pool smaller because you primarily rely on referrals from your (already not especially diverse) workforce and applications from people off the street? Are you recruiting from universities that are predominantly white, as a result implicitly biasing your available pool of recent college graduates? Are you recruiting only from your local area, subjecting yourself to the local race balance?
If hiring diverse candidates costs more money, that can absolutely still be worth it if hiring diverse candidates provides measurable upsides. Of course, the cost matters - if it's 10x or 100x it's much harder to see a positive return. But dismissing it for being 'too hard' or 'too expensive' without actually doing an evaluation of the pros and cons is a little lazy, isn't it?