Imagine you have a 3 digit PIN code. That’s 1,000 options to brute force. If you remove 000-009 as “too weak”, you still have 99% of the search space available. These numbers scale favorably as the private key size gets bigger.
In short, raising the floor doesn’t really hurt the search space meaningfully.