The biggest pain point seems to be car rentals as they seem to have constantly shifting and strange rules about whether and when they accept debit cards. The next seems to be car insurance discounts as insurers seem to have strangely connected your driving risk with your credit score.
All the banks that I spoke with would concede in person that I am an extremely low risk, but they all intended to sell my mortgage on the secondary market (but still do the servicing themselves). So, they wouldn't / couldn't give a mortgage to someone who doesn't fit the "normal" boxes.
Finding a bank that was willing to not sell the mortgage took a bit.
It's not very rational, since I'm perfectly happy to leave a deposit higher than some people's credit limits.