I bought my first property in 2000, when I was in my very early 20s, and definitely wasn't wealthy. I bought a house in Ann Arbor in 2004, when I had no savings and was living on an ordinary developers salary; another in Chicago in 2005 (don't do what I did) when were starting Matasano. We sold Matasano in 2012 and my credit score was bad enough then that I was still required to get a secured card despite a relatively enormous sum of money parked in my account.
I think EFTA covers the mechanism of how debit cards work, not the institutions that issue them, but I'm not an expert. I would lean towards keeping an account for the card I use in normal transactions at one of the Big Four banks.