More customers could arguably give you free word of mouth marketing, but fewer "qualified" customers could give you better leads.
Lower-paying customers almost always are more demanding, less appreciative customers. I've seen this in my biz (infoproducts, courses & SaaS), heard it from friends in theirs (SaaSes, restaurants, training companies etc). It's a truism probably because it is largely true.
As you brought up in your last sentence, there's also opportunity costs of going with a lower price.
You've also taught those customers to only pay you $3-6, which means you may have a harder time selling them on fancier products later.
Finally: Especially with infoproducts, when it's a no-brainer, people are likely to buy on a trigger and then never use what they bought. If your goal is to help people, change their minds, build a community, this can be a kiss of death.
I think it matters a lot WHAT you're selling.
At $500+ a head for a day, our JavaScript workshop is more expensive than just about any other way of learning JavaScript. But we get almost exclusively rave reviews, because that price keeps out anyone who's not really committed.
And I've found that, as I charged more for my product class ($500 -> $1250 -> $1550 -> now $2450+), people have become MORE enthusiastic, MORE involved, MORE likely to finish, MORE helpful to their fellow alumni, because the price is a filter and it's a kick in the ass. (When you only pay $500, you don't feel the pain so much when you let it fall by the way side. But nearly $3k is a LOT to sacrifice to laziness.)
Obviously, this wouldn't work if I wasn't great at teaching. And I am great at teaching. But I've watched the enthusiasm level increase with the prices, rather than the other way around, even though I, the teacher, haven't changed, and neither has the material all that much.