"The distribution part of software can be rather trivial for niche markets where there aren't any big vendors around - with GPL I'd have to assume that I can only sell once and then the customer will just redistribute it - whether free or not - to whoever wants it, at the very least within his/her organisation."
Many enterprises won't buy something at all unless there's a price attached presented by a company they think will be around a while and help them with any customization. This market is less likely to trust a competitor to Windows Server or Exchange if it's being offered for $50. ;) Plus you can point out it's better to buy the real thing supported by developers that invented it instead of cheap knockoffs. Call them cheap knockoffs.
So, there's some risk but it can be done. Best model is having some proprietary addons, though.
"I think it would actually encourage to have a nice long thought about what dependencies you want to introduce. "
I thought about this as I wrote it. The reason I didn't add it is because you'd have two, competing solutions: FOSS community that mixed and merged no matter the size; shared source community where things cost more as complexity went up. I figured the latter would be largely boycotted if former was available. The former has so far been fine for vast majority of freeloaders. I doubt that would change. The latter could become a niche market with low complexity a differentiator, though.
"Yes, to me also this seems to be the best way forward with commercialising OSS without going the service route."
It's what I'm going to do. If money rolls in enough, I'll also submit select improvements and/or donations back to the source projects. Just on principle.