Licensing and software patents!
Put simply, you can afford to license some proprietary format from a complimentary product that people use (ensuring that it is one of sufficient complexity such that will take months to reverse engineer).
Make your software compatible with any number of other proprietary software packages, using these proprietary formats and protocols that open source developers will never have access to without expending huge amounts of time and energy. Then, to really firm up your market share, be sure to invent some more proprietary formats and protocols that future developers can use to integrate with your product. Be sure to share your technology for free with other complimentary products so that you can claim that it's a "standard". Then, encumber your proprietary "standards" with the sorts of licensing or NDA requirements that put them beyond the reach of open source projects.
I'm not even being totally sarcastic here. If your entire business model is dependent on being able to compete with open source, then I think utilising all the hard work that previous companies have put into achieving vendor lock-in is more worthwhile than a good installation process, a marketing budget, or having more features than the other guy.