What we learned open sourcing a major part of Mailgun
blog.mailgun.com
blog.mailgun.com
I've also found that integration tests are a great thing here. If an internal application depends on certain behaviour, it's not likely that you'll have a test to begin with, but when the open source product conflicts with the internal, it's a perfect place to make integration tests to make both sides more stable.
I'm not a complete believer in all kinds of tests at all times, but I've found that the more extensive your tests are around things like this, the less breaks in the long run.
The only thing not covered was that as a result of using a public codebase at the core of your app you've got to be extra careful in checking what's in each pull request... probably moreso than if they came from your employees.