BitKeeper actually prohibits their customers from contributing to competing projects?
Using the product to reverse-engineer a protocol to develop a free client is not 'contributing to a competing product'. Which is against their license (though that may be void because the law grants you reverse-engineering rights, no matter what a license says. Otherwise nothing could ever be reverse-engineered at all).
they were afraid that he "might be moving BitKeeper technology into Mercurial."
That's entirely reasonable. If you have deep knowledge of how proprietary system A works and are hacking on free system B that purports to do the same thing, there is every chance you inadvertently include proprietary knowledge into your code for the free product. It's very hard to avoid using the knowledge you have.