Are your paying clients the same people clamouring for open-source?
If you customers are already asking for it to be open-sourced, or if you have partners in mind who will support this endeavor, this is a good start. If, on the other hand, you open-source without a good PR push to partners/customers/developers it's like a tree falling in the forest; will anyone be around to listen?
We open-sourced from the beginning and then found clients; it was a long haul. We eventually found a toehold in developing countries, and were able to build up client credibility from there. If you already have a plan for using the OSS to sell right away, then maybe this plan is for you.
However, non-customers can be a lot of people which would spread the name by word-of-mouth. Also those customers who intend to buy it will get a feeling of what they are getting, instead of you describing a black-box with significant marketing efforts.
However, if you feel the potential to gain traction outweighs the threat of losing an edge (or a paying customer, since they can just use your code), open-source the engine, but hold back on the parts that are unique to your offering and/or make your engine easy to work with (like a GUI, simplified wrappers or builds for different platforms), though this depends on the nature of your product.