In my limited trials it was not terribly performant, running at 15fps or so only.
Use the NN to find the faces, and the non-NN to track faces once the NN has found them, and use the NN to periodically check to see that the non-NN is still locked on.
Suppose the NN is only 25% of the speed you need to support the frame rate you want. Then every time you get a new face blob list from the NN, the non-NN tracker would to track the blobs for 3 frames. My guess is that in most common photography situations where you need face detection, faces won't move very far or won't change orientation or lighting very much in 3 frames.