Have you considered using an off the shelf differentiable programming implementation? Or are the requirements for real-time applications too demanding for existing software?
The real-time requirement is fairly hard. The robots I work on use a 1 kHz feedback loop, so it has 1 mS in which to recalculate everything.
Caffe is pretty efficient if you can supply batches of values, but the overhead is high when running real-time.