I like TensorFlow with Keras much more than using DeepLearning4j, but DL4J is callable from Armed Bear Common Lisp. I have simple examples in my book that you can read free online: https://leanpub.com/lovinglisp Unfortunately the example in the book is a little weak because I wrote the deep learning code in Java and just called my Java code from Common Lisp. I should modify the example to build models, train, and evaluate using the DL4J APIs directly from Lisp.
EDIT: in the book example, I just call train and evaluate entry points in my Java code from Lisp so I can train new models from Lisp but can’t create new models with new architectures without modifying the Java code. I really should change that.
Java itself tends to be fairly verbose I know, but not only are there wrappers but for inference if you want to use python for basics and then us for inference. We enable several ways to do that. Folks tend to oversensationlize these "us vs them" framework "fights".
Rather than do that, why not just use what's good for each use case? Python for training, java for deployment?
Beyond that, you could also consider building your own wrapper maybe? In clojure, there's jutsu.ai for example: https://github.com/hswick/jutsu.ai
I use Keras at work so that is why I prefer it - I am very used to Keras, and to a lesser degree TensorFlow.
Just tonight I was experimenting with jutsu.ai, and your advice of training on Keras and running on DL4J is good advice. I hacked up a little code to import Keras weights into Racket, but I can only process dense layers, and not RNN or complex architectures.