I wonder how performance sensitive this code has to be? Clojure is a pretty bad choice for numerical work in general since function parameters are usually just Object and this leads to a lot of boxing. You can type hint them as primitives but it's pretty funky internally once you start doing that. If code can offload the numerically intensive bit to either Java or something native (as Neanderthal does) then it's more tractable, or of course if it doesn't actually require very high performance.
Still, awesome effort - it looks like a ton of work.