For those who are not aware, there is https://github.com/migueldeicaza/TensorFlowSharp for your arbitrary TF in DotNet needs. Last I played with it, it even work well in F#. Which, let's be honest here, is the dream right? Right?!
I really want to use F# for training deep learning models, but at this point it doesn't seem feasible, at least not without a lot of hassle compared to using Python.
Poking around it looks like Accord.Net is interested in getting CNN support in via TensorFlowSharp: https://github.com/accord-net/framework/issues/92 . Then there is ConvNetSharp.
I'm sure a fun project for someone would also be to implement all this in a pure F# library. It supposedly has some pretty good GPU libs.