Do 1080tis have fp16 support? Seems like a waste if the model can be fp16 trained and you're using full 32bit.

Similarly you should probably try a bunch of other frameworks (caffe2, cntk, mxnet) as they might be better at handling this non standard configuration.