Although, I guess the cutoff might have to be a bit higher than 8.23. Maybe the neuron activations would never exceed that range, but some intermediate computations could.
Supposedly the new Maxwell GPUs have some new instructions for working with half-precision floats ( https://developer.nvidia.com/sites/default/files/akamai/open... ). I wonder how complete this implementation is, because half-precision might be sufficient to train a convnet, and this would result in a significant speedup.