You also cant clip them and get away with it, you have to either renormalize the layers to do half-precision (and live with the extra cost) or stick to full-precision. I was doing fun stuff early this year that did fixed-precision nets (8-bit/16-bit). Things get very interesting :)