Hacker News Reader

Interesting comment on machine learning.

My understanding is that they don't train with half precision, but quantisation allows even integer values to be used for inference?

Yet the reasoning still makes sense. Can training take place at half precision or even if you intend to run inference at halfp do you want doublep for training?

View on HN

My guess is that very small and very large values in the weights are already trained away due to the regularisation of the cost function, so insignificant changes in a network don't tend to have significant changes in the output.

You gain more by being able to run gradient descent faster than by having higher-precision floats.

Absolutely. And beyond weight regularization, for any weighting followed by a sigmoid or other squashing function, large weights simply tend to saturate the squashing function and there is very little gradient (quickly effectively zero) to benefit from increasing the weight value past that point.

Training with half precision (or mixed precision) is common.

Reply on news.ycombinator.com