Quantizing is porting the weights from using high-precision floating point decimals to lower-precision numbers, e.g., 4-bit and 8-bit ints. Less memory used by the weights means we can run bigger models on the same hardware.

The idea behind quantization is that these models have so many parameters, they'll still work even if we reduce each node's flexibility.