Optimizing AI Inference at Character.ai

Noam has been cooking at character.ai in stealth. Their model is impressively engaging

Is it really "stealth" or is it just a modest level of marketing and popularity?

didn't catch hype and big VC money..

Training in int8 is noteable (to me). I've been out of date with ML research for a bit now but last I recall, people were mostly training at full precision and then quantizing after training and finetuning a bit on the quantized model afterwards.

Dunno. It could also just mean the so-called "Quantization-aware training" where your weight, activation and gradient is still bf16 and just before use it gets quantized to int8 in the same way you'd do it during inference.

This gives you the same "no mismatch between training and predict", and was a standard technique back in vision days (~2018).

> we implemented customized int8 kernels for matrix multiplications and attention

I would be curious how this differs from [1] which is supported in Huggingface’s transformers library.

[1] https://arxiv.org/abs/2208.07339