mlgoatherder

Ah but LLM.int8 (eg. as in huggingface transformers) isn't actually int8, it's a mixed precision encoding scheme that is nominally eight bits per parameter. This means custom cuda kernels etc, these kernels could be improved but without hardware support its always going to be slow.

Straight int8 quantization generally does not work for post training quantization of transformers. The distribution of weights includes a significant amount of outlier values that seem to be important to model performance. Apparently quantization aware training can improve things significantly but I haven't seen any developments for llama yet.

Interestingly on the 4 bit front, NVIDIA has chosen to remove int4 support from the next gen Hopper series. I'm not sure folks realize the industry has already moved on. FP8 feels like a bit of a hack, but I like it!

I've done some experiments here with Llama 13B, in my subjective experience the original fp16 model is significantly better (particularly on coding tasks). There are a bunch of synthetic benchmarks such a wikitext2 PPL and all the whiz bang quantization schemes seem to score well but subjectively something is missing.

I've been able to compare 4 bit GPTQ, naive int8, LLM.int8, fp16, and fp32. LLM.int8 does impressively well but inference is 4-5x slower than native fp16.

Oddly I recently ran a fork of the model on the ONNX runtime, I'm convinced that the model performed better than pytorch/transformers, perhaps subtle differences in floating point behavior etc between kernels on different hardware significantly influence performance.

The most promising next step in the quantization space IMO has to be fp8, there's a lot of hardware vendors adding support, and there's a lot of reasons to believe fp8 will outperform most current quantization schemes [1][2]. Particularly when combined with quantization aware training / fine tuning (I think OpenAI did something similar for GPT3.5 "turbo").

If anybody is interested I'm currently working on an open source fp8 emulation library for pytorch, hoping to build something equivalent to bitsandbytes. If you are interested in collaborating my email is in my profile.

1. https://arxiv.org/abs/2208.09225 2. https://arxiv.org/abs/2209.05433

Great opportunity to pitch my recent work on a similar solution for gRPC:

https://github.com/sillystack/api/blob/main/transport/transp...

I'm not going to make any security claims as of now (it's almost certainly broken) but it uses the Noise IK handshake.

Long term I think NoiseSocket would be a good way to go https://noisesocket.org/post/1/ but perhaps in service-to-service use cases ciphersuite negotiation can be simplified.

I recently implemented a very similar service to service gRPC authentication mechanism for Go using EC25519 and the noise protocol framework.

https://github.com/sillystack/api/blob/main/transport/transp...

The code is very fresh and hasn't yet gone through a audit so please don't use it for anything where security actually matters.