Llama 2 70B on M2 Max at 7 tokens/sec

twitter.com

42 points·tikkun··6 comments

For context: based on current gpt-4 times,[1] GPT-4 is about 12 tok/s on avg. GPT-3.5 is about 35 tok/s. OP got 13 tok/s on an A100. And 10 tok/s on M2 Ultra.

[1]: This site that I run https://openai-status.llm-utils.org

Step by step guide: https://mlc.ai/mlc-llm/docs/get_started/try_out.html

Two variables here when comparing to GPT-3.5 / GPT-4

time taken to generate a token (speed)

quality of tokens generated

How far off is Llama 2 70B in terms of hallucination/qualify of output compared to GPT-3.5 / GPT-4?

Looks great! Are there other benchmarks? How does the speed compare to other LLM engines like llama.cpp / vllm (on GPUs)? Is it able to do continuous batching of incoming requests like vllm?

You're probably aware, but OpenAI has different speeds for different kinds of customers. The most prioritized ones are accounts that have gpt-4-32k.

An easy to see this difference is comparing a trial account's Turbo speed to a pay-as-you-go one.

With an account with gpt-4-32k I observed around 125 tokens/s for gpt-3.5-turbo-16k maximum (4k turbo was slower by the way), as for normal gpt-4 on the same account - I don't remember exactly, but it was at least 20 tokens/sec.

In short, you can clearly observe the difference in speed (even for Turbo) like this:

trial account < paid < paid with gpt-4 access < gpt-4-32k-enabled account

Similarly: Full F16 precision 34B Code Llama at >20T/s on M2 Ultra https://news.ycombinator.com/item?id=37346727

Although that is for an experimental branch of llama.cpp, "PoC for speeding-up inference via speculative sampling".

Pretty cool. I'm trying to wrap my head around this. Can someone explain what this mlc tool does? My understanding is that it "compiles" an LLM to C code that can run on any platform? How is this different from "quantization"?

Reply on news.ycombinator.com