Hacker News Reader

It appears to generate 30 tokens/s on iPhone 14 pro: https://i.imgur.com/AWTXtGA.png

But for some reason it dramatically slows down after a few messages

Edit:

Oh no, this one also gives lectures instead of answering questions.

https://i.imgur.com/eiuGzK4.jpg

I'm afraid, in near future the only organic content on the internet would be only the type of content that LLMs refuse to generate.

View on HN

Tokenization is a probable issue here, the longer the context the longer the initial processing with llama I think. Possible tokenizer is not optimized.

That’s very unlikely, tokenization is really simple and usually quite fast (scales with input size). Unexpected slowness with a larger context window might point to a non-existent or unoptimized KV cache.

Reply on news.ycombinator.com