Hacker News Reader

Comment by mmastrac | Hacker News Reader

> MobileLLM-125M/350M attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M SoTA models on zero-shot commonsense reasoning tasks

Small models, slightly improved, probably still not good enough for the same use as online models. Nothing wrong with incremental progress, however.

1.5B parameter model does seem to be a pretty decent step up, even beating larger models by a wide margin. I'm not sure why they didn't go larger -- having a more efficient model that fits on hardware the size of the RPi could be a gamechanger (IIRC TinyLlama 7B does run, barely).

View on HN

>> Small models, slightly improved, probably still not good enough for the same use as online models. Nothing wrong with incremental progress, however.

An even smaller language model should still be useful as part of a speech-to-text system. These should benefit from using the language model to narrow down what word is spoken in the face of ambiguity or noise.

ASR systems already use language models during decoding, though mostly not large decoder-only LLMs. However, incorporating LLMs into ASR is currently at the center of a lot of research, e.g. using a speech encoder like wav2vec 2.0 or the whisper encoder with a Qformer etc. and a LoRA adapter on an LLM trained for ASR.

Really interested in this! Do you know of some good reading in this area?

But imagine if these models were baked into your Instagram app and then used for ad targeting using your own compute. Then Facebook gets to look at tons of other data and for less cost (and much less litigation risk) to them.

In this application it’s unfair to compare tiny models to cloud models. Moreover any incremental precision boosts to tiny models would be notable (and directly translate to revenue).

> I'm not sure why they didn't go larger -- having a more efficient model that fits on hardware the size of the RPi could be a gamechanger (IIRC TinyLlama 7B does run, barely).

I'm not sure that RPi is the right target for the next step of local LLMs, and I think that it's worth considering web-deployment on engines like WebLLM [1].

A 7B model may "run fine" on a Raspberry Pi, but I've (personally) found 7B models to be a bit larger than I want to download / run for web-based interfaces.

However, a solid 125M model is the sort of thing that I can run on a webpage, and the time it takes to download to the local user's browser (combined with my bandwidth costs) aren't exorbitant.

[1] https://github.com/mlc-ai/web-llm

Llama-3-8b runs fine on raspberry pi

how fast is that for you?

Reply on news.ycombinator.com