Hacker News Reader

It's apparently running at ~11 tok/s on a 4090: https://github.com/Magnvs7/Promethevs/issues/7

Strata runs considerably faster than that on similar hardware, though this one doesn't seem to be a fork.

Oh, I don't know - it could be useful for long-running and/or background tasks in some cases.

The problem is that this particular model is so cheap, compared to others. Optimizing to host it locally with relatively poor performance just doesn't make sense when yesterday I burned ~180M tokens for $2.20.

Surely local hosting is not so much about cost savings as it is about data ownership and risk management. e.g. can't leak a secret if you're not calling out.

Strata for running from HDD seems really popular, to get big models running.

Is that faster?