Hacker News Reader

Comment by brianjking | Hacker News Reader

Nice, I wish it was a little easier to integrate these models into Chat UIs like the one from Vercel or even a simple Gradio app.

Does anyone have any Spaces/Colab notebooks/etc to try this out on?

Thanks!

View on HN

I've found https://chat.lmsys.org/ to be a useful multi-LLM chat app without scary ToS data-mining clauses.

There are many UIs for running locally, but the easiest is koboldcpp:

https://github.com/LostRuins/koboldcpp

Its a llama.cpp wrapper descended from the roleplaying community, but works fine (and performantly) for questioning and such.

You will need to download the model from HF quantize it yourself: https://github.com/ggerganov/llama.cpp#prepare-data--run

Plenty of ways to run locally - I'm looking for ways for inference via Colab or Huggingface.

Oh, well you cab login to huggingface and deploy it in a space with a button above the model.

Not sure about colab at the moment.

That can give you an inference endpoint for the API, I'm talking about a full chat UI where you can set the temperature, etc.

This isnt specifically Colab or HF, but have you checked out any of the community Runpod templates? Theres a few out there that give you a mostly turnkey way to deploy models and test via Oobabooga, koboldai, or similar.

I use the one-click UI from TheBloke pretty frequently for inference testing, and I know theres some newer ones that also give you fine-tuning capabilities as well.

Link to the runpod templates?

These run on your local machine or can I use Colab/Huggingface GPUs?

https://www.runpod.io/console/templates

This is the readme for the one I mentioned: https://github.com/TheBlokeAI/dockerLLM/blob/main/README_Run...

> can I use Colab/Huggingface GPUs?

You use these templates on the runpod platform itself. Theres no free tier equivalent like you have with Colab/HF, but currently you can rent an RTX 4090 for $0.69/hr so its pretty affordable.

Yeah, but I'd like to try deploying this on Huggingface or Lambda Labs.

Do you know if that is possible?

In theory yes, since its fundamentally just a Dockerfile, any provider that supports docker is possible. I know HF Spaces have a "docker" sdk option[1], but I personally haven't used them myself.

[1] https://huggingface.co/docs/hub/spaces-sdks-docker-first-dem...

Reply on news.ycombinator.com