Hacker News Reader

Comment by neilv | Hacker News Reader

> Chat with GPT is an open-source, unofficial ChatGPT app with extra features and more ways to customize your experience. It connects ChatGPT with ElevenLabs to give ChatGPT a realistic human voice.

Looks like only GUI aspects of the UI are self-hosted, but that the text and speech aspects of the UI (and the bulk of the computation and IP) are provided by two SaaS services.

Self-hosted (and some degree of open) ML models are what a lot of people might want, so we should probably be careful when saying "self-hosted" right now, to not disappoint people or confuse discussion when talking about what we want.

View on HN

It's somewhat ambiguous language - "self-hosted ChatGPT UI" could lead many to believe it's completely self-hosted.

However, sophisticated readers familiar with ChatGPT will know the model and weights haven't been released and absent a leak/hack/release by OpenAI a completely self-hosted ChatGPT solution is impossible. Eventually we'll almost certainly see a "Completely self-hosted ChatGPT equivalent" (similar to Dall-E vs Stable Diffusion) but that's another thread for another time.

Based on my native speaker parsing of English "Self-hosted ChatGPT UI" is accurate and I'm not sure how else I would write it to disambiguate between a self-hosted UI and a completely self-hosted ChatGPT with a UI.

Ironically, this sounded like an answer from chat gpt.

But more to the point, a fully self hosted solution (llama), even running on a cellphone, is entirely believable. Look at some of the recent developments with llama.cpp and Stanford over the last week.

"Show HN: I made a self-hosted UI for ChatGPT," perhaps?

> I'm not sure how else I would write it to disambiguate between a self-hosted UI and a completely self-hosted ChatGPT with a UI.

"Show HN: I made a self-hosted UI for the ChatGPT API"

"Show HN: I hade a self-hosted UI for a local GPT model"

It's a self-hosted UI for ChatGPT right now, but my primary goal is to build a good open source chat interface that can be adapted to open source chat models as they become available.

Integrating with Alpaca, Llama, ChatGLM, OpenChatBox and whatever comes next should be straightforward once people figure out reliable and fast methods to run the models locally.

I think if you integrated with Llama, this repo would be wildly popular. I downloaded the weights over the weekend and decided I didn't want to spend my free time working on an acceptable UI.

I honestly am not that familiar with this space. How realistic is it that someone could self-host a ChatGPT instance?

Assuming the model was available, how big are the models and what kind of hardware is necessary to run the instance?

I think you’d need 2x A100 GPUs, which is $4.18 an hour on Runpod. If I was super bored I’d probably be willing to drop $50 for 10 hours to mess around with it.

https://www.runpod.io/gpu-instance/pricing

OpenAI hasn't published any information about the size or hardware requirements for running ChatGPT. Reading between the lines, the default ChatGPT Turbo model seems to be significantly smaller than GPT-3 (it's a distilled model), but probably still heavier than the Alpaca and Llama 7B models people are running (very slowly) on their single GPU computers this week. You'd probably need multiple A100s to get comparable performance to the ChatGPT API.

Does the llama code that dropped leverage the GPU at all? On an M1 it appears to just run on as many CPU cores as you want to throw at it. The 65B heats up 8 cores real nicely, and it's slow, but I imagine it would be a lot faster on the GPU.

I've seen people saying that limiting it to 4 cores out of the 8 total can actually lead to improved performance. Have you seen that?

8 starts and runs a bit faster for me if plugged in and before the fan kicks on and the CPU starts throttling. Once that happens it's probably better to stick with 4.

All of the llama implementations for Apple are CPU only afaik.

If you run it with 4-bit quantization completely on the CPU (similar to llama.cpp), ChatGPT should run in about 90 GB of RAM. Which is easy to get your hands on for a desktop, but it's out of reach for notebooks.

Also expect performance of couple seconds per token in that setup, for now you need something involving GPUs

Probably should just call it OpenAI/ChatGPT Client.

It says self-hosted ChatGPT UI in the title, was that different when you posted?

I think it's within the broad meaning of self-hosting. One thing it means is that there's no random new company that you have to trust with your data.

I think ifs pretty clear what he meant by self hostef chatgpt ui. To assume the non ui aspected is also self hosted is illogical no?

Edit i think the title name was changed. Dang can you please show revision history otherwise i cant dicuss properly

Reply on news.ycombinator.com