Hacker News Reader

Comment by jacquesm | Hacker News Reader

We did the same thing, on much less powerful hardware but it has served us very well and we don't need to worry at all about the question whether or not the suppliers of inference services can be trusted long term with our data. The answer to that question in my opinion is 'absolutely not'.

View on HN

How do security conscious self-hosters sandbox their runtimes?

Your inference server is just an inference server that spits out tokens. You can use the same harness, etc. you’re basically just changing the API base URL.

By pulling the plug. This box simply has no pathway to the internet. I do pretend I have one and it's interesting to see what it tries to do. For instance, Qwen tends to try to hit ali related URLs for websearch and is very disappointed when the mothership is not responding. I've yet to catch a model on an exfiltration attempt, either they are so good at this that it escapes my scrutiny (possible but unlikely) or they are simply not doing it. I would bet on the last one. Just don't say you're working for the Tibet Liberation Frontier or the Keep Taiwan Independent movement.

> Just don't say you're working for the Tibet Liberation Frontier or the Keep Taiwan Independent movement.

Has abliteration/decensoring been at all useful or relevant in your setup?

I have downloaded a number of abliterated models and done very brief tests on them (just to check for basic functionality) but on local models I have yet to run into refusals or output where I thought it might make sense to retry with an abliterated model. It was more out of a precaution that NV might drop those models from HF.

Why would you sandbox the runtime? The harness is where things go awry when it does, when the agents have access to more than they should and start deleting stuff. People aren't expecting the weights themselves to be "infected by malware" or whatever.

I think GP meant their agentic runtimes.

Can you share some details about your setup? Would probably be interesting for a lot of people.

Sure:

https://jacquesmattheij.com/local-inference-with-large-langu...

If you have more questions feel free to mail or ask here in the thread and I'll do my best to answer.

The hardware related parts are near the end of the article.

Your sites SSL certificate just expired..

Thank you, I'll fix it right away.

Edit: ok, should be good again. For some reason on that box running Python from cron jobs is unpredictable, usually it works but every now and then it does not and I've yet to find out why.

That was a good read!

Thank you. That was a lot of work to put it all together.

It's basically the document I wished I had when I started all this.

Reply on news.ycombinator.com