skohan

Agent sandboxing/access control is one of the biggest problems to be solved before this technology really should go mainstream.

Even as a technical person, it's not trivial to sandbox agents correctly. The fact that an mis-clicked permission popup could give an agent unrestricted access to a user's disk is a massive risk vector in the hands of lay people who barely understand how any of this works.

So much of current security depends on the model of tying access control to a user account. A lot has to be re-thought in terms of how to grant access to an agent working on the user's behalf, in a way that doesn't make it completely useless, and also doesn't require every user to become a sysadmin managing fine-grained agent permissions manually.

A more cynical version of me could imagine scheduling an outage to make people think about how dependent they are on your product in advance of the IPO

I recently got my feet wet with Nix for the purpose of agent sandboxing, and it's a really great concept!

The issue I have run into is that for running llama.cpp, which is a very actively developed bleeding-edge software, it seems like experimenting with different versions/configs/patches etc. was fighting with the Nix philosophy of immutable software, so I ended up just managing it outside of nix.

But this could be user-error, I've only spent a couple of weeks with it.

1.2 T/s is not that modest is it? That's very close to an RTX pro 5000

I don't think smart people generally solve problems by talking through reasoning steps at a mile a minute. They clear their mind and let the solution come.

Of course I don't know if there's really a way for this to be molded in current LLM's (sounds more like diffusion)

16GB VRAM is probably not quite enough. With 32GB, or maybe even 24GB, you can do serious coding work with local models.

For decode, memory bandwidth is the main bottleneck, so these machines will likely perform well even without a ton of GPU horsepower. Not as well as Blackwell, but I expect they will be a reasonable choice in terms of price/performance if you want to run large models with a lot of context.

The main place they are a bit behind is in the number formats they support natively. Iirc M5 doesn't have native FP8 support, so you will take a speed penalty on quants where other architectures get better acceleration.

That's what I mean though. The comment I was responding to was talking about "forever laptops" - my point is there's plenty of room for new capabilities which will make current hardware obsolete. Just like how GPU's didn't exist at all, and became a standard part of computing.

And given how fast the hardware and software is evolving, I can easily imagine a future where we all have very capable models running on our own devices for an embedded intelligence layer that's doing most of the day-to-day tasks, and only have to outsource to a super-smart cloud model for specific things.

As someone who does a lot of work with local LLM's, today's systems feel woefully under-powered. I'm looking forward to a future where my laptop has 10x the memory, 100x the memory bandwidth, and optimized cores to make inference workflows that currently take minutes or hours go down to seconds or milliseconds.

While we're at the point where traditional software is pretty much fast enough for all but extreme use-cases, with LLM's it feels like we're back to the days where you press compile and go have a coffee or chat to your colleague.

Ok that's fair if it's targeting a non-technical audience.

But I think this will eventually be a problem solved at the OS level in a more streamlined way. I.e. there will be fine-grained permissions you need to approve to give an agent access to the system.

That's the most basic version of a harness. But a harness is really about automating context delivery to the LLM based on your use-case.

Is that the harness' job? It seems to me the best place for sandboxing is at the OS level (i.e. running the harness inside a container with correct access configured).

And despite the enormous capital expenditure, Chinese models are nipping at their heels at what must be a fraction of the cost. Sometimes constraints are healthy for inducing creative solutions.

Is huggingface profitable?

I'm not a fan of big-tech acquisition results either, but one benefit can be that a product continues to exist when it would otherwise become insolvent.

That could be a factor, but the optimistic interpretation would be that they want to support the open ecosystem because it sells more chips.

Models are already largely hardware agnostic. It would be pretty hard to put that cat back in the bag.

I could imagine them building value-added services on top of HF to advantage Nvidia products (i.e. "run this model on NVIDIA cloud" with one-click), but in this moment it's hard to imagine how they could actively disadvantage models built to run on other platforms.

They didn't, but their price increases haven't kept pace with general memory price increases (yet) so currently their pricing seems reasonable.

Yeah I think this is probably one of the best outcomes you could hope for if you want to use open models.

I'm using unsloth dynamic Q4_K_XL.

My use-case is coding, currently working on a project with a Rust backend and TS/React/Vite frontend, with probably tens of thousands of lines of code total (including tests).

I'm using the unsloth dynamic Q4 and getting good results. I was running Q5, but Q4 gives more context headroom so I can run two agents in parallel with ~100k context each with 32GB vram.

3.8 doesn't have a minimal thinking mode, only low, medium and xhigh.

That sounds like something is off - I'm using UD-Q4_K_XL on pi with xhigh thinking, and unless I'm vastly underestimating the complexity of the script that's the kind of task I would expect to take a couple of minutes (getting ~30t/s decode). What server are you running, and are you using the recommended parameters from qwen/unsloth?

Your assumption is incorrect.

I'm using pi inside a self-made harness. I've found going super lightweight with context (AGENTS.md is maybe 20 lines) and letting the model discover what it needs to gives the best results.

I've had the exact opposite experience. I've been using 3.8 for my daily driver since last week, and I've gradually been giving it more and more complex tasks as it continues to deliver high quality results. Now I am basically handing off large complex features, and 3.8 is doing the planning, task breakdown, implementation and review with just a few notes from my side.

The tradeoff is time (especially on RDMA4 hardware) - it does take a long time and spend a lot of tokens to get to the result, but I've found I can trust the results enough that I can queue a lot of work, essentially have it running all the time and achieve a decent velocity.

It's the first small local model I've felt like I can do real work with.

> interesting features

Can you elaborate at all?

Well I wouldn't say a billion times better. I've actually been having a surprising amount of success working with local models. And my investment has only been the equivalent of 4 months of a Max x20 subscription.

Experimentation and privacy are definitely advantages, but it's also quite a lot of fun.

This also seems like quite an esoteric use-case to me, but I guess some people might need to know when text follows a fibonacci sequence in terms of word count.

This is a big part of the reason I went local-only. Subscription limits are horrible for having a decent workflow.

I would happily buy up a load of datacenter GPU's at deep discount

Glimmer benchmarks around Qwen 3.6 27B levels no?