grey-area

The way we store memories is to adjust the weights in our brain (as far as we know). The way models store memories seems to be to write them down on a scratchpad someplace (details vary, but roughly this is what they're doing).

The article explicitly excludes this from consideration early on though:

By “agent memory,” we mean knowledge that Claude gains from experience...saved as context it can read later. It excludes other ways models can learn or manage context, such as changes to their weights

Those seem fundamentally different processes, and adjusting weights does seem a much better idea after all, but then in that world everyone has a different AI and how do you apply updates? For a company like Anthropic this is of course a big problem and that is probably why they don't like that route. Could we store weight deltas or something similar and reapply them over a different model?

This is not just about email addresses. It’s a bigger problem than that.

The supercharger stations will obviously keep working during the apocalypse.

Sure, it's a hard problem, that doesn't mean we shouldn't tackle it. It is in fact one of the more urgent problems for Anthropic to address properly.

I'd feel a lot safer granting fine grained permissions like that, than knowing that the only thing stopping the LLM reading files is that it hasn't thought to do so.

I mean, I think Claude has some controls on file access, but I'm not exactly sure any more what is and what isn't allowed. The impression their desktop app gives is that you need to give permission for each and every file or folder access, but I'd feel a lot safer if that were imposed by an operating system level constraint, not claude trying to read the use of tools and ok/disallow them based on what an llm thinks.

I fervently hope you're right.

From the doc, this suggestion is hardly satisfactory:

The sandbox covers shell commands only. Claude’s file tools, MCP servers, and hooks run outside it.

Also this is in my view the main job of a scaffold like claude code - to restrict what the agent can see and make sure it doesn't use info it should not. It's not something a user can easily or should have to attempt to control themselves.

There are ways to sandbox this, and it should have been the first thing Claude Engineers were focussed on, it's a basic first requirement that the app doesn't have access to files outside a very limited set it requires to perform the task at hand.

Why does Claude have access to to your email or any other info about the user by default?

Biggest questions here, which I think need answering:

Should Claude have access to user details? I’d say no, unless explicitly entered manually into a chat.

Should Claude have access to anything outside of the sandboxed project files without explicit permission? I’d again say no.

Should Claude have internet access without a clear log of all requests made?

Instances of Claude should be heavily sandboxed and kept away from user data, far more so than they are now. The impression given by the desktop UI is that the LLM only has access to things explicitly added to a project.

This impression is not true, as evidenced by Claude pulling data from other sessions outside a project even when not instructed to (I’ve seen this a few times), and in this case pulling your email (from account details, user contacts? I wonder where it got that info?).

None of this information should be available to Claude, by design, not by instruction. Using text instructions to try to limit these machines is an incredibly flawed approach and should not be necessary for important info. Anthropic should have the model access completely locked down by controls at the os level over file access, not open by default. That is how to fix this.

If a user wants to open access that should be an explicit choice from them, with a warning about possible consequences.

You will be turning off any behaviour like this by default, right? Why are you suggesting users turn it off with a mod?

Only has to fall by 33% to go back 5 years then.

A few days at > 10% and that's perfectly doable in a week. Unlikely I'll grant you but this is a large move at the top of a bull market amid huge concentration and weakening fundamentals. There are reasons to pay attention to it.

Do we even care about MCP any more? It really doesn't add much to an API.

It will go the way of Openclaw, Personal Agents, Scaffolding, Skills and all these other things people prematurely tried to build around LLM models.

Sure, he's also a self-centred fool who hates being ignored and thinks he can run again in 2028. I expect him to bomb Iran in the next few days, and congress/senate to be too slow/weak to stop him.

Not sure it works like that: https://pmc.ncbi.nlm.nih.gov/articles/PMC9122750/

Inequality declined temporarily during the recession, but reached new highs during the recovery.

2008 decreased inequality for a very short time, but inequality increased overall. The rich have the available assets to profit from instability and are able to ride out unemployment, for the poor a crash and losing their job can be existential.

Statistics about up and down days or particular periods of the stock market can say whatever you want, because they're so easy to cherry-pick.

A more interesting stat is the current CAPE.

https://www.thetrading.tools/shiller-cape

It is at levels only seen three times in the last century.

Unfortunately this is not what happens. The Big Short is a good (and fun!) primer about collapses:

https://www.youtube.com/watch?v=0k5aVLi_yhM

When the market crashes, a lot of people lose jobs, and a lot of people die. Everyone suffers but those with money suffer the least, by far.

Cloudflare is now a mature company, they don't have a huge amount of room for growth, and no, they will never sit in front of the entire internet.

According to these figures, their growth is levelling off.

https://news.ycombinator.com/item?id=50382740

And nowhere near enough to sustain that valuation, particularly in a market which is near the end of a bull run.

They don't need your prompts. Ideas have never been hard to come up with.

The hard work is in building something over years, supporting customers who want something you don't, persuading people to trust you and pay money for things, making hard choices to keep a product on track, etc.

None of these things are any easier, as you'll find out if you try to sell the things you have made. The first step is the easiest and it's hard to stick with things.

If the releases change frequently though, this would be of little use to anyone save those who feel sticking with one model is best. The weights are not exposed so you have to trust Anthropic that changes they make are in your best interest.

1. That sounds like an Anthropic concern, not a concern for most customers who don't run their own model. 2. This still requires trust in the provider, I imagine these proofs would change weekly as they tweak their models, and you have zero knowledge of what the changes are, you just know you're getting version 11.12.1999 - does that really help you if the provider is maliciously downgrading after release? All you know is you got the same version as everyone else, not what that version is. 3. Again, if you don't trust the provider, it's game over - you're sending them all of your data. Knowing a version number is not going to help you much.

I'm not convinced this is useful for problems most people face using models, it seems a very niche academic concern frankly and an incredibly complex system to prove not very much.

But it still requires trust in whoever is running the server that tags version 'x' as x. All you have proof of is that they return the hash for a given set of weights which is the same as someone else's hash, and those tags/hashes will be changing all the time as the models evolve, so people are not going to be able to keep track of whether Anthropic say is dumbing down their model or not after release, because this is a zero knowledge proof, all it tells is that a certain set of weights was used (you have no idea what weights, how many weights etc etc).

So I don't really see this as useful, except in the very limited use case of using a third party proxy who is trying to cheat you (in which case why are you sending all your data to someone you can't trust?).

Professional food photography is expensive. This is a case of the free mediocre offer crowding out the much better but expensive option. Same way that free mail with ads crowded out email without ads.

I’m not sure what you mean by correctly here - do you mean with the model expected? That’s not related to correctness.

Pleased to hear somebody cares about this. What is it a bottleneck for though? Are people actually doing this right now in non-academic settings? I had the impression this is not in fact usable with larger models.

If only the internet had some way to encrypt data in transit between two parties.

This is a solved problem if you trust your counterparty and if you don’t it’s the least of your problems.

All this provides is a tag saying 'x' version (which your counterparty defines anyway) is being used.

But if you’re using them they’re probably already stealing your data, fake-models are the least of your problems.

I didn't find any of it compelling, perhaps you do. That's what I was asking really, which of these use-cases is valuable for you?

For me if you use a routing service you have to trust it, it sees all your inputs and outputs.

If a cyberattacker can modify the token stream (i.e. man in the middle your traffic), you should be worried about a lot more than which model you're using!

At first I thought this was work on the far more important problem of proving that LLM output is provable and correct given some source of truth. But it's about the trivial problem of 'proving' which model is being used.

Who will find this useful? I can't really think of a use for it, because if you don't trust the provider of your models, it's game over really - you're already giving them all sorts of private data, you're relying on their service, you have to trust them to some extent and this doesn't really help because unless you are running the service yourself. Seems like more of an internal problem for Anthropic than something that would interest others.

ok, I'm just describing problems I haven't seen LLMs improving on for the last 4 years. These are important problems, and I'm not convinced a machine-generated ontology is the start of the answer to them.

Happy to admit relative ignorance of the internals and hear a more informed opinion, but it is not convincing to hear that they'll inevitably solve all the problems and we'll live in the best of all possible worlds next year, without hearing a proposal as to how they'd do that.

These machines are trained to be convincing and addictive, very similar to our social media platforms now, that leads to a lot of over-optimism about what they might be able to do one day, with the next frontier model.

It’s perfectly plausible that unscrupulous people at OpenAI (and that’s the culture right from the top) searched user transcripts for relevant approaches to problems they were working on, or searched them for problems to look at, and nudged models based on that.

I agree it’s unlikely this happened by accident, or entered training data (though not impossible), but it could certainly be within the context agents were using when tackling problems.

We have no evidence either way, but they had all the data they would need to do this.

Hmm, seems their ontology didn't fix this assuming they use it themselves.