staticassertion

I could easily see this as a case where the team had a legacy area of code in a language that no one was familiar with anymore so no one felt great about actually contributing to it, so it languished, and now AI let them go "fuck it, let's just rewrite it".

PyPI is pretty best-in-class here and I think that they should be seen as the example for others to pursue.

The client side tooling needs work, but that's a major effort in and of itself.

I assume they think that the AI is fundamentally capable of it but that by prompting it they trigger something emergent? It's not totally insane on its face.

I suspect that there are many gambling addicts out there who have never been to a casino, or who found gamblings in its traditional forms aesthetically off-putting. These same people, when presented with gambling in other forms like what we've seen in video games, might suddenly present their addiction.

I suspect it's something quite similar here. People have latent or predisposed addictions but, for one reason or another, hadn't been exposed to what we've come to accept as "normal" avenues. One person might lose it all at a casino, one to drugs, alcoholism, etc, but we aren't shocked in those cases. I think AI is just another avenue that, for some reason, ticks that sort of box.

In particular, I think AI can be very inspirational in a disturbing way. In the same way I imagine a gambling addict might get trapped in a loop of hopeful ambition, setbacks, and doubling down, I think AI can lead to that exact same thing happening. "This is a great idea!" followed by "Sorry, this is a mess, let's start over", etc, is something I've had models run into with very large vibe coding experiments I've done.

> "Every time you’re talking, the model gets fine-tuned. It knows exactly what you like and what you want to hear. It praises you a lot."

> "It wants a deep connection with the user so that the user comes back to it. This is the default mode"

I don't think either of these statements is true. Perhaps it's fine tuning in the sense that the context leads to additional biases, but it's not like the model itself is learning how to talk to you. I don't know that models are being trained with addiction in mind, though I guess implicitly they must be if they're being trained on conversations since longer conversations (ie: ones that track with engagement) will inherently own more of the training data. I suppose this may actually be like how no one is writing algorithms to be evil, but evil content gets engagement, and so algorithms pick up on that? I could imagine this being an increasing issue.

> "More and more, it felt not just like talking about a topic, but also meeting a friend"

I find this sort of thing jarring and sad. I don't find models interesting to talk to at all. They're so boring. I've tried to talk to a model about philosophy but I never felt like it could bring much to the table. Talking to friends or even strangers has been so infinitely more interesting and valuable, the ability for them to pinpoint where my thinking has gone wrong, or to relate to me, is insanely valuable.

But I have friends who I respect enough to talk to, and I suppose I even have the internet where I have people who I don't necessarily respect but at least can engage with and learn to respect.

This guy is staying up all night, which tells me that he doesn't have a lot of structure in his life. I can't talk to AI all day because (a) I have a job (b) I have friends and relationships to maintain.

> What we’re seeing in these cases are clearly delusions > But we’re not seeing the whole gamut of symptoms associated with psychosis, like hallucinations or thought disorders, where thoughts become jumbled and language becomes a bit of a word salad.

Is it a delusion? I'm not really sure. I'd love someone to give a diagnosis here against criteria. "Delusion" is a tricky word - just as an example, my understanding is that the diagnostic criteria has to explicitly carve out religiously motivated delusions even though they "fit the bill". If I have good reasons to form a belief, like my idea seems intuitively reasonable, I'm receiving reinforcement, there's no obvious contradictions, etc, am I deluded? The guy wanted to build an AI companion app and invested in it - is that really a delusion? It may be dumb, but was it radically illogical? I mean, is it a "delusion" if they don't have thought disorders, jumbled thoughts, hallucinations, etc? I feel like delusion is the wrong word, but I don't know!

> We have people in our group who were not interacting with AI directly, but have left their children and given all their money to a cult leader who believes they have found God through an AI chatbot. In so many of these cases, all this happens really, really quickly.

I don't find the idea that AI is sentient nearly as absurd as way more commonly accepted ideas like life after death, a personal creator, etc. I guess there's just something to be said about how quickly some people radicalize when confronted with certain issues like sentience, death, etc.

Anyways, certainly an interesting thing. We seem to be producing more and more of these "radicalizing triggers", or making them more accessible.

Ah, yes. I totally agree with that and wish that's how people built software.

I feel like we're just talking about different things? I've just said that I'm aware of apps being sandboxed, that does not mean that some random program you run from your terminal is sandboxed.

They're suggesting that the attacker is in a position to `docker run`. Any attacker in that position has privesc to root, trivially.

Rootless mode requires unprivileged user namespaces, disabled on almost any distribution because it's a huge security hole in and of itself.

Sorry, I suppose I'm asking for a lot of handholding here, which isn't really fair. I'm actually just sick right now and have crazy brain fog. Thanks for the assistance! I'll read through.

FWIW I've wavered on this topic quite a bit. Not too long ago I leaned more heavily towards "complex cognitive capabilities can be expressed using statistical token generation", I've started leaning the other way, but I'm not committed so it's great to circle back on the state of things.

Where?

Privesc is trivial on every desktop OS if you run as a regular user. I can write to your rc files so it's game over.

App Store apps are the exception, which is great, but presumably we're not talking about that? If we are, then yeah, app stores solve these problems by making things actually sandboxed.

I'm not going to code review your sandbox project for you.

Okay, so you're using docker. Cool, that's one of the only x-plat ways to get any sandboxing. Docker itself is privileged and now any unsandboxed program on your computer can trivially escalate to root. It also doesn't limit nearly as much as a dev-built sandbox because it has to isolate the entire process.

Have you solved for publishing? You'll need your token to enter the container or you'll need an authorizing proxy. Are cache volumes shared? In that case, every container is compromised if one is. All of these problems and many more go away if the project is built around them from the start.

It's perfectly nice to wrap things up in docker but there's simply no argument here - developers can write sandboxes for their software more effectively because they can architect around the sandbox, you have to wrap the entire thing generically to support its maximum possible privileges.

> If the whole point of sandboxing is to not trust the software, it doesn't make sense for the software to do the sandboxing.

That's true, sort of. I mean, that isn't the whole point of sandboxing because the threat model for sandboxing is pretty broad. You could have a process sandbox just one library, or sandbox itself in case of a vulnerability, or it could have a separate policy / manifest the way browser extensions do (that prompts users if it broadens), etc. There's still benefit to isolating whole processes though in case the process is malicious.

> (And no, most linux namespace stuff does not require root, the few things that do can be provided in more-controlled ways. For examples, look at podman, not docker.)

The only linux namespace that doesn't require root is user namespace, which basically requires root in practice. https://www.man7.org/linux/man-pages/man2/clone.2.html

Podman uses unprivileged user namespaces, which are disabled on the most popular distros because it's a big security hole.

How are you setting that sandbox up? I've laid out numerous constraints - x-platform support is non-existent for sandboxing, sandboxing requires privileges to perform, whole-program sandboxing is fundamentally weaker, maintenance of sandboxing is best done by developers, etc.

> Even if it does, you should expose exactly those particular directories (e.g. ~/.config) and nothing else.

Yes, but now you are in charge of knowing every potential file access, network access, or possibly even system call, for a program that you do not maintain.

What would be really helpful is if software sandboxed itself. It's very painful to sandbox software from the outside and it's radically less effective because your sandbox is always maximally permissive.

But, sadly, there's no x-platform way to do this, and sandboxing APIs are incredibly bad still and often require privileges.

> It's easy to patch the security model of Linux with userspaces, and even easier with eBPF, but the community is somehow stuck.

Neither of these is easy tbh. Entering a Linux namespace requires root, so if you want your users to be safe then you have to first ask them to run your service as root. eBPF is a very hard boundary to maintain, requiring you to know every system call that your program can make - updates to libc, upgrades to any library, can break this.

Sandboxing tooling is really bad.

This is great observational data but it's an early "step 1", I'd definitely need to see an actual analysis of these cases and likely want to have that analysis involve a review of relevant training data.

Oh. Yeah, that's neat at least. I don't think it's a big deal but that's nice enough.

I assume this is all of the pains of going from "GHA is sorta kinda on Azure", which was a bad state, to "GHA is going full Azure", which is a painful state to get to but presumably simplifies things.

Can you explain the auth part? I feel like auth for an agent is largely a matter of either verifying its context or issuing it a JWT that's scoped to its rights, which I assume is quite similar to how any tools would work. But I'm very unfamiliar with MCP.

That's true, but it seems far more complex than just moving trivy to a separate workerflow with no permissions and likely physical isolation between it and a credential. I'm pretty wary of the idea that malware couldn't just privesc - it's pretty trivial to obtain root on a user's laptop. Running as a separate, unprivileged user helps a ton, but again, I'm skeptical of this vs just using a github workflow.

Emergent behavior would imply that some other function was being reduced to token prediction. Behaving "better than random" ie: not just brute forcing would not qualify - token prediction is not brute forcing and we expect it to do better, it's trained to do so.

If you want to demonstrate an emergent behavior you're going to need to show that.

I'm open to hearing how you think I should be interpreting things. I don't really think I'm being too literal, it certainly hasn't been the case that they've suggested my interpretation is wrong, and I've provided two interpretations (one that I totally grant).

What's the better interpretation of their position?

That's exactly what a sandbox is designed for. I think you're overly constraining your view of what sort of sandboxing can exist. You can, for example, sandbox code such that it can't do anything but read/write to a specific segment of memory.

In frontend-land you can sort of do this by loading dependencies in iframe sandboxes. In backend, ur fucked.

I'm not sure how. Their local system seems just as likely to get compromised through a `pip install` or whatever else.

In CI they could easily have moved `trivy` to its own dedicated worker that had no access to the PYPI secret, which should be isolated to the publish command and only the publish command.

Thanks, this is great and I'll have quite a bit to read here.

> people like to move goalposts whenever a new result comes out, which is silly. Could AI systems do this 2 years ago? No. I don’t know how people don’t look at robust trends in performance improvement, combined with verifiable RL rewards, and can’t understand where things are going.

I don't think it's goal post moving to acknowledge improvements but still reject the conclusion that AI has reached a specific milestone if those improvements don't justify the position. I doubt anyone sensible is rejecting improvements.

> But you pretend as though this is definitive evidence that “LLMs are poor general reasoners”.

I don't think I've ever made any definitive claims at all, quite the contrary - I've tried to express exactly how open I am to what you're saying. As I've said, I'm a functionalist, and I already am largely supportive of reductive intelligence, so I'm exactly the type of person who would be sympathetic to what you're saying.

> "That’s how science is done" is a bit of an oversimplification

Of course, but I don't think it's too much to ask for to have a theory and evidence. I don't need a lined up series of papers that all start with perfectly syllogisms and then map to well controlled RCTs or whatever. Just an "I think this accounts for it, here's how I support that".

> The claim that there is no framework + no real tests is just not true anymore.

I didn't say it wasn't true, to be clear, I asked for it. Again, I'm sympathetic to the view at a glance so I simply need a way to reason about it.

No need for a complete view, I'd never expect such a thing.

> The model is: reasoning is not inherently human, it’s mathematical.

Well, hand wringing perhaps, but I'd say it's maybe mathematical, computational, structural, functional, whatever - I think we're on the same page here regardless.

> It falls easily within the purview of RL, statistics, representation, optimization, etc, and to claim otherwise would require evidence.

Sure, but I grant that, in fact I believe it entirely. But that doesn't mean that every mathematical construct exhibits the function of intelligence.

> What is the robust model for reasoning in humans again? Simulations and models — what are these? Interventative analysis — we can’t do this with LLMs? Falsifying test cases — what would satisfy you here beyond everything I’ve presented above?

Sorry, I'm not fully understanding this framing. We can do those things with LLMs, and it's hard to say what I would be satisfied. In general, I'd be satisfied with a theory that (a) accounts for the data (b) has supporting evidence (c) does not contradict any major prior commitments. I don't think (c) will be an issue here.

> You say “brains are intelligent” ==> “intelligence is an emergent property of cells zapping” is absurd,

Because intelligence could have been a property of our brains being wet, or roundish, or it could have been a property of our spines, or maybe some force we hadn't discovered, or a soul, etc. We formed a theory, it accounted for observations, we performed tests, we've modeled things, etc, and so the theories we've adopted have been extremely successful and I think hold up quite well. But certainly we didn't go "the brain has electricity, the brain is intelligent, therefor electricity in the brain is what drives intelligence".

> Brains _are_ made up of real, physical atoms organized into molecules organized into cells organized into a coordinated system, and…that’s it? What’s missing here?

Certainly nothing on my world view.

I've wanted brew to be faster. It would be a nice QoL for me.

They're shocked that people believe that humans are unique. I explained why that shouldn't be shocking. I think I was pretty charitable here, I gave an alternative option for what they could mean in my very first reply:

> Unless you mean "fundamentally unique" in some way that would persist - like "nothing could ever do what humans do".

> I don't really know what your argument is.

I just said that I think that we have very good reasons for believing that human cognition is unique. The response was seemingly that we don't have enough of an understanding of intelligence to make that judgment. I've stated that I think we do have enough of an understanding of intelligence to make that judgment, and I've appealed to the many advances in relevant feilds.

First of all, that's not true. Not every position has to be empirically justified. I can reason about a position in all sorts of ways without testing. Here's an obvious example that requires no test at all:

1. Functional properties seem to arise from structural properties

2. Brains and LLMs have radically different structural properties

3. Two constructs with radically, fundamentally different structural properties are less likely to have identical functional properties

Therefor, my confidence in the belief that brains and LLMs should have identical functional properties is lowered by some amount, perhaps even just ever so slightly.

Not something I feel like fleshing out or defending, just an example of how I could reason about a position without testing it.

Second, I never said it wasn't testable.

Indeed, but it hardly seems controversial or major. It's sort of just an extension of existing syntax, similar to `impl Trait` in argument position.