staticassertion

I don't need to do that unless you think that neurons interact exactly the way that LLMs do? That said, we have detailed, microscopic models of neurons, the ability to even simulate brain activity, intervention studies where we can make predictions, interact with brains in various ways, and then validate against predictions, we have cognitive benchmarks that we can apply to different animals or animals in different stages of development that we can then tie to specific brain states and brain development, etc.

So we're in a very good position to say quite a lot about the brain, an incredible amount really. And that puts us in a very good position to say that our brain is very different from other animal brains, and certainly in a very good position to say that's very different from an LLM.

Now, you can argue that an LLM is functionally equivalent to the brain, but given that it's so structurally distinct, and seemingly functions in a radically different way due to the nature of that structure, I'd put it on you to draw symmetries and provide evidence of that symmetry.

> The “good deal of evidence” is everywhere. The proof is in the pudding.

I'm open! Please, by all means.

> the blog article (not an actual paper?) rightfully derides benchmarks and then…creates a benchmark?

The blog article is a review of benchmarking methodologies and the issues involved by a PhD neuroscientist who works directly on large language models and their applications to neuroscience and cognition, it's probably worth some consideration.

> Anyone who says that “statistical models for next token generation” are unlikely to provide emergent intelligence I think is really not understanding what a statistical model for next token generation really means.

Okay.

> That is a proxy task DESIGNED to elicit intelligence because in order to excel at that task beyond a certain point you need to develop the right abstractions and decide how to manipulate them to predict the next token (which, by the way, is only one of many many stages of training).

This isn't a great argument. It seems to say that in order for LLMs to do well they must have emergent intelligence. That is not evidence for LLMs having emergent intelligence, it's just stating that a goal would be to have it.

As I said, a theoretical framework with real tests would be great. That's how science is done, I don't really think I'm asking for a lot here?

> It’s like saying “I think it’s surprising that a jumble of trillions of little cells zapping each other would produce emergent intelligence” while ignoring the fact that brains are clearly intelligent.

Well, it is a bit surprising. But we have an extremely robust model for exactly that - there are fields dedicated to it, we can create simulations and models, we can perform interventative analysis, we have a theory and falsifying test cases, etc. We don't just say "clearly brains are intelligent, therefor intelligence is an emergent property of cells zapping" lol that would be absurd.

So I'm just asking for you to provide a model and evidence. How else should I form my beliefs? As I've expressed, I have reasons to find the idea of emergent logic from statistical models surprising, and I have no compelling theory to account for that nor evidence to support that. If you have a theory and evidence, provide it! I'd be super interested, I'm in no way ideologically opposed to the idea. I'm a functionalist so I fundamentally believe that we can build intelligent systems, I'm just not convinced that LLMs are doing that - I'm not far though, so please, what's the theory?

I think "novel" is ill defined here, perhaps. LLMs do appear to be poor general reasoners[0], and it's unclear if they'll improve here.

It would be unintuitive for them to be good at this, given that we know exactly how they're implemented - by looking at text and then building a statistical model to predict the next token. From this, if we wanted to commit to LLMs having generalizable knowledge, we'd have to assume something like "general reasoning is an emergent property of statistical token generation", which I'm not totally against but I think that's something that warrants a good deal of evidence.

A single math problem being solved just isn't rising to that level of evidence for me. I think it is more on you to:

1. Provide a theory for how LLMs can do things that seemingly go beyond expectations based on their implementation (for example, saying that certain properties of reasoning are emergent or reduce to statistical constructs).

2. Provide evidence that supports your theory and ideally can not be just as well accounted for another theory.

I'm not sure if an LLM will never generate "novel" content because I'm not sure that "novel" is well defined. If novel means "new", of course they generate new content. If novel means "impressive", well I'm certainly impressed. If "novel" means "does not follow directly from what they were trained on", well I'm still skeptical of that. Even in this case, are we sure that the LLM wasn't trained on previous published works, potentially informal comments on some forum, etc, that could have steered it towards this? Are we sure that the gap was so large? Do we truly have countless counterexamples? Obviously this math problem being solved is not a rigorous study - the authors of this don't even have access to the training data, we'd need quite a bit more than this to form assumptions.

I'm willing to take a position here if you make a good case for it. I'm absolutely not opposed to the idea that other forms of reasoning can't reduce to statistical token generation, it just strikes me as unintuitive and so I'm going to need to hear something to compel me.

[0] https://jamesfodor.com/2025/06/22/line-goes-up-large-languag...

Doesn't ripgrep support indexing too?

Nice, thanks! Yeah, so exfil is definitely still a thing to watch out for, even if you run in an unprivileged env.

Yeah idk what you're going on about lol

> We both know that you are not getting that published in a reputable journal without a lot of effort beyond merely 'publishing the language I created'. But sure, you can get something on arxiv.

lol what? There are papers on programming languages all the time.

> 1. One is something that has been done many times before and the other an unsolved problem. It doesn't take a genius to see one estimate is likely much stronger than the other.

Building a compiler for a new programming language, building net new code, etc, is all stuff that was unsolved / had not been done before.

> 2. A moderately interesting open research problem is not the same thing as a moderately interesting problem and you seem to be conflating the two.

Feel free to explain the difference, I guess.

> What do we have that apes don't, and which directly enables intelligence?

Again, there are multiple fields of study with tons of amazingly detailed answers to this. We know about specific proteins, specific brain structures, we know about specific cognitive capabilities in the abstract, etc.

> What do we have that LLMs don't?

Again, quite a lot is already known about this.

This feels a bit like you're starting to explore this area and you're realizing that intelligence is complex, but you may not realize that others have already been doing this work and we have a litany of information on the topic. There are big open questions, of course, but we're definitely past the point of being able to say "there is a difference between human and ape intelligence" etc.

> "More success" is a bit vague in this instance but building a compiler that would take a single programmer 1 to 3 months is not comparable to this result regardless of whatever similarity exists in time completion estimates. That's the point.

I guess we just disagree on this. It's not clear to me that these are totally different in terms of what they represent.

> You can publish a paper (and in fact the researchers plan to) off this result. A basic compiler is cool but otherwise unremarkable.

Publishing papers means very, very little to me. I can publish a paper on a programming language, you know that, right?

> You are leaning too hard on how long the researchers (who again did not manage to solve the problem in their attempts) estimated this would take and the "moderately interesting" tag of again, what was an open research problem.

I obviously estimate my "leanings" as being appropriate. I'm just using the researchers direct quotes. Factually, they had already come up with the approach that ultimately panned out. Factually, they estimated that a human could do this in some timeframe. What am I overly leaning on here?

> This, alongside a few math results that have cropped up in the last few months is easily more impressive than the vast majority of work being done with LLMs for software.

I think both are impressive, I don't know that I would draw some sort of big conclusions about it at this point. I definitely wouldn't draw the conclusion that AI is better at formal mathematics than producing software.

But... we're talking about our intelligence. So obviously it's quite relevant. I didn't say that AI isn't intelligent, I said that we have good reason to believe that our intelligence is unique. And we do, a lot of good evidence.

I obviously don't believe that all intelligence is related to specific brain structure. Again, I'm a functionalist, so I believe that any structure that can exhibit the necessary functions would be equivalent in regards to intelligence.

None of this would commit me to (a) human exceptionalism (b) LLMs/ Agents being intelligent (c) LLMs/ Agents being intelligent in the way that humans are.

You're just defining intelligence as "undefined", which okay, now anything is anything. What is the point of that?

Indeed, there's quite a lot of work that's been done on what these terms mean. The fields of neuroscience and cognitive science have contributed a lot to the area, and obviously there are major areas of philosophy that discuss how we should frame the conversation or seek to answer questions.

We have more than enough, trivially, to say that human intelligence is distinct, so long as we take on basic assertions like "intelligence is related to brain structures" since we know a lot about brain structures.

I don't get what either of your points is intended to demonstrate. Let's revisit the first post I replied to:

> It's deeply surprising to me that LLMs have had more success proving higher math theorems than making successful consumer software

As far as I can tell, they absolutely have not had more success in this area relative to making successful consumer software.

Writing a complex parser or certainly a compiler is a 1 - 3 month project, for example.

Again, I'm not trying to downplay this, but to frame this accurately. I think an AI being able to build a parser/ compiler is cool too.

> One of the more strange phenomena with machines getting better and the incessant need (seemingly driven by human exceptionalism) to downplay each result, is that you just end up belittling humans in the process.

I don't believe in human exceptionalism at all, don't attribute positions to me.

> I had previously wondered if the AI’s approach might be possible, but it seemed hard to work out.

They didn't solve it, that's fair. They did consider the approach already.

I actually still don't see the source for them trying several times, but we can take that for granted. Regardless, as I said:

1. It's labeled as "moderately interesting"

2. They said that they expect an expert could solve it in 1-3 months

3. They had already come up with the solution that the AI had but weren't convinced it would have worked

So how big was the gap here, do you think?

That's interesting context, where do you see that? I'm going off of the label "Moderately interesting".

edit: I see in the full write up that the contributor says that they'd estimate an expert would take 1-3 months to do this. They also note that they came up with this solution independently but hadn't confirmed it.

Is this novel? It's new. But we already know AI can generate new things, any statistical reassembly of any content will generate new things.

It's not to downplay this, but it's unclear what "novel" means here or what you think the implications are.

They haven't, not at all as far as I can tell. This math problem appears to be a nice chore to be solved, the equivalent to "Claude, optimize this code" or "Write a parser", which is being done 100000x a day.

> So no. LLMs are not brute force dummies. We are seeing increasingly emergent behavior in frontier models.

Woah! That was a leap. "We are seeing ... emergent behaviors" does not follow from "it's not brute force".

It is unsurprising that an LLM performs better than random! That's the whole point. It does not imply emergence.

It doesn't. I actually completely reject that theory, and it's nice to see that Wikipedia notes that it is "controversial". There are extremely good reasons to reject this theory. For one thing, any quantum effects are going to be quite tiny/ trivial because the brain is too large, hot, wet, etc, to see larger effects, so you have to somehow make a leap to "tiny effects that last for no time at all" to "this matters fundamentally in some massive way".

It likely requires rejection of functionalism, or the acceptance that quantum states are required for certain functions. Both of those are heavy commitments with the latter implying that there are either functions that require structures that can't be instantiated without quantum effects or functions that can't be emulated without quantum effects, both of which seem extremely unlikely to me.

Probably for the far more important reason, it doesn't solve any problem. It's just "quantum woo, therefor libertarian free will" most of the time.

It's mostly garbage, maybe a tiny tiny bit of interesting stuff in there.

It also would do nothing to indicate that human intelligence is unique.

> I don't know why I am still perpetually shocked that the default assumption is that humans are somehow unique.

Because, empirically, we have numerous unique and differentiable qualities, obviously. Plenty of time goes into understanding this, we have a young but rigorous field of neuroscience and cognitive science.

Unless you mean "fundamentally unique" in some way that would persist - like "nothing could ever do what humans do".

> People constantly make comments like "well it's just trying a bunch of stuff until something works" and it seems that they do not pause for a moment to consider whether or not that also applies to humans.

I frankly doubt it applies to either system.

I'm a functionalist so I obviously believe that everything a human brain does is physical and could be replicated using some other material that can exhibit the necessary functions. But that does not mean that I have to think that the appearance of intelligence always is intelligence, or that an LLM/ Agent is doing what humans do.

I really wouldn't worry much. Over the last decade of rust, very few of the articles exploring new syntax have turned into anything controversial by the time they were merged (I can't even think of big rust syntax changes other than `impl T` in arguments etc). The singular example really is async/await and, having been quite worried about it / against it at the time, it was really nothing to be concerned with at all.

Nice, yeah I think they should start to migrate to that behavior by default.

No one has ever said that supply chain attacks are limited to npm.

I assume this is because it is modeled after git tags, and at this point it would be a major change to move away from this. But it should probably get started at some point.

Their question isn't about pinned versions, it's about immutable versions. The question is why it is possible to change what commit "v5" refers to, not "why would you want to write v5".

You already don't get updates pulled in with the system unless they swap the version out from under you, which is not a normal way to deploy.

You only need internet access to grab the image, I don't think trivy requires internet access itself. All of my image scanning tools run in isolation.

Yeah, we got a signed letter of engagement from our auditor, which was enough to unlock a customer without having to go through any sidestepping process.

> You can get a long, long way without SOC2;

Yes, that's true. I edited my post to be a bit clearer about this. When you need a SOC2 is going to depend a lot on your business. Lots of companies can make exceptions very easily. Type 1 is easy, I would highly recommend starting there pretty much no matter what since it'll be good practice before your SOC2.

> The idea that SOC2 forces you to do important stuff gets it backwards;

It's the goal behind SOC2. You're assuming a company has a security practice that informs the SOC2 but I think the idea is that companies have no security practice and the SOC2 is what forces them to sit down and build one. What you're describing is more like what happens when a company that actually cares about security goes through SOC2 - you take what you have, put it into a NIST format, and map minimal controls from your practices to the CCs. Most companies have nothing to start with.

It's complicated. In theory, SOC2 forces you to do some important stuff, like define your threat model and say "I can mitigate against the threats and prove that my mitigations are in place". The problem is always that the companies that care don't need it but are burdened with it while the companies that don't care will just checkbox their way through it. It sort of enforces a very baseline security posture, in theory, but the major win of "We've thought our security through" is more of a choice - SOC2 can't actually force you to care.

A ton of these SOC2 vendors take all of the potential good parts of SOC2 out of the equation, building the threat models for you and then you just hook up your gsuite/ github and they check boxes for you or tell you to flip a policy here or there. Delve took this to the extreme by not even asking you to flip the checkboxes.

That said, it doesn't matter if it's legit. Everyone is SOC2, and part of being SOC2 is that the vendors whose products you purchase are SOC2, so it's not a choice - you have to be SOC2 if you want to sell (industry/ product specific, but at some point it'll be clear if it applies). If your goal is security, well, SOC2 is irrelevant.

Ultimately, you'll end up having a separate compliance team to manage SOC2 and you'll actively try to keep "real security" from it because real security has to change over time. You'll encode the absolute minimum possible into your compliance for that reason so that you can easily pass every year and then, if you care about security, you'll invest in that separately.