vikramkr

No it's far beyond average human code, and it's trained in RL environments on synthetic data etc now. The average human can't solve a millenium problem in mathematics but the LLMs are able to now. Anything with a verifiable output can be RLed on to make llms superhuman it seems, and the average human is not as persistent and thorough as e.g. gpt 6.1 sol at checking every edge case and hunting for bugs because that behavior is something that's just trained into the model. Frontier models perform farm above the average human at the categories of tasks they are RLed on like code and math.

This wave is largely bugs found in pre LLM code. I'm sure there will be a smaller wave of bugs from llm code from the sonnet 3.5 - opus 4.7 era, and peeople will always write sloppy code/use dumber models under time or cost pressure, but since Opus 4.8/GPT-5.5 xhigh LLM code stopped being intrinsically worse than human code. At this point if you're shipping llm code that's lower quality than human code that's either because you don't have the money to spend on frontier models or because you actively choose a lower quality standard because of either various pressures or just not caring. if it's the latter your human code under those same conditions would also be really bad.

This wave is largely bugs found in pre LLM code. I'm sure there will be a smaller wave of bugs from llm code from the sonnet 3.5 - opus 4.7 era, and peeople will always write sloppy code/use dumber models under time or cost pressure, but since Opus 4.8/GPT-5.5 xhigh LLM code stopped being intrinsically worse than human code. At this point shipping llm code that's lower

I don't see how that's a relevant comparison. These are software bugs. Once you find them you fix them. You can't predict the harm from a small bug when AI can chain complex exploits and when your software is used in mission critical applications as a data store. There's no paradox here, bugs are bad and you want to fix them all.

A huge volume of purchases (including in the EU) are direct to consumer from Amazon. There are a lot of fly by night brands on there and you'd be surprised how many of them seem to no longer exist within a year or two. Your big company backed Samsung phone might get security updates, but the IoT junk in question is not likely to.

Re battery longevity - see the power bank bans by the TSA in the US. Maybe you are getting lucky or are only purchasing products from reputable long lived regulated vendors, but the sheer volume of vulnerable devices does not come from those products

How does it do on a more modern benchmark? Was harsh towards this in my other comment but that's a combination of this benchmark is useless, and also these open source models are pretty good and there might actually be real utility to good custom harnesses tuned for them that's getting ignored because the parity with the frontier on a saturated and contaminated benchmark makes for better headlines. I'd be curious if there's anything with these types of harnesses that make them work for realistic/relevant workflows

I was getting deja vu. Copy paste my comment from that thread to here I guess. Apparently it's really easy to get Internet points by making up a harness and showing that good open source models can do as well as the frontier, you know, on useless saturated and contaminated benchmarks from years ago that don't reflect any of the agentic work we actually use these models for. It's so stupid because these models are at the level of a decent non fable frontier model from 3-6 months ago and it's also cool that you could probably get equal perf to gpt 5.5 from qwen 3.9 27b or gpt 5.6 sol/opus 5 from a quantized DeepSeek or glm 5.3 flash. Those models were pretty useful in their time! Instead we have whatever this new form of harness/benchmark slop is

That's how i'd read it but i'm not 100% sure. legal terms can have specific meanings in a legal context that's different from general use.

but also this is one of those cases where, ok you have a law, but, that's meaningless. It's not enforceable. It's not even monitorable. It's nice in theory to write down a bunch of regulations but at best you can make a bunch of EU companies do some paperwork. anything imported from outside the EU, the company can agree to everything and then dissapear. ALl the direct to consumer slop on amazon and whatever. How is the EU supposed to enforce a law against a random fly by night brand name in Shenzhen?

From https://taxation-customs.ec.europa.eu/news/report-highlights...

> As volumes rise, enforcement is weakening. In 2025, the EU control rate fell to 65 items per million items imported. The rate of goods ultimately refused at the border due to non-compliance or because they posed serious risks, also fell to below 10 items per million.

> A large-scale customs investigation launched by the Commission in 2025 highlights the scale of the problem. Over 60% of checked toys, small electronics, cosmetics, personal protective equipment and food supplements bought online from outside the EU failed to meet EU product compliance and safety requirements.

It's just regulation theater. It's meaningless.

That is not exactly unique to PHDs. For reference, see all other politicians everywhere

Pretty sure that'll just drive people to turn it off. Or if they can't, try and set a small fire or something to get out of paying the fine

Honestly I think it's a good thing it's so hard to get AI to talk normal and I hope they keep making the voice more and more distinct. Agents should run as background tasks and automations. They should be understandable so they can communicate with you about what they did, and for observability/ai safety. Im perfectly fine reading llm speak when it's an ai telling me what work it did. When in reading a blog post with a human listed as the author though, I do not want to read an AIs thoughts that the human is laundering. Honestly I'd probably tolerate it if it was framed as "I asked my Claude to do a cool thing here is its report on what it did." But in general it's really nice to have an instant filter of - this post is a waste of time - ah this post has an actual human's thoughts

I believe the author of this article is in fact most likely a large matrix multiplication and that it is possible that no human even initiated the computation of said multiplication directly. I assume it's not hard to set up some monitor for new CVEs to try and slop out an article as quickly as possible to get as revenue

I would argue for anything security - if there is a difference between how it works in theory vs in practice - then the theory is flawed. Physics gets to have theory/practice distinctions, but engineering doesn't because the bridge can't fall down.

This doesn't have the vibes of something that's designed perfectly even in theory. This more has the vibes of "this was the least worst option"

Based on how quickly the CTF was won - yeah that seems very reasonable.

It was exploited within 2 hours of the server being fixed to be vulnerable it proved the exact opposite of shat you're saying

It was true. The author had messed up the server and it was not exploitable. It was hacked within a couple hours of the author fixing it (even with the server being down) but the author had also messed up and not put the actual Bitcoin wallet in it so it took an additional 5 hours for them to fix that. It was a mess but the vulnerability was very real

I mentioned this a couple times there but the unintended consequences of that challenge are wild. 2 hours after it the server was fixed there was a successful exploit but because it was originally misconfigured a huge number of people walk away thinking the vulnerability was not serious. At the least we now know a good way to convince large swarths of technical folks to not take new vulns seriously! Which seems not good

There are auditors and the security, plus since IPOs involve raising money from institutional investors with very deep pockets those folks are a more credible legal threat than retail

honestly the surprising thing at this point would be if america did actually pass a policy of any sort, good or bad, vs just relying on the president to yolo some executive orders now and then that change every 2 years. Plus at least for the next couple years nvidia likes open source models and the president likes the ceo of nvidia. There could be a falling out but nvidia has deep pockets to ... discourage ... a falling out.

It's worth remembering that the tariff stuff is not any sort of durable policy. It's one guy sort of doing whatever until 9 other guys a year later go 'hey wait' and then the first guy does whatever again anyway. The whole thing is just a recipe for whiplash. maybe congress remembers that they're supposed to do stuff in a few months and steps in. maybe they don't and then every 4 years we have wild swings in enforced based on which half of a percent of the voters in georgia had rainy weather where they lived in early november. More of a simultaneous "All of the above and also none of the above" deal until the dynamics of american government fundamentally shift

Tokens per second? That only really matters for things like chat and sessions where you are monitoring a session as it runs which is going to be an increasingly small percentage of usage. That's bottlenecked by human attention. Agents in the background, long running tasks, unattended tasks, all that stuff is what is even capable of driving significant usage because you can just increase consumption massively by parallelizing without being the bottleneck. I wouldn't be shocked if already the vast majority of token usage was from unattended sessions. TPS is not that important there

I think at least a few of the replies including mine are saying this is a pretty meaningless over hyped demonstration because the cnn was fine tuned on the same dataset it was tested on and the actual way the eval was structured and run is nonsensical. It's basically just slop research designed to create a fun attention grabbing headline (ironically one that trashes llms).

Presumably with the level of interest in it people have been checking what the theorem definitions are to make sure they align and I would expect Anthropic to be burning a huge number of tokens to rip into the proofs and figure out if any of them are exploiting bugs in the lean kernel etc. at a point it's as close to irrefutable as you can get

The crisis is from what it has already borne out (navier stokes + proof dump). Potential continuing improvements at anything approaching the rate of improvement they already are showing is layers of crisis on top of that.

This is not a compelling analysis.

First, the CNN is not 10 years old. The base is but it was fine tuned for this project on the same dataset the test images were derived from:

> The test: The lesions come from MILK10k, a public dataset released through the International Skin Imaging Collaboration [ISIC] in 2025

> The CNN is deliberately not state of the art. I used Inception v3 pretrained on everyday photographs and then fine-tuned all layers on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU

The test images themselves were filtered out at least, but that's not really enough. The CNN has a home field advantage because the test images are going to be in distribution with the training images.

Second, Looking at the code, for frontier models probably you should be giving them image tools and using them in an agent loop and not trying to one shot results (e.g. https://platform.claude.com/docs/en/build-with-claude/prompt...). It is also not meaningful to ask an LLM for a numeric confidence score.

Third, all 200 of the the test images were biopsied by the clinician originally. So when you ask the model "You are assisting a dermatologist" and "Decide whether this lesion can be safely discharged, meaning it needs no treatment, no biopsy and no follow-up" why would you penalize the model for making the same call the physician did even if it was benign in retrospect?

Are you evaluating the model on if it can replace the need for a biopsy itself (in which case the ground truth is whether it in retrospect was harmful or not) or in if it can match physician judgement in determining when a biopsy is needed (in which case the ground truth is if they physician ordered a biopsy)? You're asking the model to do the latter and grading it on the former.

Overall yeah probably specialized models make a ton of sense in a bunch of areas still but idk this does not seem like a good demonstration of that

yeah assuming no funny games the threat is real and the primary thing this CTF did while it was misconfigured and on the front page was convince people that the threat was not real. that is probably not an ideal outcome.

similarly to the guy from Trail of Bits talking about how it looked like a trap - apparently this PoC was not impossible for someone else to figure out, the window was not as long as you'd hope (again assuming no funny business), and the "trust me bro this exploit is real" messaging just kind of kills overall credibility and does more harm than good. As does a misconfigured CTF, potentially. People do not have a positive view of tech and especially AI right now and if there's a real threat then fueling the "it's just hype to boost stock price for IPO" narrative is maybe a bigger issue than the vulnerability itself no? Because you also tank your ability to get people to take any future vuln seriously.

Plus a broken CTF is apparently a good way to get people to not take a current vuln seriously while also working as a honeypot to get PoCs. Probably everyone involved with good intentions should make sure they're really thinking through all the implications (preferably ahead of time so there's a playbook). Meaning societal/psychological/narrative implications too and not just technical. Security of all fields should be deeply familiar with how human behavior is kind of critical, no? How many people reuse passwords? The AI industry is uniquely incompetent at this aspect. Like all the datacenter stuff where they make secret deals for tax breaks and cheap land with city councils under NDA and then are suprised that constituents react badly to being taken for a ride by an industry and elected officials they already don't trust. Like just from an optics perspective, charging for daybreak or similar frontier cyber-capable AI to defend from AI discovered/led vulnerabilities and attacks - y'all get how this looks like a protection racket right?

The worst outcome here is AI cyberwar/CVE-apocalypse hitting while the industry is treated as the boy who cried wolf even when there actually were previously wolves because everyone involved just sucks at comms and doesn't think things through and now has no credibility to organize a response.

From that latest update, your window seems to be closing now

OP just posted that there's a solve. And from a previous update thread apparently their first config was not actually exploitable? So ... seems like it is happening. Maybe not instantly, at least not if the site just remains down for hours because it's functionally DoSed, but it still is apparently exploitable.

The problem is that it's a very simple, older benchmark, and qwen was trained after fable 5. You need to test against similarly modern models so you aren't putting the models on uneven playing fields in terms of contamination. And for the stuff qwen is trained to be good at (agent stuff) opus 5.5 is light-years ahead of fable just because it's a newer model amd they RLed on it.

Though point taken that in a case where they didn't even bother making sure they were using the same qwen quants for every run or even test fable in their harness probably that is the least of the issues here.

Who said it's my assumption for code I generate? We're talking about how I evaluate code other people generated. If that wasn't my default assumption - then you should never trust anything I create because I would be naively trusting other people to maintain throwaway projects.

Are we losing our accumulated engineering best practice knowledge? Yes. Have you seen the quality of new hires recently? The ones who've cheated their way through school and don't know what a for loop is? People that have those skills aren't letting them atrophy but the chain of transmission is in crisis. Breaking news, sometimes things suck