Hacker News Reader

Comment by dfee | Hacker News Reader

There's an anecdote about curl, but I'm not sure if there's enough in here to draw conclusions from. 20% of vulnerabilities have been made public, and many of those are marked as "withdrawn" or "duplicates". However, it's unclear whether the remaining 80% have not been made public because they're not fixed or real. In other words, this is analysis of survivors (public); not much can be said about those still under embargo.

View on HN

I'd expect the remaining 80% are delayed not because there's something statistically special about them, but from the massive workload thrown at humans in the loop.

Anthropic needs to teach their systems to propose a fix too. Detecting a coding problem and proposing a patch is what we should expect from a company with supposedly the best coding LLM.

Claude is perfectly capable of fixing most vulnerabilities; if that's what maintainers want, they can just ask for it. If the concern is cost, Anthropic offers a Claude for OSS program.

The reason Anthropic isn't proactively sending patches is, I suspect, that those patches would not be welcomed, because trust in the models isn't currently at the point where maintainers are willing to merge them without human review, which takes approximately as much effort as writing the patches themselves. You can imagine the wailing and gnashing of teeth about "slop PRs" and "extractive contributions" that'd be all over HN.

Anthropic's oss-scanner includes patches. We've received a batch and I'm working on some of it.

A good amount of the patches is straight garbage. It's not helped by the fact that a good amount (20~30%, maybe more) of the issues is also junk themselves.

I'll hopefully be able to share numbers (or data) after we're done with the batch.

70-80% not being garbage sounds like a good thing :)

I suppose it depends on:

- How many patches you receive

- How easy it is/long it takes to distinguish the good from the garbage

You're assuming the other 70-80% is actually good. It's not, there's a whole spectrum between "junk" and "good". A whole bunch of this stuff is "technically correct, technically even a security issue, but also just plainly not of actual importance".

You get free meals, 70-80% of them tasty on different degrees, and the rest burnt. That's OK because free meals, right? Except the small letters say you have to eat 300 meals a day.

I won't claim that my experience is representative, but what I've seen is closer to 50% being false positives, and then maybe 20% of the remainder being bugs but not vulnerabilities (aka lower priority bugfixes). The suggested patches have a much worse hit rate though, to the extent that I accepted only a small fraction after reviewing.

That said, an automated system generating only 30% true positives is probably still ahead on ROI as long as we're talking about tens of findings and not hundreds. The amount of human time spent investigating things that turn out to be false positives being the main difference for how over or underwhelming it is.

Yeah, I was being "optimistic" in my estimate. And of course with a batch, you start at the high severity end, so it gets worse as you progress through. (Well, I assume at least, even some of the "critical" was just plain "not even a security issue" in our case.)

> If the concern is cost, Anthropic offers a Claude for OSS program.

thanks! It turns out that I am apparently qualifying for it and I was unaware

lets see whether Anthropic agrees

> Anthropic needs to teach their systems to propose a fix too. Detecting a coding problem and proposing a patch is what we should expect from a company with supposedly the best coding LLM.

Of course it's trivial to have an LLM generate a fix, too. They probably do. It doesn't mean a project (1) wants to take an LLM-generated fix at all, or (2) can immediately commit an LLM-generated fix without review (which takes time).

Maybe not. And that’s totally fine.

But right now they’re not even offering one. They’re just putting the burden on the project without even trying to help fix it.

They have all the resources in the world, and they won’t even try.

And that’s of course assuming it’s a real bug, as we’ve seen a ton of them don’t seem to be. The last time we went through this, curl reported not only that a bunch of them weren’t bugs, but some of the bugs were listed previously on their mailing list, and that’s probably where the LLM got them.

So not only is it finding small bugs. And nonbugs. It’s reporting previously found bugs.

Pilling work on maintainers. Over and over again. To help? No. To make big numbers to show off for their IPO documents.

> But right now they’re not even offering one. They’re just putting the burden on the project without even trying to help fix it.

> They have all the resources in the world, and they won’t even try.

You sure about that?

https://news.ycombinator.com/item?id=50448558

Could also be that most of the findings are low signal noise. Even the batch that was sent to curl was one medium and 39 low severity. One can only assume that the vast numbers of findings that haven’t been sent are even lower severity.

I'd be careful to take an examplary projects like curl and assume that level of quality is the average or even close to it.

Compared to curl, even the Linux kernel is garbage, and the Linux kernel is probably higher quality than all of the enterprise software that runs the world.

And on top of that, it's a bit negative. How many millions of dollars is one perfect-10 scoring CVSS finding, total destruction style RCE, found in curl worth? Considering how many people use it.

It's worth zero millions dollars to curl. It might be worth something to attackers, but "wide use" is not the same as "the ability to compromise everyone" because most of the time, curl isn't used to automatically fetch untrusted URLs in a way that's controlled by the attacker from an environment they care about. It's worth much, much less than a 0-day in an iPhone or PHP. You'd need a buyer with a very specific itch to scratch and there won't be a bidding war.

By extension, if a vuln is worth something to attackers, it's probably also worth some fraction of it to their potential victims. But if you're an interesting target, your defense can't be based on the ability to find out about every single software defect ahead of the time. You need layers. And if you have that, the importance of proactively acquiring every 0-day is diminished.

Your layers need to be secure too, or you just have colanders stacked together.

A client-side bug in a command line utility that is mostly used as a debugging tool or glue in shell scripts where the URL is not attacker controlled? Probably close to nothing. Maybe if you can find some other software which lets you run curl with an attacker controlled URL, you could package it up as an exploit for that piece of software, but it's not a common config.

Curl is also a library. Just saying :-).

Fair, libcurl is pretty widely used in PHP, but it has to be in the core.

Still way less impactful than the PHP2Shell issue.

libcurl is used in a lot more than PHP. libcurl is installed on more than 20 billion devices.

... 50% of which probably never get connected to the internet, and 50% of the remainder probably only connect to one hardcoded domain name.

The most likely explanation is that it isn't worth submitting a massive stack of bullshit to human review.

If they actually were sitting on a PR gold mine of vulnerabilities they wouldn't be throttled on review resources while artificially inflating/misreporting their numbers. They would be diverting resources to clear the review queue.

It's fake and it always was fake.

Reply on news.ycombinator.com