Hacker News Reader

Comment by dangelosaurus | Hacker News Reader

This is my team’s work. I’m happy to try to answer questions or take questions back to the team.

And we're hiring! https://openai.com/careers/search/?q=cyber

Do you have a list of the publicly discussed or fixed ones?

(As someone who used to handle nonpublic Linux kernel security reports and is still very interested in this stuff, I’d love to review a small sample of these reports to get a feel for what kinds of issues are being unearthed.)

Also, please don’t stop at LPEs and information disclosures. If you find cases where Linux can be convinced to misbehave or where the implemented semantics don’t make sense, these are good, too. I’ve found plenty of interesting kernel bugs, including some juicy LPEs, and almost none were memory safety errors :)

It's a bittersweet result, like finding asbestos in your house. I'd rather know than not, but I would much prefer not having it in the first place.

I understand this work was based on a dated snapshot. Will there be any continuous effort, scanning new PRs as they land, to prevent new weaknesses from being added?

(I am the author the pipeline, just created an account to answer some questions :D) My goal is to work towards a vuln-free Internet. What I observed is that stronger models can find many more vulnerabilities that weaker models cannot. If compute permits (there are other projects to secure besides Linux), I plan to scan Linux again with every newer generation of cyber models.

Have you given thought on how you can use this info to estimate how many vulnerabilities there are?

It will be difficult to claim that the problem has been solved without an idea of how big the problem is.

Given that vulnerabilities are almost entirely unknown, you are inherently trying to measure the undetected.

Is there a way to hypothesize how many vulnerabilities there are and them make a prediction on how many of those will be found by a model to get a sense of how accurate the estimate is?

How can they estimate the existence of a vulnerability that has not been discovered and documented? The hubris to assume zero latent factors!

From what I understand, biologists have the same problem when estimating the number of undiscovered species. One technique is to look at very popular, well studied biomes, and extrapolate the results to less studied biomes. Or ask niche specialists to guess different groups of species and add up the results, which is usually better than trying to estimate the population as a whole.

The errors bars will be huge, but it's better than "zero or more".

I think it is hard to estimate what we don't know. In our first scan when we found 166 vulnerabilities, we wouldn't be able to guess there were 900+ more vulns to find in the code base. We are in the same situation again: now we find 900+ vulns, are there no more vulns to find, 1k more, or even more than that. I think it'd be an interesting benchmark problem if we can have a formally proved safe, large program as the ground truth, inject synthetic vulnerabilities to it, then ask the model to exhaustively find vulns, and use that as a benchmark results.

You’d know better than anyone that nothing can truly be vulnerability free. Even if the attack surface were only the code it would be impossible. But, the attack surface includes you creating the tooling. It includes me the end user. Not just our actions, but our thoughts, conscious and unconscious.

His contract with OpenAI (a line item in the marketing budget) doesnt allow him to discuss these things truthfully. They have tokens to sell.

Careful, there are upsides to that asbestos. It will last and survive much longer than anything else, avoid messing with it and you'll be fine.

Also there are varying types, its worth getting it tested if you need to do a lot of work with/around it.

As models get more advanced and generate an increasing throughput of increasingly higher quality results, how are you determining when to trust its results and when human review is needed on what?

E.g., 948 results is a high number to review in a timely manner unless your team is very large or spending several months combing through the results

EDIT: nvm i shouldve kept reading, answered

> At this scale, manually investigating every report is impractical for our research team. We therefore focused on validating findings that a) were reachable by a low-privileged user, b) required no special capabilities, and c) produced measurable evidence of memory corruption.

Why would they review the results? They want to publish flashy metrics, this is a marketing team not an actual cyber dept. Cant risk discovering that 35% of your results are due to reward hacking and ruining the headlines you were hired to create.

Thanks. I'm not a programmer so I can't understand what is happening in the kernel by looking at the changelog, so I'll ask: Are the problems remediated structurally so that nothing like it can occur in the future, rather than just patching it up? Are there "lessons learned" that are being applied and documented here (like, "this problem looks like it's shape transports to other subsystems as well, let's fix them all in a disciplined way")?

I can't comment specifically on the Linux kernel piece, but I want to point out, broadly, that security has generally been an afterthought, and very little gets invested into it in terms of time and money - as a general rule.

Multiply that behavior times the number of commercial and mobile and b2b apps over 50+ years, and that's where we are.

There will likely be many more leaks, and hacks, and outages. And then in trying to be proactive to fix old systems, that will cause new proactive outages, and then eventually many things will be better.

But, it's going to take a lot of time and investment, and a lot of (majority) of currently unsupported devices will have to be complete replaced / scrapped / ewaste. On the scale of Nation-State initiatives.

This is across all operating systems, firmware, software and devices of all kinds, etc.

LLMs to security are like a bomb went off. We still have to wait for things to fall over before the dust will even clear enough to see what we have to work with.

I'd like to see a break down by age of code and part of kernel (drivers, networking, core ,etc ..)

So far OpenAI's security work has focused on identifying vulnerabilities in open-source code. It occurs to me that using LLMs to scan binaries for supply chain issues and implants, and publicizing the results/certifying particular hashes as safe, could also be very valuable.

Presumably, the barrier to sophisticated supply chain attacks is falling rapidly due to LLM agents. Such attacks could even be utilized by a rogue AI. OpenAI could easily develop internal expertise in detecting such attacks via the following method:

* Train an LLM agent to create binaries with subtle backdoors/bugdoors

* Train another agent which ingests data in the form of clean binaries and poisoned binaries, and learns to tell the difference

* Train in a loop, where the first agent "wins" if it fools the second agent with an implant judged useful by a 3rd un-incentivized agent; the second agent "wins" if it's able to detect increasingly obfuscated backdoors/bugdoors

Over time you could train a small model for high performance on this task, which continuously scans open-source package repositories, firmware, OS release images, etc. flagging issues, and certifying hashes as free of apparent malicious code. To save on hardware resources during continuous monitoring, it could also learn to scan binary diffs. A certification API could then be exposed for use by maintainers of apt-get and other package tools.

This could be excellent as a goodwill project to help prevent security apocalypse and harden against rogue AI.

Originally had this idea in this subthread: https://news.ycombinator.com/item?id=50379912

The high number is really impression to the point of making me skeptic. Are these distinct vulnerabilities or are a lot of the related in the way vulns often are?

The article says "The target virtual machines ran a stock desktop operating system". Are the vulns in the kernel specifically or does this include software running as root?

These are distinct vulnerabilities counted by the number of patches. The patch generation system performs variant analysis. So, each patch can potentially stop multiple exploits. The raw exploits number is higher than what we reported in the post. We don't want to cause the confusion why #patch != #exploit, so we used the smaller number there. These vulns are all kernel specific. Userspace software running as root was not in scope of the analysis.

I'm not skeptical. Linux has had so many LPE bugs in the past that most people consider it to be a "best effort" thing and nobody actually relies on it. Shared machines are mostly used in companies or universities as a kind of admin tool - the users are more or less trusted.

The only exception I know of is Android. It would be interesting to know how many of these vulnerabilities work there. Can't be zero anyway...

Unfortunately no remote openings and a very limited set of offices. You’re missing out on a lot of talent.

25% hybrid is such a slap in the face. If it can be 75% remote it can be 100% remote and not done in the bay area.

Given their salaries, I really doubt it.

I think you'd be surprised that top tier talent (and people in general) usually don't worship money and aren't willing to degrade their standard of living for an extra 75k annually after taxes.

I suggest you update yourself on the monetary incentives/benefits for employees that US frontier labs provide. 75k is a drop in the bucket of the amount of money those employees gain in an IPO event, let alone bonuses.

You should also upgrade yourself on what people expect nowadays. Might not be top talent but before I return to an office, hell will freeze over.

You mean an extra 750k

You guys have been watching too many AI silicon valley hype youtube content.

OpenAI has 5k employees now.

Yes, you see these headlines about ml researchers getting paid a ton, well thats because there's 5k of them on Earth.

Unless you're a VP or PhD you're not getting life changing compensation. And guess what, all those people can get similar compensation elsewhere without slaving away 4 days a week in an office in San Francisco.

Your average employee is not making an extra 750k. You know you can look at compensation online you cant just do vibes based analysis and just invent your own reality. Find me a position at OpenAI thats paying 750k compensation.

Literally from the link above: https://openai.com/careers/researcher-frontier-cybersecurity.... $400k in cash plus substantial equity is going to easily cross 750k

The person hired for this role will almost certainly have a PhD.

Right, the context was missing out on "top talent". Top talent gets those crazy salaries.

A lot of us already make a lot of money. It’s not about the money. It’s about the work. I would love to work on the things OpenAI is working on. I don’t expect it would be much of a pay raise. But I’m not moving my family for it.

I wish these blog posts included statistics on the categories of deficiencies that cause these vulnerabilities, and proposed measures to avoid them

A system that admits THOUSANDS of such vulnerabilities is certainly flawed in several fundamental aspects

It sounds like they can’t go through them all enough to have that data yet.

Kernel is far too big to throw at a single LLM, just like a human. It is.. I think somebody said 40mil lines of code.

So you have to chunk it down to workable pieces, and then if you collect enough data of that process you can look at higher level stats.

Same for every other existing piece of software, btw.

Thanks for trying to help on the defensive side!

And, well, since you didn't _specifically_ ask for questions about the blog post:

Why are you not quitting your job in protest and screaming at the top of your lungs about it, seeing so closely what these models are capable of that your company is creating? There exists now something that is able to find 948 kernel LPE vulnerabilities. How, exactly, will humans be able to control and contain, say, next year's models for a nonzero amount of time?

He quits and screams from the rooftops. Then what? That stops the approaching tsunami? This is out of anyones control already. Pandoras box is already opened. The person you replied to merely found a more buoyant life raft than you or I.

We might end up with a secure kernel, and you want them to stop?

I'm going to assume that there was "old" code that caused LPE vulnerabilities, but, The Linux Foundation have their own project Sashiko (https://sashiko.dev/), which analyzes each commit to the Linux Kernel for bugs and vulnerabilities using Googles models, my question is then, how many of the vulnerabilities you found were also caught by Sashiko?

If Sashiko had found them then they wouldn't have been merged, so my guess is 0

That's fair, but I guess the question is whether analyzing each commit can find vulnerabilities or it takes more than that (aside from using Google models). It is possible they might have dismissed Sashiko review because they thought it wasn't possible or didn't make sense.

Then the question becomes how many vulnerabilities Sashiko didn't catch (If they use/review Sashikos review)

I’ve always felt that the problem after finding all of these is that you’re going to have to patch and upstream them. I assume you’ve probably asked your models to do that but at this rate you’re probably bottlenecked on people to review these and then the kernel maintainers are likely also heavily time constrained on their side to do the same. What ideas do you have to improve this situation? Do you foresee a process where you’re not just finding an ever increasing number of bugs and instead can bring that number down with different approaches to attacking the problems (e.g. designing new mitigations?) I don’t really see this described in your career listings so it’s hard to tell what you’re doing in preparation of better models :)

My 2 cents: it would be beneficial to involve companies like RH and the teams behind many distros to work through those, it's not just about validating and patching but also about delivering them to the systems involved, a lot of people need to be involved, hence collaboration is required

> Enable JavaScript and cookies to continue

Would be nice if the page could be readable without.

> This is my team’s work. I’m happy to try to answer questions or take questions back to the team.

...and then proceeds to never answer a single question. Classic.

Congrats! Two questions:

Greg Kroah-Hartman showed [1] how thousands of AI reported bugs can shrink to a handful after maintainer review. How many of your 948 LPEs have been independently confirmed upstream, and do you expect similar consolidation?

Are there plans to make these cyber capabilities more accessible to independent security researchers? Current restrictions risk putting legitimate defenders at a disadvantage.

[1] https://www.youtube.com/watch?v=NnV_cWeoo5Q

Reply on news.ycombinator.com