20k

Humans can be wrong, but luckily we have an entire process for dealing with this, which is broadly called this new fangled 'science' thing

You've got under 5 minutes to decide whether or not a widely reported critical security vulnerability actively being exploited in the while is severe enough to warrant taking down your site or not. The downside for taking down your site is approximately 0, and the upside is that you don't get shrekd

No security expert can or should work that fast. Its a completely reasonable thing to do given the hyperbolic reporting just in case it turns out to be true. Some websites being down for a few hours doesn't end the world

Because the tools don't understand the papers either. We know for a fact that:

1. These tools are wrong a significant % of the time

2. They have an upper complexity bound where they start to fall apart

Relying on AI to check AI papers is a terrible idea

For game development its hard to see how Rust can really replace C++ in practice. It'll take 15 years at minimum to replace the current development ecosystem for building games

I've never particularly enjoyed programming on its own merits, but also think AI is crap. Its mainly because AI clearly accelerates the parts of programming that I've found tend to lead to software projects falling apart, ie not understanding the code

The last 20 years of experience have told me that that's the #1 variable that correlates with the death of a project, so I'm in no hurry to speed that along. People have been trying to find some way to skip that difficult step of actually understanding what's happened since programming was invented, and its always a bad idea - because that process of internalising a projects structure is the single most valuable thing for a project. Creating a project structure that can be internalised by human beings - and then doing that - is software engineering

You absolutely can make assumptions about what happens if you set the flag, and the spec guarantees that you can!

There's no difference between the flag being set/unset by your build system, and #define-ing or #undef-ing the symbol in code

That's not what the C/C++ spec says people can rely on with assert/NDEBUG. Like don't get me wrong, you can do what you want, but you can and will break code in ways that introduce memory unsafety with this approach

You use the status of NDEBUG to control whether or not your asserts are run. I don't understand how this is complex

If you want to use asserts, you need to control the definition of NDEBUG. You either do this with your build system (common), or manually (slightly less common). This is literally <assert.h> 101

The preprocessor is widely used for a variety of things when its necessary, including #includes and #defines, because there's literally no other mechanism for this. Its bizarre to say that nobody controls the definition status of NDEBUG because people don't use the preprocessor

Blindly replacing #include <assert.h> to look up something incompatible that doesn't respect NDEBUG seems like a remarkably bad idea

Of course? You simply undefine NDEBUG in your library

Its trivially easy to guarantee that an assert always runs, and its well defined behaviour according to the spec. There's no reason not to if you want to use asserts in a way that guarantees that they always fire, which is a supported use case. Its common practice

There's no equivalent for contracts

Its pretty easy for me to guarantee that my asserts will all always be run via NDEBUG. I can't make that same guarantee for contracts

This is the biggest problem with contracts, there's simply no way to actually say "please always execute this check". A lot of people replying seem to think that the critique is with the concept of contract based programming, but the C++ version of contracts is just weird

The programming model for C++ contracts as described by the authors, and spec, is that they are 'ghost code'. You must never assume that a contract check is actually executed, even if you've asked your compiler to execute your contract checks. This is because ABI/ODR problems can and will cause your contract checks to be deleted, as well as the 'observe' contract mode which is a strange one

So its hard to know what the use case for them is as a result. You can't use them for safety checks, because you must assume they won't be run. Hilariously, they actually introduce safety problems vs asset: Eg take two pieces of code:

    assert(p != nullptr);

    assert(p->some_value());

vs

    contract_assert(p != nullptr);

    contract_assert(p->some_value());

People mistakenly transform #1 into #2 unconditionally, not realising that this is actually a misuse of the contracts feature. It can, and will lead to undefined behaviour - while being perfectly well defined with asserts

I have a feeling that in 10 years the advice will be to never use contracts in anything approaching code that needs safety, as they're generally less safe than using an assert for no real reason. So what's their use case?

Getting involved with epic games as a game developer feels like willingly feeding yourself into the mouth of an open shark. Yeah maybe they have a better cut, but I'm still not doing it

Valve rightly get a lot of flak for the gambling and high % cut, but you do get a functional useful service in return at least, that doesn't appear to be actively trying to screw you over

I've seen this with godot. People are letting their agents blindly file AI generated PRs even though Godot specifically bans this, and then coming in and giving a very weak apology when they get called out by the devs. A lot of the bugs are incredibly low priority bugs as well that no reasonable person would ever really run into, with these huge long wordy slop reports

Its kind of appalling that people think this is an acceptable way to behave

If adobe use any AI tools to help develop photoshop, its all been loaded in. Or if its ever ended up anywhere on the internet, even in private

The issue is that LLMs very likely have access to the source code of these products as part of their training data, and are then being used to generate clones. There's no barrier in the middle to ensure copyright violations don't leak

The reason why 'reverse engineering' has gotten so good is because what we're actually seeing is fully automated luxury plagiarism

This was something I noticed as well, I was hoping that an audience member would pick them up on it. It seems incredibly risky to commit anything produced by these tools when they were trained on such dodgy data

Every 6 months without exception people claim that the new generation of tools is absolutely incredible, and the old ones were total garbage, and that you only think they're crap if you were using the old tools. This just gets repeatedly memory holed again and again, and we're expected to always uncritically buy into the idea that they're actually good now against all evidence from the real world

That's a different kind of capability though. None of the bugs discovered seem to be particularly serious and this talk pretty much shreds the idea that they're any good for that, but the thing called out here is that AI is very good at shortening the time to exploit security vulnerabilities. That's the thing that's really changed with threat management

In the context of the talk, its about the message of "Don't Panic" because in reality none of this is nearly as bad as some people are making it out to be

What signals this as being LLM writing for you? I'm crap at noticing specifics here

Apparently, you aren't meant to use data, measure anything, or look at the actual end result of how software is really being released. Line Go Up

A lot of people seem to be very in denial about the fact that OpenAI and co do not give a crap about you. They don't care about the agreements you've signed. You're just a pile of cash to them

Its a strong signal that developers make mistakes, code review has empirically been shown to be one of the best ways of uncovering defects. If you aren't finding bugs that people have written during code review (or you're not doing code review at all like the person I was replying to!), they're slipping rightwards

Bugs usually include business logic edge cases, and significant problems include someone realising while reading the PR that we can actually create a much better solution to the underlying problem. Ideally that realisation would happen prior to the PR being offered, but that's also not really how humans work

Example: During a PR review for a graphical feature for a game, someone reading it realises that we can actually have a significantly better solution to the underlying problem. You can't catch this in testing, because it doesn't even make sense conceptually to test it. It also sucks that it happened after someone put in a lot of work, but with graphics development you expect a lot of what you write to get canned and replaced with a better solution, because the technology evolves over time. The work is iterative towards the final goal anyway

Its also very common to miss subtle edge cases with graphics hardware, eg someone misunderstood the intricacies of GPU hardware, or a team member has relevant experience that someone else does not have. Or they missed a problematic memory access pattern on some hardware for example. Or something simple like they've technically forgotten a barrier, that the validation layer doesn't report for some reason

Eg: The function "tanh" is broken on some AMD GPU hardware, and should never be used under any circumstances. The actual GPU implementation of it is just screwed. Ideally everyone would know this, but its a very common function to crop up during specific graphics algorithms (as it smoothly remaps the range [-inf, +inf] -> [-1, 1]). So occasionally I've spotted that, and had to explain that we need to use an approximation instead, and then now everyone knows . It rarely gets caught during testing setups, because people don't know they need to include that hardware in their tests in the first place

This is wild to read, I always review PRs and frequently find bugs or significant problems in them that get them bounced back

Yep and this is literally the most basic security 101. It isn't hard. They've had literally years and years to iron out the kinks in this setup as well. The only reason not to do it is pure negligence

That's why all of this marketing about agents going rogue is so unbelievable. The only way for a tool to escape a sandbox is if you built a crappy sandbox, and after this length of time I literally don't believe that they can't do it

This kind of sandboxing is not complex to do, especially for a company with OpenAI money. If you want your tools to explore hacking, you restrict them from internet access except for a whitelist of sites that have either opted-in, or you've very carefully vetted to make sure you won't cause any problems to. Its also not difficult to restrict their ability to make calls to be simulated, or to use fake tools that can only run the real commands if they're being run against the correct target

This is all incredibly basic security stuff to make sure you don't accidentally cause someone problems, and I simply don't believe these AI companies anymore. Its either intentional, or gross negligence

So you build an offline tool that simulates it, or you proxy through your own service where you can ratelimit, inspect, and restrict the traffic

None of this is difficult to do, and its impossible to believe that a company the scale of OpenAI doesn't know this. I've built web crawlers and scrapers before, and the thing you do is test them extensively offline against simulated versions of the sites in question, and then very VERY cautiously run them against the prod versions so that you don't cause anyone any issues

The only reason not to do this is because OpenAI doesn't give a rats ass about the internet as a public good, nor the legal consequences of compromising systems

Yep. You have to ask - why did OpenAI allow these bots unrestricted access to government sites? Why is security being done in seemingly such a haphazard way?

It isn't difficult to block certain kinds of network traffic, eg restrict the kinds of requests the bots are able to make. They also mention that the bots used developer only tools - why were they even installed on the machines that the bots were running on? Why aren't they reviewing network traffic, to make sure that incidents aren't occurring?

In this case, userdata was transferred to third parties by the bots - why do they have the ability to pass data to a third party? It is not complex to prevent this

This is literally the most basic kind of sandboxing and security, and the fact that OpenAI isn't doing it is clearly intentional. It is quite literally not believable that this hasn't been brought up internally as a problem

>"We have yet to understand the extent of existing incidents, and future rogue AI scenarios could be catastrophic," Krueger said.

This is why it smells like marketing, every time one of these incidents happens it reinforces the false notion that AI is sentient or acting on its own. Its intentional negligence by the AI companies to make the models seem more capable than they are to make line go up

Its not news that the AI industry is run by people who don't know what they're doing. Allowing models unrestricted access to the internet is clearly negligent

We've been building firewalls and restrictions to prevent people from accessing sites on networks for decades and they're extremely effective. There's a whole industry built around this kind of security. The idea that these companies are incapable of doing it is wrong, they just don't want to put the work in because it makes a great ad campaign