At work we have put a full halt on feature work because the number of security related tasks has become staggering and urgent.
We will hopefully get over this wave and be able to move forward with much more secure software, but it's crazy right now.
We will hopefully get over this wave and be able to move forward with much more secure software, but it's crazy right now.
i would say the recent slew of vuln reports/breaches making the news rounds are doing work!
This is the silver lining i suppose!
Agreed however getting humans to understand probability is tricky.
I had this discussion with my son recently about considering the severity of outcome multiplied by the probability of that outcome been the way to look at some things.
Taking options off of the client’s table is the core responsibility here.
In the end, it's all business decisions. If it's something I don't like as a customer then it's to be infantilized.
(If you disagree then please don't complain ever again about LLMs refusing to help you reverse engineer a piece of code or explain how a virus works.)
I’m not saying that’s right, I’m saying that’s how a lot of businesses think.
Bottom line: it's a good thing. My software will be better after that. Even if it's currently not good for my ego (and my health). But I'm glad for all the tools that help me improve quality.
And sure, I can just ask LLMs to fix the bugs, which works well half of the time, and produces bad slop in the other half. This is where human expertise and judgement is still necessary to avoid letting our code bases drift into LLM spaghetti hell.
While I did fix most of the things because while they might not have been exploitable they were bugs waiting to happen, in all but one case I haven't been very impressed by any of the LARGE IMPORTANT SECURITY HOLES that were found.
But that doesn't matter for me. It shows me real bugs and less than 1% false positives. Some bugs are truly nasty and urgent, most are trivial and not exploitable at all, but they're bugs and so I'm fixing them.
Mythos isn't mythically superb for being the best LLM model, but only for not having these annoying guard rails that make all others randomly refuse to cooperate when you need them the most. Whoever invented guard rails - please stop! - they just slow the good guys down in the race.
> This has been stressing me out
> the bugs follow me in my sleep
Sounds like you need some vacation.
If you don't mind answering, how come? Is it specifically that you dislike how abusable your code was, or something else?
To err is human, I know that as much as anybody. Before the advent of LLMs, there were several (human) audits of my code for vulnerabilities, but they found little, which I was proud of. And now, Mythos had a quick look and found hundreds and hundreds of bugs (many of which were critical and even more were really emberassing).
Of course, I knew there were bugs that just weren't found because the auditors weren't looking closely enough, didn't have enough time/budget etc., but this unspecific feeling is different from having all the bugs accumulated over a quarter century slapped in your face at once.
If you get a good auditor they can find a ton, but most are box checkers there to make the audit company as much money as possible.
After my Mythos experience, I was wondering if there is really still a market for human code audits. I guess yes, but the work will be different, just like coding (and marketing and so on...) is different now. You still need to have an expert to wield the new powerful tools.
LLMs finding unpatched vulnerabilities faster than they can be reasonably patched, bringing all other software development to a halt.
And even if the impossible occurs and LLMs hack everything that's connected to the Internet, so what? That would be inconvenient but it wouldn't cause doom. People would muddle through and eventually fix the critical problems, just like they always do.
I wouldn't be shocked if in 1-2 years non-security vetted software released into the wild will be hacked on arrival, like Windows XP without service packs used to be.
If you consider it to be productive work, it's a massive productivity boon happening right now. Of course, product managers might disagree...
Also, in the article they mention a significant number of patches for older versions as well.
It's not that the LLM code is that great, but that many overestimated the human code and the human review.
Also: most tests are positive tests. Tests rarely observe the negative space: that which is there but shouldn't be there.
looks at particledb
What they are really good at is writing boilerplate and reading API documentation / example code. Avoiding subtle errors is not what they're good at in my experience.
I think comparing the resources they spent at it would be a more apt comparison.
AKA if you tasked 150 mathematics departments to only focus about that problem, and stop doing things like teaching and researching anything else, you might get close to the amount of resources that these companies put into solving the problem to get a press release.
Read about the entire huggingface attack and why it happened.
This is one of the least talked about and most useful features in any harness I would say, the review feature. I have never not found issues (either security or implementation) running it, no matter the code base or language.
Some people will much appreciate the holes in Postgres that are being plugged. Many more will be bored with patch releases until new features rollout.
A software affecianado appreciates patches, and might do well to teach laypeople the benefits of a good defense.
Imagine that the whole city has to install iron doors and window grilles, because wooden doors and glass windows are not safe enough anymore.
That's a lot of work, but is it productive in the end? Probably not.
With AI it's the same. World was fine with lax security approach before. Today it is not fine. Tomorrow we might need to prove every code line before shipping, will that be more productive? Hell, no.
I don't think that's true.
In medicine, science often doesn't know how specific drug really is working. They just do a lot of tests and if these tests are positive enough, they ship it. I think it's good analogy. Biology is very complicated and it's out of our grasp to truly understand how various parts of it interact with each other on so many levels. But if something helps, it's stupid not to use it just because we don't fully understand it. Sometimes it bites. Sometimes it bites 30 years later. But we still follow that approach.
Software is similar. The complexity of the systems we build is enormous, when we're taking in account all the moving parts. I just read that someone emits radiowaves toggling memory bit and ex-filtrating information. How can you account for that? Probably you can't. But we still need software for good things.
SWE's are not a serious bunch when what they do can affect the entire world.
Obviously this doesn't work on the internet since most hacking comes from overseas out of our control. Which makes me wonder if there is a possible future where we just block packets coming from countries that don't lock up the hackers.
Not obvious to me. I am generally a fan of Cyberspace manifesto and the free Internet, I think it's great people can reach outside their ingroup and talk to strangers and ostensible enemies.
But, that doesn't mean we should give up physical security, the guarantee that the connecting party comes from a certain physical location (state, company). I think allowing connections from anywhere by default is fundamentally flawed security architecture, for certain things.
If you're running a proxy, it's your responsibility to keep NKans out.
I am not saying close down all the Internet. But I do believe there should be tiers.
You cannot make defense a binary. We live in an adversarial world. It is required to survive.
You have a perverse view of product ivity
The algorithm is: if the threat model is valid, there are vulnerabilities and you either have some software everybody knows about or a gang of hackers specifically targeting you, you should patch.
The part of the industry that profits from security theater will want you to patch no matter what, even if your business goes down in flames due to downtimes and new bugs introduced by patched components. They won't remind you to assess the threat model first.
Digital threat models are very topologically flat. One exploit made in one place can be instantly copied and used anywhere.
Also once hacked you can be used as a base of operations to attack others. There is no neutrality on the internet.
> That's a lot of work, but is it productive in the end? Probably not.
Your imaginary scenario defines the solution to be pointless. If that same city was subjected to weekly riots smashing windows & doors, and people regularly died, stopping "productive" work to install things that keep people alive and productive? That's productive.
The point it, this was not happening before. LLMs made cheap cyber attack possible. But it didn't make the implementation of fixes correspondingly cheap. So now everyone needs to spend extra money/effort to install grills to cover where LLMs fall short.
For postgres, this absolutely makes sense though as long as people are willing to shell out the profits from LLM use to support postgres development. That is one way this could workout for everyone..
Then they have a pikachu face when someone smashes their car window to steal something worth $50 they left on front seat because “who would do that?”.
What is even better thief will most likely pawn this for $10 but for him it is effortless $10.
One way to think about it is that production allows you to pay for stuff. Not necessarily directly but by way of trading. It is the output you use to buy things like security.
Consider an apple farm. What does the apple farm produce to pay its bills? Apples. They could also dedicate resources to pears, &c, and that would add to their production -- to the goods they produce, the goods with which they (indirectly) purchase tractors, fence posts, &c. They could also dedicate resources to security -- more fences, &c -- and this may even be prudent. Maybe thieves are getting taller. This doesn't really add to production, though; at best, it maintains production at the status quo ante (assuming the fence is effective).
Security bugs are just bugs, as Linus likes to say. And the bugs are often also crash or correctness bugs (use-after-free for example usually causes segfaults). For the most part this actually is a productivity boon in the sense of getting higher quality goods out the other side.
To use your apple farm example, it would be like an apple farm where, if the apple rots, it's sometimes just gross but also sometimes it explodes and kills you. Would it be productive to reduce the rot rate of the bomb-apples? I think so!
If that same city was subjected to weekly riots
That's productive.
The software is better in the end but not all of it will be productive.
https://en.wikipedia.org/wiki/Parable_of_the_broken_window
Basically, if you accept this line of reasoning, then someone going around breaking windows and someone else going around repairing them can be seen as good because it creates jobs. But that would be nonsensical and create poverty. That's why economists focus on wealth creation not job creation.
Fixing security bugs is only needed because people exploit them. It creates work but it doesn't create wealth.
Protecting yourself from bad actors also doesn't create wealth but it protects wealth you already have.
You know cybercrime and APT groups are not bored kids in their parents basement, there are groups that are running automated tooling to find vulnerable companies and if possible take all their wealth.
Just look at Binance of course, but if you are small company there are also people targeting those especially with AI assistance nowadays.
It does not. Scroll down in the given article, and you will find war as another example.
As GP said, it's a terminology issue. War spending, including defense spending, is not productive, because it does not produce wealth. It does not mean that protecting yourself from enemy countries or criminal is useless, or not worth it.
Nobody claims investing in security is a bad idea. It's just not productive, in the economic sense.
it is distinct from question whether it is productive to defend from evil people
evil people + protecting from them is obviously net negative!
but it is not changing that of evil people are present then reasonable level of protection from them is needed, and setting it up is productive
It becomes productive only once your business stops for days because somebody brought it to a halt by breaking into the servers and deleting them.
By the way, I think that even without safety concerns seat belts are productive because they anchor the driver more firmly, so we can brake harder and turn faster (a lot of roundabouts here) and get to our destination in a shorter time. Of course this is against general road safety.
(1) We live in nations of laws. The threat of state violence deters most thieves.
(2) You don't actually keep most of your valuables in your home. The stuff the thieves want is (mostly) actually stored somewhere else.
Security work absolutely is productive if it actually prevents theft. Securing your home is the wrong analogy, because it doesn't really prevent meaningful losses. Securing the bank? Different ballgame.
Human-written code isn’t inherently buggy, and LLM-written code isn’t inherently bug free. What is irresponsible is shipping code you don’t understand, and that will only increase with LLM-written code. A major reason why so much of the world’s software sucks is precisely that we think of it in terms of what’s “economical” and not what’s good or right. What we need is to slow the fuck down and stop treating every facet of life like a transaction. Software is filled to the brim with dude bros who “move fast and break things” and actively exploit people as their business model and somehow are called “the brightest minds of our generation”. Our priorities aren’t right.
I am not doing it anymore.
What makes you think that AI trained on slop is suddenly going to produce quality code?
But any kind of mistake, any kind of ambiguity and you'll be in for a wild ride. It usually takes a while for the rot to set in. I've had to revert - more than once - to many commits prior to undo a bunch of work that had me completely puzzled as to what the reason that particular fork was taken to begin with. Usually I end up figuring it out but definitely not always. In some cases this kind of wild modification fed back on itself until the situation was beyond rescue. I'm a bit quicker with such reverts now than in the past, for the first year or so I would try to save what could be saved, now I just restore and try again.
But if you're not sharp when evaluating what gets done there is a good chance that you'll miss it and then it can pop up way later.
What I've noticed as correlation (but I'm not sure if that is the root cause) is that non-local effects and stack effects are particularly hard as is everything that requires a working model of the world.
edit: this comment seems to make people uncomfortable.
Harness, prompt etc etc are all peripheral IMO, though people like to think their super long prompt, custom harness and MakeNoMistakes.md skills are what contributes to their special skill with AIs, but really models are just improving.
You can just trust the AI to generate a perfect answer I suppose (and why are humans slowing it down at all if so?), but if you don't expect that yet you have to put in the work.
I'm not sure who is studying this yet, but there has to be a statistical relationship between three factors of
1. intelligence of the human: the design patterns and architecture is already modelled in the mind, steering the AI becomes easy to know what to say next
2. quality of the codebase: the LLM can pick up the adjacent patterns and write code that looks and works cleanly.
3. quality of the prompt: the prompt keeps the AI focused and ignores the trivial side effects; keep the AI wandering away from critical path work; sometimes the AI can't make the decision what call site is next
In summation, what if claude is just a mirror pointing back at us? It certainly would explain wildly divergent opinions of opus 5.5
To me the order of importance would be:
1. The model - if you use a terrible model you'll get bad output
2. The codebase - this definitely forms the model for what claude outputs to some extent.
3. The human operator, what they accept and reject etc.
4. Things like prompts, skills, harnesses etc. The quality of the prompt.
So I think our positions are really just a different ordering of what is important rather than fundamentally opposed.
Re Claude being a mirror, if it is it's a particularly distorting mirror which often changes things in subtle ways and is determined to make people feel smart and addicted to its output - it has been trained to do so after all. I really don't think a mirror is the best analogy, because it's too independent and seductive to be a mirror.
There's a good example of the sort of thing they produce without good guidance as a dead comment on this thread as a reply to yours. Encouraging, positive and superficially plausible, but without any actual meaning or intention. The equivalent of that in code is what we need to be wary of.
And on the other side, I have seen AI produce exceptionally efficient, productive and effective code - the opposite of slop - when it is treated like a junior developer who will do any idiot thing, except when it is mentored by an actually experienced, professional developer.
The source of slop in AI code is not AI, its not the training data - its in the actual workflow of the human involved in its production.
If you just turn it loose on a codebase and ship it when it passes the tests you're asking for getting burned.
What's your workflow, out of interest?
Good one, that seems about right by my read of things.
Review is more important now than ever. My workflow has 3 review stages .. after analysis/specs/design, just prior to starting development (intake), and in the quality stages of testing and so on.
Well, there's also an after-release review, but the sales/marketing guys do that as part of their followups.