Bank CEOs, NSA Tell Congress That AI-Driven Patches Are Behind Outages
washingtonsun.com
washingtonsun.com
This is very ambiguous and click-baity. Reading just that title gives - at least me - the impression that AI patches are behind the outages. Read the content, and it's the patching of legacy systems in a rush after getting AI feedback about vulnerabilities, that caused the outages; 0 mention of how much AI was used in the actual patching. The former places blame squarely on the use of AI for patching, and the latter places it on the rushed fixes.
Also I'm finding it pretty wild that all these outages are suddenly occurring within this short a timespan. It's as if there was something time-based, like the Y2k bug, except nobody had a clue.
I think it's a safe bet they tokenmaxxed
I think it's pretty obvious why all these outages are occurring within this short a timespan. It's because the downside risk of a couple of days of outages of even major services is completely dwarfed by the downside risk of security Armageddon.
- We used to take 1 month per patch. Mostly manual process.
- With AI, we implemented the process of 2 patches per week. Same old process. Solid verification is missing. Rollback is manual.
Still won’t fully solve the problem but it would be a step in the right direction.
How is a system designed for a certain rate of patches?
"We're submitting more PRs than can be reasonably reviewed by people with the right SME/context."<br> "We're pushing more through testing that QA can handle."<br> "We're shipping more LoC than can reasonably be understood line-by-line by the supporting team."
I may just be projecting... but I see the push to ship more code at my company and those pushing are frustrated by the human processes slowing down delivery. Meanwhile, I am grateful for the bottlenecks for giving us time to figure out how development, delivery, and support works in this new world.
I've been working on a back end system that is only accessible from other internal systems, and the pressure to just accept the 10k line PR that should hopefully fix a retry counter bug, that should have been to move the counter outside a loop, because "AI said this was better".
It is either just a huge reformatting update, or the potential side effects will be massive. But no one has time to check that it is doing what is being requested when it is that large.
The risk equation is actually: risk of a global outage of services (fix them), vs. risk of unrecoverable security compromises of epic and unprecedented scale (don't fix them).
The latest Windows Update addressed 1,200 CVEs in existing Windows code, some large portion of which are in incredibly crusty code. And at least another 500 AI CVEs whose disposition is unknown. (Publicly quoted figures vary slightly).
People who are dealing with these problems should be loudly proclaimed as heroes, even if they do occasionally take down major services.
The other similar quote is this, "they’re pushing new technology on top of really old computer systems, and it just seems to overwhelm the system", which I suppose means that this is a legacy code base written before modern/robust design principles were popular, and on top of that, everyone who understands it is gone, so the AI patch is some spaghetti code that technically fixes the problem but creates 10 new ones and simultaneously sabotages performance.
Now you’re being way faster with AI, so you ship a patch every day or something. Now you have an outage every few months.
If the odds your release is buggy is a combination of code quality, tech debt, architecture, inadequate testing, CI & release processes, humans involved, management pressure, etc… But you’re not improving any of those factors, and maybe even making some factors worse (more pressure to ship more things with less human verification with AI), then your product’s stability gets worse if you ship faster.
So you could sort of say it’s designed for a certain release process and cadence
> Wall Street executives and the National Security Agency have told members of Congress that this week’s widespread tech outages were caused by attempts to fix vulnerabilities identified by artificial intelligence, according to three sources familiar with the private conversations.
aka, possibly rushed patches/patching, in response to AI-discovered vulns. As far as I can tell "AI-authored patches" is not really mentioned in this article. (Whether that's true or not, I won't speculate, but it's weird to see the several comments just jumping on "AI patching bad".)
One thing that is certain is that AIs are much better at finding bugs that former senior Microsoft talent.
I'm not sure how one would go about deciding whether AIs are better at fixing the problems they find than senior Microsoft talent was. But my intuition is that AI-assisted coding is dramatically better than code that unassisted senior Microsoft talent would write. Speaking as a non-Microsoft senior talent, I don't think I would bring a whole lot to the table if I were asked to fix bugs in code that I wrote in the 80s and 90s.
And I'm pretty sure that every senior Microsoft talent would tell you that if you want to fix the problem in months instead of decades, leveraging every advantage that AI can bring to bear on the problem is not an option.
The logical thing to do would be to slowly adapt processes to include AI and keep the senior talent around for their input on the matter as well as to supervise things that lower ranks have little to no comprehension or experience on.
Instead, like many companies recently, they kept their mountain of existing technical debt and strapped on a technical debt accelerator known as poorly applied AI.
The tool isn't the problem. The problem is how it's being used.
The vulnerability was identified by AI but it existed before. Not clear if the company used AI to write the patch. The causal role of AI is zero. They responded with bad code that caused a broken system. Their testing should have caught the regression. If it didn't, they need better tests.
This is a testing/QA failure, orthogonal to AI issues. I don't deny that the pace of AI-revealed vulnerabilities increases the pressure to rush out patches. But the response to vulnerabilities is entirely in the hands of the operators and test writers. Take some responsibility FFS.
It's not unreasonable to assume either way sadly. Some areas of code development still may outright prohibit the use of AI genrated code, if a developer listens or not before just asking chatGPT is another thing. Other areas are pushing their developers to write no code at all, only prompt the AI and not even try to fix it themselves if people are to be believed.
Only people who will get anything close to a real answer are the people who get to read a forensic level postmortum on these incidents.
I'm not particularly disagreeing with anything you have said; but it's hard to wrap one's mind around what's happening these days. The extent of it. The rate at which fixes must be provided. The existential risks if one does not fix things fast, and occasionally break things.
Big companies have been doing software releases for years, a release shouldnt take a company down.
Unless of course each and every one of these companies had a really bad vuln and they couldnt risk it and had to immediately stop business just in case.
Seems strange that they all managed too coordinate and decide to "act" at the same day at the same time and yet nobody else knows why.
Developers will turn into tech priests.
Don't know about the causal relationship but the correlation between major software outages and devs turning their brains off and beginning to use LLMs is interesting.
Incorrect use of spanning tree has never taken out a network segment
We've always had TLS cause we always put customer security first
Heartbleed never happened it was all an illusion
GitHub never been out for a day
API keys never ended up in a public repository
Databases always used encryption in transit and rest
Scala. That's all there is to say about that.
Apple keyboard has always been incredible
Windows Me
Every website is as light and nimble as HN minimizing unnecessary complexity
Back in the rose colored era when devs always had their brain on nothing like this ever happened.
...or maybe as time passes and only the now is in view the brain performs a kind of compression/garbage collection pass and cools off the neuron clusters related to memory of those events
Unfortunately devs get to take a lot of the blame for having their brain on less often than they should have making themselves a nuisance too often and jading people against them
Here; I can just engage in thought terminating snark too...
Account age. Self deprecating username.
Just a grumpy American conservative clinging to who they were yesterday. Ya already seen all there is to see so you know everyone else is wrong.
If you just seek validation and ego fluffing ...I mean that's exactly what LLMs do right?...
"What an insightful idea! You just cracked this wide open! Genius! You did it again!"
Really mixed signals.
Occurrences of cancer did not go up. Detection methods improved and more people got tested.
Just because you're not standing in Antarctica doesn't mean it doesn't exist.
Also my list was not just about bugs. Things technically worked without SSL and TLS. Socially not so much when everyones bank account numbers are encoded as plaintext on the open internet.
Enabling spanning tree protocol when you should not is not a bug. It's a bad decision by a human.
I feel like IT outages in the earlier days took longer to resolve, and the internet was slower and sites less reliable as a whole anyways.
I do think reliability and performance got better, and maybe there was a golden sweet spot. Certainly, GitHub is having the worst reliability in its history.
At this point, services at all these companies have likely gotten very complex compared to ten years ago. And are operating under higher scale. That’s gotta have an impact on things.
Just rattling off a list of face plants
And again not just calling out subtle bugs in code.
Prod keys in public Github is not a buffer overflow.
AI is making a lot of human mistakes because it trained on documented instances of humans making those mistakes.
Would not be shocked if some statistical analysis in a model is leading to conclude since humans allow themselves to wing it 96.788% of the time too YOLO[1]
[1] completely made up number; just making the point
The two gigs I had to use it, late 00s-early 10s we had strict syntax guidelines, limited the acceptable patterns, cause in the early days foot guns were so common while it matured they almost seemed like a feature of the language itself
That one was a bit tongue in cheek ;)
Not sure why your style preferences are relevant to the point. Not sure there is an obligation to avoid points others may not like.
If capitalism priced in externalities it would fall apart and as a result we seem to have fallen into a cognitive habit of ignoring our own existence in deference to dogma, refuse any sort of reflection that may ultimately help solve unsolved problems; we quit learning of we just "this is fine" our way through
Such a narcissistic, head in sand culture. Don't look up!
Because opinions are a dime a dozen, so if I'm going to actually read something and care, it's not going to be the arrogant/sarcastic one. There are tons of respectful HN comments about how the rate of outages is actually just news bias, which I kinda think is the explanation too.
You're talking past the other guy as well. The point wasn't that there have never been outages before AI.
Already pointed out not to expect responses that perfectly suit you. Yet you persist with vague rules I can't know as they only mean something relative to your internal parser.
I see deflection of all but your cherry picked emotional preference of "zero chance it was me" as that makes you feel best. That's fine; your way of expressing self but you have the same real evidence to support your position which is zero.
The flight outtage was from a contractor doing a router upgrade over day, instead of overnight. Has nothing to do with AI driven patches.
Microsoft is just too incompetent to run updates anymore. Driven by AI, yes, but failed by incompetence.