Elevated errors on claude.ai, Claude Code, and platform.claude.com
status.claude.com
status.claude.com
I have experienced, half a decade ago during some more frequent but less lengthy O365 errors that there are some higher ups who get when this is not something you can fix and some you very much can't. Wishing strength to any "but it's our mail" and "but we bought the Claude" meetings that happen today.
Imagine we had an AI system that could monitor 38 different patient signals (from blood pressure electrical activity in the brain) during a surgery and give the surgeon warnings whenever these drifted out of the expected range. It feels plausible to me that using such a system might improve surgery outcomes by, say, 20%.
I would prefer to have a surgeon who used the AI assistant rather than a surgeon who chose not to.
Claude is going to write a banger medical drama about this
Amusingly, I did the same and it told me there's no point in taking more than a single dose in my uncomplicated case (prophylactically).
it is a bit sad comparing that to "I can't check for bugs without an AI", though.
Some people have asked me, aren't you afraid you'll make a mistake if you fix your own car? Well, aren't they afraid their mechanic will make a mistake?
I think it's good to realize that AI (famously) also makes mistakes, and not to infantilize yourself to the level of not feeling "comfortable" trusting your own thinking.
Though, I do see younger people today whipping out their phone calculator to "double check" something like a 75% increase on 120.
Yes, of course. Why wouldn't I be?
Everybody makes mistakes, and of course tools to prevent that are good. But what you're saying makes no sense, have you never written code without using type checking or static analysis?
Sometimes good enough is good enough, it's silly to think that a word without bugs is even possible. And besides, using LLMs to get there is a fallacy in itself, because it's a fundamentally non-deterministic system.
Why are you still employed? What do you bring to the table if you’ve off loaded so much to AI that you can no longer review a pull request…?
What more needs to happen before you accept that human review does not catch all the bugs that AI does?
For now. Wait until we start hearing things like "Mythos reverse-engineered Apple ARM architecture and implemented the whole roadmap of the Asahi Project".
IME if anything, people are relying on LLMs for ops more than for development! Devops convinced management that ops isn't actually all that valuable and that devs can just do it. IaC is usually a mess of boring YAML based syntax that no one in their right mind want to learn anyway. Learning Ansible doesn't feel like a transferable skill unlike learning a programming language or paradigm, and in most teams the quality of the IaC stuff (abstraction, code reuse etc.) never mattered much.
As soon as LLMs became good, generating IaC stuff was like the first thing to go full vibe coding, at least for me. I'm proud of my craft, I don't produce bad code with my name on it, I very carefully review all the code changes etc. Devops YAML stuff? Couldn't care less.
My soul just dry-heaved from laughter. Nice.
O364...
But I find it difficult to motivate it from the "productivity" angle. Like if I work for a day without an LLM today, it feels like I could go skiing instead, and the LLM will do what I would have done today in 10min tomorrow...
Im not complaining Im just pointing out how quickly the overton window is shifting.
Obviously that's not the case for everybody. 3 years ago, LLMs were very far from producing the quality of code I wanted in my domain.
It was obvious in 2023 and it is obvious now yet we still have people in denial even today. These people need AI to take over the world and threatening them with eternal suffering before they will change their minds even a bit.
Things are still found but normally it is a new test case or an edge condition we don’t have an automated test for. The ever growing e2e suite has reduced the amount of regressions pretty drastically.
EDIT: I forgot to note that automated e2e tests were always the answer to improving quality. AI has just made it cheap to have several layers of test coverage.
Seriously, why do people assume that what they feel should be how everyone else should be feeling?
Like: I'm normal, and I do X. Therefore other normal people will also do X. It's the basis for in-group/out-group. We're all normal and they are not. (for some definition of normal).
If I wait until tomorrow, though, my company won't see the difference.
All the work most people are doing it just going to be Ctrl+A deleted and replaced with someone else's LLM slop using Opus 8 in an afternoon.
All the shit people are building right now just feel ephemeral, like its going to float away on a breeze.
I don't know any software engineer who can say "if I was not doing what I do, society would get a little worse".
In the worst case.. sometimes you learn what's best by experiencing what's not.
This seems wildly inaccurate.
You sound self-employed.
Ironically, I also had a change window last night to make some configuration changes to improve site reliability. It was a little challenging to get customer approval for both the cost & downtime for this. Let's just say... this will be a much more fun conversation to have next week, and for the account manager to have. "Yes, we'e proactively improved your reliablity, and here's why you were very smart to approve it and pay for it."
Our house has three carpeted steps between levels. Is there a Roomba that can clean those steps whilst cleaning the rooms on each of the levels? Or do I need one per step and the levels?
a) I’m not lazy enough to buy a Roomba. If I was, I definitely would be too lazy to nanny it around the house, lifting it up the three steps. b) When I am feeling lazy I hire a cleaner, who uses their own hand vacuum cleaner.
Time will tell.
At our office we have a cleaning crew and when they come by to vacuum during office hours, everyone in our section takes a 5 minute coffee break and then it's done when we get back.
We can of course do that privately, for just the heck of it, but it's not the right response for anything at societal level. We simply need to work harder for more of these 9s
I realise people on the monthly plans can't do it as easily. But I would say that as far as possible, having a plan B is the foundation of resilience.
Another reason to have good Evals…
That doesn't work for all systems, and not for all businesses, but I would say the mass majority can 'suffer' 24 hours without email or an LLM without everything going up in flames.
That's not always easy to do, but with something like inference providers it is, so why not kill two birds with one stone?
Well, I just finished a meeting where we diagrammed out how change authority is to be structured as we split one group into two, which teams will retain what authorities. Zero claude involvement. Ten people in a room. Whiteboard and markers with a "scribe" taking notes. I am now editing old SOPs to reflect the decisions made in the meeting ... not something we would trust to claude or any other AI.
Let me know when the tech industry workers stop doing this and gains some solidarity, and I'll join you in that sentiment.
Until then, I pretty much need to bolster myself on the idea that one day I will need to switch frameworks, IDE's, and even OS's themselves. I'm the ones guiding the tools, not vice versa.
>We simply need to work harder for more of these 9s
The only 100% thing I will guarantee is that these tools will align to the incentives of their leadership over time, not the customers using it. add 10 9's if you want, that just guarantees that humanity is the weakest link.
I'm not sure if this is your intent, but this paragraph reads like satire given current real world events.
The systems you mention in your second sentence are failing at the "right response" you mention in your third, which dictates that contrary to your first sentence alternative action is necessary.
You cannot harden systems you have no control over. Societal level assurances require law and goverment regulation and enforcement. It is juvenile to pretend otherwise. If we were talking about government services, then as a taxpayer you would have some avenue to enact change.
Your only option when dealing with private companies is to try and become anti-fragile as much as possible at a personal level or at the level of your own company.
> Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.
You are also a system. And, if you, as a system, don't run with Claude, you're not sufficiently hardened. People have made all sorts of helpful suggestions, like de-risking with multiple providers or just getting yourself a better Mac to run a local model.
The idea that: "No, no, we must have Claude, b/c it's the same as having a pen and paper!" is woefully naive. It's hard to attack the supply chain of pen and paper if you have a few extra reams and a few extra sticks. It's pretty easy for our geopolitical enemies to attack computers located far away from you, especially those systems are designed like fragile Rube Goldberg machines; furthermore, if we build an entire society (a system) to be productive only when we have LLMs, our issue isn't "insufficient-hardening". It's that the design is wrong.
Or maybe more prescient; machine operators going home on a cold day because they can’t get their loader started.
I’m pretty sure doctors don’t abandon their patients during a blackout either, or even a war for that matter.
Mature professions have procedures and backup procedures for pretty much every non-catastrophic situation.
I'm not just talking about tools either. You mentioned redundancies and contingency plans. Layoffs have stripped workforces to barebones headcounts. How can you have redundancy when one sick man-day reduces your eng'g team by 50%?
> Which begs the question is Software Eng'g/Development a mature profession?
For every sense of the word pair "less mature," including, but not limited to, "less responsible for consequences of actions."
Of course proof by analogy is silly anyways, my email going down is not nearly as bad as a patient's medical equipment going down.
And even that only does so much, it was 45 years ago but my Grandmother had a stroke on the operating table because the power went out and generators didn't kick on until someone could run and manually start them because the autostart failed.
The equivalent for a dev is laptop, another AI, mobile hotspot and work at a cafe/library.
Microsoft being down is like having a contingency for a tornado.
Sounds like a normal thing to do if you live in Kansas. And perhaps we are all about to start living in the software equivalent of Kansas?
I'm probably misreading your point, so, apologies, but I'd still just like to say:
Microsoft is a profit-driven, human-run corporation that mostly offers business infrastructure services. I think contingency planning for Microsoft service failures is almost entirely unlike contingency planning for adverse weather events?
From what I hear, the project managers are having a hard time sending emails anyway. We'll call it quits, winkwink.
Likewise, Anthropic / Dario has been busy proclaiming why nobody except themselves should be allowed to develop frontier AI, that we need "kill switches" in all AI models, and that we need a pause / slowdown on AI development because of the threat of open source models.
Essentially, they want complete lock in to their platform, so yes, it's very appropriate to ask questions when they can't meet basic 3 9's of reliability.
I was unable to do things beyond basic banking with my bank yesterday (and today) due to this outage. It just involves making changes to a merchant processing account for a nonprofit, but the point is that normal business activity grinds to a halt when it relies on an unreliable provider like this.
yeah, couldn't be me
That's a scary thought.
I recently asked Claude to edit a few targets of a Makefile. It decided: "OH BOY HERE WE GO LETS FIRE UP ALL THE B200S BABY" and went way deep adding tests...I stopped it when I saw something like: "Adding guards to prevent homebrew injection threat...". It had written an entire fucking build subsystem, probably larger than make. For a 5-line change.
1. We haven't learned anything from Google, who maintained Chromium and Android before LLMs
2. Your LLM is not generating code as maintainable as Chromium and Android (which are gigantic open-source projects that probably 99% of your devs have used)
3. Your company has no faith or interest in teaching devs how to maintain code nor how to build maintainable code
And that's pretty pathetic. It's solvable, but it's pathetic.
I will be amused if those 3 points lead to this conclusion:
4. If you aren't asking your LLMs to write maintainable code, and they aren't smart enough to do it by default, you are pretty quickly gonna hit a brick wall where your LLMs are also just slopping mud onto the big ball of mud. Then nothing will save you, if your leads haven't heard of "quality".
(Yes, this is projection. Yes, I'm late for work)
There are companies telling their developers that they’ll never write code again. They manage agents now. It’s like asking PMs how maintainable the codebase is.
Most people people here do not code. They wish they do, but they don't.
People can and do still write code. Especially in regulated industries. The internet is just full con artists and hype followers not thinking about the future at all.
but I can say that, anyone experienced will notice how you really can't just "let them loose" on the codebase because they'll lose context, not understand the architecture, not have good vision/ideas about the codebase etc.
so I'm bottlenecked by following the code generation and then having to review and respond
and there's so many times where I go "wait but can't we just do x?" and then the model goes, "oh let me see.. yes we can"
if you were to just use agents w/o looking at code you'd have complete spaghetti in no time
maybe the future is that you embrace the spaghetti and just treat code as a black box where all you care about is the input/output
but my intuition tells me that it's not really viable yet. you'll take many steps forward and many steps back.
Is going back to just a couple of months ago. Jesus.
But honestly, the practice of programming then was not dramatically different from programming in, say, the mid-2010s. Obviously we leaned more heavily on the official documentation, and tbf, a lot of the docs were provided in PDFs, which is probably the single worst way to provide documentation.
Sometimes, the docs were provided as collections of papers bound together by a "spine". This is actually a wonderful documentation format because our brains can associate physical locations in the physical media with specific pieces of information.
(this is the oldest I have ever been in my entire life)
Tractor breaks, you complain about shoveling again and someone says you’re in the wrong job.
It's asking a tractor why the ground is full of dirt, despite having tractors to move the dirt, and tractors to confirm there is absolutely no dirt left before all the tractors can rest.
The tractor informs me that my tractors don't have permission to clean the dirt, nor see whether there is dirt, nor even distinguish that from "no permission to see dirt".
It found that faulty ACL was a known issue, and is a high priority bug for a team I've never heard of, and thousands of tractors are waiting for this.
One tractor found a webcam that overlooks the dirt patch, which bypasses the ACL, but that is now overloaded. Another tractor has setup multicast streaming to broadcast that camera, with low latency, hardware acceleration, and no single point of failure.
We're back in business.
Bypassing ACLs, multicast streaming, and using image recognition APIs for coding usecases are all against company policy, but nobody wants to pull the plug on any of this.
"Tractor, please add the multicast address to my AGENTS.md, and check my tractors have resumed."
"I've updated your AGENTS.md. The tractors have already cleaned the dirt. Here are images from the camera over the past 5 minutes, correlated with logs of the reporter tractors that reviewed progressed, to demonstrate the reports are now accurate too. I found an integer division bug, now fixed. Shall I schedule a tractor to oversee this every hour, by checking the camera?"
"Yes. And raise a blocking bug to fix all this up when the high priority bug is resolved."
8 minutes have elapsed.
Some examples include requiring that someone addresses every review comment separately, or that a PR description be formatted a specific way, or that support requests have existing JIRA numbers etc. All of these make the developer experience much worse and would never fly if generating text (that no one reads) wasn't cheap now. A lot of this is cargo culting to make vibe coding feel more serious and entreprise.
The teams that have gone full specs driven development are not AFAICT more productive but good luck getting any change in without 10$ of API usage, even for a one liner. In some cases development is now slower with LLM than it was before!
- JIRA ticket - audit trail
- PR format - easier for people to scan, and to make sure things aren't forgotten (you can even have the template inserted when it's created)
- address review comments separately - not sure what to say on this one because I don't understand how _else_ you would address them - I've always addressed each one individually
The team I'm on prides itself in providing great service to our clients (other devs in the company). This includes responding to support requests even if the client isn't entirely sure how to ask or who to ask.
> not sure what to say on this one because I don't understand how _else_ you would address them - I've always addressed each one individually
What I mean is that it used to be enough for you to address PR tasks by changing the code and pushing the updates and marking the tasks as "resolved". Now in some projects at work you also need to reply to the comment explaining how the change you made addressed the task. The idea obviously is to stop LLM from just ticking tasks without actually doing the work, but it's onerous and based on distrust. Not trusting LLMs is warranted but those rules get applied to everyone.
So in that case, you would create the ticket on their behalf before getting to work, right? Because how would you track what you have been working on, if something has happened already in the past, if there is a pattern in support requests, or how well your team reacts to support requests?
The team I just left had a PR template that was probably 60 bullet points (never bothered to count, just grumbled). Certainly was not easy either to scan or to update manually.
Those examples sound like responses to being swamped with LLM PRs:
* "every review comment separately" [so the bot doesn't get a chance to miss it]
* "a PR description be formatted a specific way" [so the bot has guardrails]
* "support requests have existing JIRA numbers etc" [so we can have a chance of understanding the context]
> The teams that have gone full specs driven development are not AFAICT more productive
I suppose time will tell. It's difficult to generalise my experiences beyond my immediate team/company/industry.
> In some cases development is now slower with LLM than it was before!
Yes, pre-LLM I wouldn't sign up for formatted PR descriptions!
I suppose the equivalent would be Google search being down 10 years ago and being forced to read man pages again. Annoying but not literally "snow day" levels of lost productivity.
I never did find out how they see code, but I have to imagine the right verb for them was "deciphering" instead of "reading"...
But I'm sure in a few years AI will be so good we won't have to worry about any of that anymore. After all, it's not like those systems learn how to code by reading our code, is it?
I’ve been in this situation before, an external API is basically down and my managers solution is to blame me.
Didn’t last there too long.
But at the same time was surprised I could still google and resolve things myself when I started thinking again lmao.
On a normal environment the old Graybeards would fix it, after grumping on being taken away from playing with their latest Emacs config.
But this is the Valley, and we dont do experience only VibeOps. And half reading this post, are wondering if Emacs is a new frontend framework they never heard about....
If people lose enough money then they will come up with contingency plans etc. that may even work in the next outage! Absolutely no one will be held responsible though. The kind of management folks that made those decisions will just turn around and work on those contingency plans, and everyone (including themselves) will believe that they had been doing so all along.
At least that's my experience in big corporations.
You get to pass the buck and management can’t be mad at you (sort of)
You could ask that question about the MS outage as well and million other things.
I suppose I can't understand because where I work our scale is small enough that we can have an actually resilient infrastructure and can stay in control of almost everything we need, and when you get bigger it's more difficult? I mean, even an outage from our email provider could be worked around in a matter of a couple hours at most.
I'm really not sure it's not just because most managers are clueless and shortsighted though.
It's a culture thing, if it leans "engineering-heavy", you usually have contingencies, tests restoring backups and lots of other "goodies" that gets skipped if you don't lean that way. Microsoft and similar organizations been a sales-and-enterprise leaning organization for most of its life, so of course they're not gonna think about what happens when their "always online cloud tooling" goes offline when setting things up, it's much more important to finish things quickly and have a long list of features to be able to present to customers.
I get it, we had to sign separate HIPAA BAAs with Anthropic and OpenAI - but we did sign them both, if only to be able to have access to a wider array of models.
Their moat isn't large.
It's just one of those things, you'd think the economy or country would collapse effectively but things just tick on by they way they always did. Imagine the banks in the US closing for 6 months?
https://en.wikipedia.org/wiki/Irish_bank_strikes_(1966%E2%80...
OAuth request failed Temporarily unable to authenticate. Please retry.
Can this be fall-out from the Microsoft outage?
I don't have that email account on my phone. So to log in, I have to open my laptop, get the 'magic' link, find a way to send it to myself on my phone that doesn't make a request (e.g. for an unfurl) expiring it, and then open it on my phone to log in. So magic.
So basically you start login on phone, keep page open. Open laptop, click on link on laptop and you'll get a code. You type that code on your phone and that's all.
Solving problems that don't exist, and creating problems that don't need to.
They just outsource the security into your email provider.
94 points by cloudbreaker 22 hours from now | flag | hide | 64 comments
Copilot is down too, so it's probably Microsoft engineers trying to vibecode their replacement Azure infrastructure before the current garbage fire finishes burning out.
Reminds me of when we built Facebook apps and were at the whim of the platform.
Now we need to handle this stuff on status pages because this is not clear to the users.
Are people who don't use M365 experiencing issues here?
It seems resolved now.
Solid state NAND has 100x the latency of RAM but in parallel it's just as capable of the bandwidth to feed a gaming GPU as RAM since bandwidth scales. Much much cheaper per byte too. The latency isn't a blocker in inference because the weights needing to be processed are predictable. You can actually work around latency, and in general that only affects time to first token not total tok/s, which on local inference is much faster no matter what you do compared to the internet. For local AI all you need is storage/memory bandwidth and that's actually perfectly scalable for any storage mechanism.
The bottleneck today to running Kimi 3.1pro at home is storing the weights in a way that you can get them into and out of the GPU for processing fast enough. A good gaming GPU can process the ~100billion active weights fast enough for a reasonable tok/s for a single user but we cannot get the weights from either system RAM or high bandwidth storage into and out of the GPU fast enough atm and we can't keep the weights in GPU RAM because that would cost too much. The way it's done in datacenters is just to have many GPUs working together and all the weights always in GPU RAM across a cluster of GPUs.
The datacenter approach of all weights in RAM always is great for serving 1000's of people at once, each with 100's of tokens per second but for home use it's a waste. You'd never individually maximally use a cluster of GB200's.
So for home use what we really want is a lot of high bandwidth storage of any kind (it can seriously be solid state NAND, just needs a 10x wider interface for bandwidth) hooked up to a moderate amount of compute. That'd get us Kimi on a single card.
I'm going to assume someone will make it at some point in the near future. At that point we're free!
I guess anecdotal but people nowadays are incredibly addicted to LLMs, to the point where they cant or dont want to work when there's an outage
Time to warm up the GPUs I guess, since I can’t be fucked to do it myself lol. Thank gods for qwen flash next and DeepSeek!
TBH the only reason I’m not working mainly with flash next anyway is that I can only run inference for one agent at a time on my setup.
3 minutes later... I can reconnect, and the "crashed" instance woke up.
That seems faster than usual, compare to Outlook (down for a day) and GitHub. It's still embarrassing for a heavily-relied service, but better, and I wonder if Claude (or another LLM) helped diagnose and fix the outage?
This allows their architecture to be simpler, and scale more easily.
GitHub meanwhile is highly relational, across repos and users, and provides real time updates over all this. Outlook is similar.
https://status.claude.com/incidents/nn11wk485zfn to be clear
I was using Claude Code Friday morning, and I wanted to know what the issue was and how I might have been effected
I do not know, after reading their notice
All the more reason to hurry up our move away from Anthropic.
Claude Code is good. But if when it has problems it issues notices like this, being good is not enough
If they expect to be as valuable as they seem to feel entitled to, they must do better
What a shame
And jarring. When interacting with colleagues in teams that I haven't worked with, there's an uncomfortable "where are they on AI...", which will decide whether I have to dial down my usage, or whether we can both let it rip.
In mere months.
After learning to use a calculator, did it promptly become laughable to do arithmetic by hand (or even in your head) if yours broke or otherwise became unavailable?
currently running a Claude model through github copilot CLI. Pretty odd given the two ongoing outages.
Even if it vaguely is, why do people think subjective anecdotal thoughts about which model is better are a good discussion topic. If there’s an objective evaluation idea that would be useful.
I run evals for a very specific domain. Opus 5.5 is still the industry leader (we can’t use Fable class yet).
It's relevant because there's only so many "yep it's down" comments that are interesting. They have a technology problem which isn't fixed. They're about to go IPO. Their marketing is clearly still working, but subjective experience with both contradicts the marketing.
> I run evals for a very specific domain.
What do your evals test for?
you'd rather have a subject that does exactly as told, or that does what's right?
Office 364.3789278392647
(estimated)
Well, it was my employers wish that we devs incorporate more AI, and we did. It's their misfortune now.
Depending on how widely this was deployed, perhaps that's affecting usage too...
(I know CC =/= AI, but put that aside for the sake of the joke)
Was pretty nice though, less stress.
try this one /s
well, it works on my machine.
Jim Cramer has $MSFT as "DON'T BUY"
And now we have Claude and MS365 down.
6 Hours later after this comment [0] Claude also went down. It's now up to the human engineers (if there are any) to fix this outage.