Apple Services Down
developer.apple.com
developer.apple.com
Apple was fine, but this poor old woman was at the genius bar trying to get some accounts connected. The genius at the Apple store was trying to explain the outage - and it just was NOT clicking for the woman. He must have said 4 variations of "Yes your accounts SHOULD connect, but Microsoft is having a major outage right now so they do not."
TSA is down too.
Do you really think that when these events happen, every single computer is patched at the SAME time ?
Does that sound intelligent ?
Does this happen in other industries ? Does the electric company shutdown every single transformer/station/big machine i dont know the name and perform servicing at the SAME time ?
Of course not, if they do they shouldnt.
That's not the entirely of their cloud exposure, however. A ton of their stuff runs in AWS, I wouldn't be surprised if they had things on other cloud providers as well.
Apple has literally no idea what they are doing with Linux and cloud.
Bricked from the very beginning.
The only chance we have for a more reliable world is if these outages and hacks become more and more common. Institutions and people will be forced to wake up and find more local and diverse alternatives...
"Putting everything on a server (and asking the world to rewrite all of its software along the way) may seem attractive if you build servers, but it ignores the real concerns and needs of many customers."
These were quotes from Paul Maritz (a VP at Microsoft) made about 27 years ago to the WSJ[1] in a back-and-forth discussion with a Sun executive (Pat Sueltz). Some of the comments were quite prescient, others aren't.
This is true. However:
1. We now have tens of millions, maybe even hundreds of millions, of users who have little experience with traditional desktop computing. They grew up on Web services that they access primarily using smartphones, Chromebooks, or web browsers on desktop computers. There are many people with little-to-no experience with desktop applications like Microsoft Office and the Adobe Creative Suite, but they do their work on Web applications such as Office 365 and Google's suite.
2. Even for traditional desktop computer users, they often spend considerable amounts of money updating their environments once every few years. Windows has excellent backwards compatibility, but most people aren't interested in running Office 97 and Photoshop 5.5 on Windows 11 even if those programs can run on Windows 11. I like Office 97, but I need to deal with modern Office documents, plus I want the improved multi-language support and extra features that have been added in newer versions of Office. The Mac is much more aggressive about cutting off support for old programs; my PowerPC Mac OS X software can't run on my Apple Silicon Macs, and even support for Intel Mac programs will be dropped soon.
Because so many computer users these days didn't start off with traditional desktops, and also because of upgrade churn, this is how we ended up with the current situation, where many people rely on centralized solutions.
However they'll blatantly spew out spam to the rest of the internet, including through allowing ongoing abuse of their Google Calendar invites, Google Drive and Keep share notifications.
Price is vaguely related to efficiency, so that's what the private sector optimizes for. Think about that next time people complain about Government efficiency.
As one NASA chief said: "if we could blow up rockets as much as SpaceX did, yeah we'd be much faster and cheaper"
If the NASA blows up some rockets for fast development, people get angry, politicians get scared, funding gets cut, important leaders get fired. So they have to try really hard to guarantee no failures. Everything takes way longer and costs way more. They still fail sometimes, and just as often a project is cancelled before completion, but that's way better than an explosion, politically.
For most people in the country, despite being online constantly, they would have never known.
If anything it shows that many aspects of the Internet ARE decentralized well, even within one company. I can use a device with apps and accounts that aren’t dependent on the device manufacturer’s uptime.
YouTube videos—download.
Music—all stuff I ripped from my own CD collections. Newer digital purchases (Bandcamp) are backed up with everything else.
Magazines from archive.org and worldradiohistory.com—local.
Etc.
AWS et al. (yes, even Microsoft) are far more reliable than the smaller players, let alone your own rig. The flip-side is that when we go down, we all go down together, and I'm fine with that.
I'll take my own rig 1000 times out of 1000 for reliability. This would be true even if AWS, Microsoft and the rest weren't parasites looking to constantly spy on me.
How are you defining it?
Early Apple history is often spiced up with tales of Woz's exploits using blue boxes, exploiting an inherently insecure centralized communications network.
We Millennials mock our parents yet mimic their behavior. Memetic hyper-parasitism.
The only thing I saw was memes and news events, 0 direct impact.
A lot of the people responsible for it all going so wrong are present on this very forum. We made good money doing it, but in return what could've been one of the most positive liberating technologies in history ended up another lever of centralization and control instead.
I regret my part in it. I wonder how many feel the same. Clearly, a lot of people are still happy to push on. There's IPOs to be done and stock to vest, after all!
I remember that every time there was a choice between a positive choice and the easy choice, the average person always chose the easy one. There were plenty of alternatives to Facebook yet billions of users went to Facebook. The whole Snowden thing probably registered less acutely than the final season of Game of Thrones with the voting public.
Yes, people like us built the both good and bad platforms, but ultimately the decision which one to use was on the users, and they chose the world we have now.
Basically, I see recent history as Eternal September and the Internet turning into, through sheer volume of demand, into Super Cable TV.
It's difficult for me to understand your view so hoping you can expand on it.
I agree with this statement, but not with your conclusions from it. It is precisely because users don't care and choose the easy way out that we have a greater responsibility. Why did we fail to make the good future easy and appealing? Why did we build the bad option at all? The only honest answer I can come up with is selfishness and greed. The bad option didn't end up being easy by accident. It ended up easy because people like you and I were given mountains of money to optimize extraction and addiction.
> Yes, people like us built the both good and bad platforms, but ultimately the decision which one to use was on the users, and they chose the world we have now.
As an engineer, you are responsible for what you build and its impact on the world. If you build a very convenient, but unsafe bridge and it collapses with people on it, you don't get to just say "oh but the people chose to go on it!". We're the experts. We're the ones that are supposed to know and make good decisions, that's what we were entrusted with and that's what we failed to do.
I agree that most average people do things that lead to bad places, but I don't think it absolves us from responsibility.
For most self hosted services, can probably get away with "set it and forget it attitude". But host a couple of publicly available services (immich to share photos with family not in your vpn for example), and asking that of the common person is a bit much.
This would also not be "free" which is pretty much the main selling point of these centralized services. Of course they use your data to create ad profiles, sell it to data brokers, and/or incorporate into data set to train LLMs.
Decentralized is possible, but it’s going up against forces that collectively have trillions of dollars at their disposal (albeit most is probably spent on stock buybacks and exec comp)
We don't need everyone to be a gardener. We just need non-chain groceries and restaurants. Folks are then free to garden and cook if they like that idea, shop and cook for a very similar cost/effort expenditure, or pay more and do none of it -- all without centralization. Well, the energy to cook is centralized in any case, other than home solar... you need to trust somebody.
I'm honestly not sure how that's different from the modern internet?
"If it goes wrong, they fix it".
https://developer.apple.com/system-status/
https://www.apple.com/support/systemstatus/
Interesting that HomeKit makes the general status page and not the developer one. Makes me wonder how they define each entry, and what makes up a system in one vs. the other.
A normal user might expect to access their home devices from outside the house (which is what I presume "HomeKit" covers). But only a "publisher" (for want of a better term) would need to access App Store Connect, or News Publisher or similar.
If it was AWS fine, that’s not uncommon, but there have been several outages, including at the nsa, which is also unusual.
Outages on classified networks tend to not directly impact the public.
In Polish folklore there is apparently a belief that a person must be buried before Sunday (pochówek przed niedzielą?) or 'soon enough', as otherwise it will cause the deceased to pull another family member (or neighbour) into the grave with them. Perhaps the FAA and/or Microsoft outages were not fixed quickly enough?
and screwing up the patches
oh wait, it's EXACTLY that
we just went through PHP having a massive hole that was then found similar in every other language and library
It’s undoubtedly a good perspective to be aware of and understand.
At the same time, we should also assume if we don’t have information to prove things aren’t correlated or reasonable evidence to think they are unrelated, we shouldn’t just dismiss them. Sometimes, correlation exists because of an underlying casual relation. It’s not always, there’s not even some frequency I could reasonably assign. So we shouldn’t dismiss it either.
I keep seeing the two polarized perspectives with few considering that it may be a mixture. Some of these events really could be related. Some may actually be pure coincidence. Some may be simply Baader–Meinhof (frequency illusion) related (we see something, therefore we look for it more, therefore we see more of it, so on) as a survival instinct.
I don’t know if there’s clarity what the current mixture is. I can read details about several of the incidents reported causes and say clearly they’re unrelated but I’m not entirely sure I trust some of the reports as it’s not in their interest to say something malicious did cause these problems.
Kids were posting videos of their drones on instagram and the nightly news was buying the rights...
Legitimate news organizations don't buy video from the general public. TMZ, yes. New York Times, no.
They only buy video from organizations that have been vetted and with which they have a previous relationship, like a known freelancer. They don't write checks to random kids.
>> A US congressman Jeff Van Drew claimed he had heard from "high sources" that the flying objects were tied to Iran. [...] the New Jersey Republican said the drones were coming from an Iranian "mothership" in the Atlantic.
There are congressmen who believe space lasers change the weather, and a president who thinks COVID can be cured by injecting bleach into his arm. Being elected doesn't make you an expert in anything.
It's used to be amusing to see how much the British press misunderstands American politics and politicians, but more and more it's annoying.
The original poster is claiming this is a news cycle perversely creating the (perception of a) phenomenon itself, instead of the more logically-straightforward but incorrect assumption that this is a phenomenon being picked up as a news cycle.
The winner of this battle of Internet commentary will of course be the one who brings statistics and outliers to the fight. Gentlemen/women/folk: stand back to back, take ten paces and then let fly your volley of anecdotes, data, or actual reasoning, if you please.
In fact, I find the sudden rush to explain away such events as more suspicious than the actual events themselves.
That said outlook being down for 24 hours, all flights being grounded, and apple not being able to do ANY payments would all make headlines on any given day. This isn't just "now that the populace is focused on it we'll report anything remotely related".
That said it does just seem to be a mix of bad practices catching up with bad decisions being made (hey lets update on friday..), but even I have to agree the frequency and duration is starting to look odd.
That said as with all these things, I sincerely doubt the ability of anyone to keep this shit under wraps. Only the US and its companies are being targeted AND no one from those companies screamed bloody murder about coordinated attacks/rogue AI before the gov stepped in and said "shut the fuck up."?
Maybe, but I doubt it? I know people are at risk of losing their jobs, but word gets around and I think there'd be a hell of a lot more discussion about possibilities by people who actually know what the fuck they're talking about, rather than talking heads and random nobodies who clearly don't know the industry (as often happens when any industry gets signal boosted by the media).
Hanlon's razor/Gellman amnesia still seem like the strongest answers mixed with maybe some "yeah so MS shitting the bed helped kick off a chain of tech dominoes"
Relying on code generation has pretty much been proven to atrophy skills in the short term and relying on opaque internet-connected black boxes for code generation is both leaking information and inviting back doors, should that black box be adversarial (whether due to the survival incentives of an intelligence or more likely due to the intentional biasing by the ones in control of the box).
NSA
Chase and banks
Airlines
Yeah, only one thing connects them: all being unprecedented and in the same week. Nothing to see here folks!
Maybe this hundred-year storm happened because the gods are angry with you. Or maybe it happened because a hundred years* have passed. You'll need some evidence for that first one.
* on average
Who is to say what said company might have done with anything it learned in the process?
These interruptions are potentially a combination of companies racing to patch faster to keep up, or a large exploit has been found, or importantly AI exploit tools like ARTEX are being used in the wild.
I have nothing else to go on
Apple Account Verification
Enter your Apple Account password in Settings.
Imagine you’re an attacker and you have a way to capture credentials at login. You’d then trigger logouts to make people log in again.Enable Lockdown Mode > put your phone in airplane mode > go to cellular > turn off Track Data Usage > press Reset Usage > go to on WiFi and turn it on > Scroll down to Connectivity Assist > Reset Data Usage > still in WiFi go to Edit > Scroll down to managed networks, select them and disable Connectivity Assist for any connections.
On my iPhone, from that point on, even after disabling lockdown mode, anytime I go to Safari with airplane mode on I get a giant “This Connection is Not Secure” UI. Apple hasn’t been able or willing to help since bringing this to their attention in September.
Good to be at the arse end of the world, nobody cares about us here. Im going to guess in the future this distance will be an even bigger asset.
I started getting notifications that my Apple ID was logged out of iCloud, and I am not suffering for it, since I am not using iCloud for much.
On another note, using non-iCloud accounts with those apps often results in a “degraded" experience. Many of the features offered with iCloud accounts on those apps are missing when connected to a different service.
Them missing features you refer to are not some kind of anticompetetive move, it's that protocols don't advance to support these things.
AI race redirected resources and attention away from operational concerns. There’s also seasonal increase in traffic in many industries, which may put extra pressure to deploy updates faster.
Edit: and apparently Apple takes appointments but has no methodology to cancel them due to catastrophic outages like this, nor do they have any ability to conduct non-digital transactions…typical tech
I can't imagine it's gotten any better.
The scale of the data center build outs are not technically necessary. It's all designed to make big tech too big to fail, scare aging pols deeply invested in big tech stocks; the real infinite money glitch...scare the bejeezus out of the aging political class and they'll bankrupt the world for you
The only SaaS app I really need is a Dropbox for offsite backup.
If shit goes to shit where we are people can ping remote contacts direct to fan out the news; proper point to point topology.
Instead we are forced into a hub and spoke topology and we are not the hubs
All other backend infrastructure could be whittled down to fiber and wireless network gear where upon inspection all ya can see is electrons or scrambled packets.
It's particularly egregious in cases like this. For example, there were so many "scary virus" stories post-covid that never would have gotten press.
Meanwhile, Apple's status pages list Apple Pay as green: https://www.apple.com/support/systemstatus/
https://old.reddit.com/r/iOSProgramming/comments/1xypp96/app...
not just software is getting enshittified, people. they are coming after your Snickers, and your Hersheys, and your Reese's
Honestly after a childhood of cheap knock-off chocolate, the real stuff tastes weird, and the cheap stuff remains a comfort food.
I think such things are happening more and more year after year, which is newsworthy in and of itself. I'm not denying that. I'm not even denying that AI/cyber incidents on a big scale haven't happened and aren't coming. This is all newsworthy. I just have doubts about how people are interpreting the events of this particular week.
The number of vulernable systems far outweighs all of the patching being done by the defenders. Exponentially increasing hacks and unexplained outages begin occurring in many systems.
AI patching turns out to not only cause the initial downtime from the accrued technical debt, in many situations the rush to put out a patch for a known exploit itself causes more outages because AI is not advanced enough to have enough context for every system it is trying to fix.
But so many of these systems can't be taken offline because the modern world depends on them. The only solution is going to be to destroy the open internet, strict firewalls between countries and regions, and excessive internal surveillance to track down any domestic threats.
The open/global internet will have to get locked down.
Following a string of high-profile online service outages this past week, several Apple services appear to be experiencing issues today.
https://tech.yahoo.com/general/breaking-news/article/apple-c...
When I heard the install work was done by L3 Harris, a company my department has worked with, my reaction went from "Damn, is AI sabotaging us?" to "Oh, that makes sense".
L3 has been on the rocks for the last few years.
Anyone claiming it's a coincident or that it's a guaranteed cyberattack is speculating.
All we can do right now is to observe and note the patterns.
One reason is that if a company is under attack, 1) usually, you can tell you're under attack, and 2) immediately broadcasting "the outage is due to an attack" gives you a bit more PR cover than "umm we fucked something up badly". Companies are generally incentivized to reveal when an outage is due to something they did not directly cause.
None of the recent popular outages have been the result of a cyber attack or AI, and nobody credible is claiming otherwise.
It hasn't stopped people repeating "AI or Cyber Attack did it!" to one another in every thread, and getting more and more sure because others believe it baselessly.
That's a ridiculous statement. Outlook down for a day is not a weekly event. FAA Ground Stops aren't a weekly event.
They are things that almost never happen.
While it might be a pattern of rushed upgrades, it's still a pattern that something isn't quite right. That might just be rushed upgrades due to fear of vulnerabilities getting exploited left and right, but it's still clearly a big difference from even 6 months ago.
- FAA grounding: 3 times in 25 years
- Outlook/MS full day outage: full failure never happened before for this length
- Multi-institution bank outages: rare on this scale
- NSA major internal network outage: also rare
- Apple login/multi-service outages: couple times per year
- Outlook may have never had full feature failure before, but other Microsoft services have outages all the time. Azure is a dumpster fire. GitHub even more so. Remember when Crowdstrike bricked 9 million Windows computers less than 2 years ago?
- What scale? JPMorgan Chase had an issue for a few hours. Is that unprecedented?
- NSA outages are unlikely to be leaked in the first place. we have no idea how frequently something like this occurs.
Would people even be talking about any of these if they weren’t primed to be freaking out about it? In my circle the people freaking out the most are the people most removed from tech and software development, in no small part because they have no idea what constitutes normal and are reacting to headlines and social media feeds.
Then cite a source claiming the FAA, Outlook, or supposed bank outage, is linked to AI or a Cyber Attack. If you cannot link to a reliable source, it is by definition baseless.
So by all means "base" it. Cite facts.
Taking msteams offline for a day, then taking the apple app store offline for a few hours days later, doesn't hurt anyone or cause much inconvenience.
I suspect that an "Attack" would mean major systems going offline at the same time and staying down for a while.
IMHO, this is more likely a combination of:
- Large companies firing a bunch of engineers because WE HAVE AI TO REPLACE YOU.
- The remaining employees don't have the expertise to guide the AI into maintaining the systems.
- AI led by the uninformed is uninformed.
- AI tells the uninformed, "It's totally safe to reboot the gateway interface without scheduling downtime".
- Move fast and break things! (and you don't know how to fix them)
- Downtime.Because now all DevOps teams are being forced to look at security issues that were previously back-burnered for feature work. If an attacker is probing around like this, they're likely to lose whatever backdoor they're counting on.
These all have their consequences and this is what we are seeing. This is what happens when you care more about the stock price (stock manipulation via stock buybacks) rather than investing into your own company.
My calendar failed to login. Then I couldn't login to account.apple.com on any device.
Personally I think there's a more likely explanation.
The arrival of sophisticated frontier models like Mythos is giving us an explosion in vuln discovery. CVEs, zero-days, critical infrastructure bugs, etc. There's a massive backlog of systems that suddenly need urgent attention/patching and a lot of this crap has probably been sitting out of sight and out of mind for years or more.
A tiny little security patch can cascade into major OS upgrades, TLS incompatibilities, dead protocols, dep conflicts, cert issues, and all kinds of unexpected noise. Multiply that by thousands of org (many of which are understaffed and running ancient infrastructure) and you got a recipe for chaos.
I think about the original meaning of "catfish" - long haul ocean carriers would put catfish in tanks with other fish to keep them agitated and moving so they wouldn't deteriorate during transport. AI security research is a catfish that is forcing an enormous amount of long-overdue infrastructure maintenance.
Could it be a coordinated attack? Sure. But I am leaning towards just general lack of IT competency that is getting some sunlight due to Mythos/Glasswing/etc
It’s the chaotic good of conspiracy theories we can use to imagine a world where there’s a plan and things are going according to it vs rudderless happenstance.
I believe Apple is hosted (almost) entirely on their own hardware? So there's no reason an outage would be correlated with other outages
edit: Their status page is showing partial outage https://www.apple.com/ca/support/systemstatus/
No and it never has been. Apple is one of AWS biggest customers.
“Apple’s cloud business is hugely dependent on Amazon The iPhone maker pays $30 million a month to use AWS, CNBC reports”
https://www.educative.io/blog/does-apple-use-aws
https://123heerasingh.medium.com/apple-icloud-dependability-...
EDIT: There is a really good breakdown of it here:
https://www.awscertificationhandbook.com/blog/does-apple-use...
Apple services mostly run in Apple's own DC. Storage make extensive use of public cloud providers
>When exploring the question does Apple use AWS, it is useful to examine the specific services that benefit from cloud infrastructure.
We were testing this and it suddenly stopped working. (Both the new undeployed version but also the production version which hasn’t changed in weeks.) Seems connected to the outage.
Apple are no different to any other service provider.
https://substack.com/@thedreydossier/note/c-389611426
Source: NSA Internal Network Down https://news.ycombinator.com/item?id=50534145
As for conspiratorial, many explanations fit the (highly limited in circulation) facts. There is no basis for demoting any explanation as ‘conspiratorial’.
They were having IT issues, so they just wrote something down on a piece of paper, went to grab something from the warehouse, noted the SKU/serial number, and gave it to me.
For context, the Retail stores are also unable to do transactions, process repairs, or use any of their internal services for customers. They are essentially fully offline.
Apple is just yet another company that doesn’t give a shit
We move onward to improvisatory business practice. Cyber should take a lesson.
It reminds me of the guy who was banned by Apple for the wrong reason [1]. It took him several days, and he had to reach important people at Apple, to get his account back.
Apple certainly makes that hard to do, especially if you publish any apps on their devices. Or use any of their services that have tight integration in their ecosystem.
Both the app store and dropbox (??) have been unable to connect for the last six hours for me while other apps on ios are fine. Dropbox's status website says everything is fine, and I can't understand how that could be affected by apple services being down.
Edit: Dropbox finally caught up and added "Sign in with Apple is not working as expected for some users" to their status page
The app store is under a lot of pressure from tons of AI-written mobile apps being submitted. Similar to Github.
And no one noticed because it was just as broken as always?
Honestly, Apple Developer and App Store Connect are miserable to work with as is the whole Apple review process.
* FAA
* Banks
* Optimizely malware
* nsanet
* Apple
Time to panic?
I'm honestly wondering if our industry is fatigued from all of the activity this year. These are not examples of the best work in the industry.
The question is whether the problems all just around patching or if there are underreported exploits that are happening.
I think my current number is sitting at about “one” at the moment.
If it’s actually red, that means things are so bad Apple can’t mask it anymore.
(btw, there was a post going around guessing that LLMs has discovered 0day with exploits in the wild so serious, that several huge companies took their stuff offline to fix it)
Giant LLMs are being deployed as a police/judge of traffic on the edge of important companies. You can't deploy slowly because China!
Did anyone experience an odd momentary issue with Apple Pay yesterday?
The very oldest TCP/IP gets on trucking, or we would not even know
I'm pretty sure it's about the midterms and not about some meeting.
Someone's trying to cause mass panic so they can pin blame.
Putin does in fact tend to attack interlocutors just before meeting them, he’s done it multiple times to Ukraine.
Of course not. My comment was about someone trying to frame them.
> Putin does in fact tend to attack interlocutors just before meeting them, he’s done it multiple times to Ukraine.
If this truly is him, then I wonder how Trump ended up on his bad side all of a sudden.
Frankly unsure if Apple Pay is worth the bother for us, at this point.
Anything that goes fast, accelerates, enhances, boosts, dazzles is, in some way, whether you realize it or not, whether you agree or not, is connected to AI.
Anything faulty, slow, outdated, error-prone, is clearly due to some human in some way shape or form. If there's an ostensible cause from AI, the actual problem is probably from a human not reviewing the code or following processes, or writing a bad prompt.
My local super market had empty shelves two weeks ago because they were switching warehouses. I live in a city and we're only three + N meals away from revolution. Make your N as big as you can afford.
Real big brains at Apple.
Time to withdraw your money /s