Google outage – resolved | Hacker News Reader

Google outage – resolved

2,316 pointsabluecloud827 comments

various services are broken

- youtube returning error

- gmail returning 502

- docs returning 500

- drive not working

status page now reflecting outage: https://www.google.com/appsstatus

--------------

services look to be restored.

View on HN

If you pay for Google Services, they have an SLA (service level agreement) of 99.9% [1]. If their services are down more than 43 minutes this month[2], you can request “free days” of usage.

Edit: Services were down from ~12:55pm to ~1:52pm, it's 57minutes. Thanks hiby007

[1] https://workspace.google.com/intl/en/terms/sla.html

[2] https://en.wikipedia.org/wiki/High_availability#Percentage_c...

Given the blast radius of this (all regions appear to be impacted) along with the fact that services that don't rely on auth are working as normal, it must be a global authN/Z issue. I do not envy Google engineers right now.

4:41AM PT, Google services have been restored to my accounts (free & gsuite).

And I have never seen them load so fast before - gmail progress bar barely seen for a fraction of a second whereas I am more used to seeing it for multiple seconds (2-3 sec) until it loads.

I observe the same anecdotal speedup for other sites... drive, youtube, calendar. I wonder if they are throwing all the hardware they have at their services or I am encountering underutilized servers since it is not fixed for everyone.

It is nice to experience (even if it is short lived) the snappiness of Google services if they weren't so multi-tenented.

So, anybody still feel like arguing that 'the cloud' is a viable back-up? Or is that a sore point right now? Just for a moment imagine: what if it never comes back again?

Of course it will, - at least, it better - but what if it doesn't? And if it does, are you going to take countermeasures in case it happens again or is it just going to be 'back to normal' again?

Help! My Waymo taxi won't open the doors without me logging into the app. Its driving around in circles on route 500 and won't stop.

/s - for now ;)

Gmail said my account was "temporarily" unavailable... had a moment considering if it wasn't temporary. Good reminder to remove my reliance on gmail especially.

The status page has proven itself useless again. According to that page, everything is working perfectly fine.

When I tried to login it said: 'this account does not exist'. So my first thought was some algorithm made a mistake and my account got deleted for no reason.

I already imagined the only solution now was to write a medium post and hope it gets some traction on hackernews and google support steps in. Thinking to myself I was an idiot for knowing all this and still thinking it wouldn't happen to me.

And even though it turns out to be an outage, it gave me a bad enough feeling to start using a domain name I own for my email.

This topic is so hot it's crashing HN! Super long server response time and I get this error quite oftenly: "We're having some trouble serving your request. Sorry!"

Just checked https://www.google.com/appsstatus

all green, which does not reflect reality for me (e.g. Gmail is down)

edit: shows how incredibly difficult introspection is

As many comments in this thread have mentioned, the crash seems to happen when when logged in; youtube and other services seem to work when don't send the session cookie. This suggests something very fundamental related to user account/sessions is failing...

Did Maria Christensen make a mistake when adding "return true;"?

(This is a joke referencing Tom Scott's 2014 parable[1] about the danger of designing a system with a single point of failure. Tom's tells the fictional tale of a high level employee at Google adding "return true;" to the global "handleLogin()" function.)

[1] https://www.youtube.com/watch?v=y4GB_NDU43Q (you might need to open this link in a private window...)

1. Phone Gmail won't load. 2. Turn off wifi and use cell network. Still won't load. 3. Check hacker news.

Don’t blame people, blame systems and processes. I assume Google has a blameless process too. If an engineer can bring down huge swaths of google that’s not a human problem. As an eng your you should heavily invest in a sane test -> deploy -> monitoring process, and reward reliability.

Give people a bonus when things didn’t break, not only when there is a superhero that fixes broken things. Then you’re rewarding fragile systems that need superheroes.

It's super interesting that all Google services that I've tried are down _except_ for Google Search. What would isolate Search from the rest of Google's products such that it wouldn't be affected by a mass outage like this?

The fact it seems to be related to login reminds me of the Tom Scott doomsday fictional lecture which I would link to but you know...

Just got a text from my kids' school saying GMail is down and to use the school's direct email. Immediate reaction was "Google isn't down you idiots, the problem is on your end", go to check GMail, yep it's down.

Unless I'm misunderstanding something, they are going to have an immense SLA claim issue. 99.9% SLA on Workspace services, so any business paying for Google for Business (now known as Workspace) is going to have a credit claim (assuming the outage is longer than 43m 49s which feels like it will be).

Edit: As I comment it looks like things are coming back! Timing or what...

Gmail and Youtube are back online for me. Hangouts is beach balling. There is a warning at the top of my screen though.

"Gmail is temporarily unable to access your Contacts. You may experience issues while this persists."

Everything is affected. Its Gaia(Google Auth) outage, most probably.

Disclaimer - Googler here whose workplace chat is not working.

Atlassian's #HugOps page[1] collates messages of support for Devops as a public embedded Google Map and it really encapsulates the current feeling!

https://imgur.com/a/3RurTYf

Don't forget to #HugOps all the people who've been woken up on a monday morning with this! Hope this gets resolved soon :)

[1] https://www.atlassian.com/software/statuspage/hugops

Google is also rejecting SMTP ingress traffic.

(delivery temporarily suspended: lost connection with aspmx.l.google.com[2a00:1450:400c:c04::1b] while sending DATA command)

First the two Gmail accounts I had in Thunderbird logged me out and asked to relogin.

Then Google said my accounts did NOT exist which made me feel very uneasy. Banned? Lost all my data? OMG.

Then I got error "502 That's all we know" after entering my passwords.

Finally I realized it couldn't be just me, so I opened HN and confirmed my suspicions.

It's all up and running now but I was really scared.

I'm really curious what happened because it surely looks like Google has a single point of failure in their authentication mechanism which is just horribly wrong. The company has over two billion users - a situation/configuration like this just shouldn't be even theoretically possible.

Google Pay contactless transaction not working in UK

news.google.com returning 500 finance.google.com returning 502

I’m seeing poor internet performance overall, even accessing HN...too many relying on Google DNS maybe? Even ISPs?

The Google outage is one of the top BBC headlines now, so it has become mainstream news: https://www.bbc.co.uk/news/technology-55299779

"The outage started shortly before noon UK time, with Google sites returning server errors when visited.

Users around the world reported problems with Gmail, Google Drive, the Android Play Store, Maps, and more.

...

Despite the widespread outage, Google's service dashboard for its services reported no errors."

This puts into sharp relief how outages and downtime can hit ANYONE.

Cheers to all the small SaaS businesses out there keeping their services up and running without much of a hick-up all year round.

210 points in 5 minutes. 500 in 10. Seems everyone see's the status page and is like "hey, time to check hn" ;-)

EDIT: You can permanently refresh and see the score increase by 2-3 points each second. Wow.

EDIT 2: 1800 after one hour, but a few points were probably lost due to HN downtime. Being quick to post is important, guys;-)

YouTube shows a "something's broken" error. Similar errors when trying to access Gmail. Tried on multiple browsers and connections, got the same result. The search engine works fine, as of now.

Services are not restored. Some came up again, some not. My Gsuite business mail is still completely down, while youtube started working again.

I'm pretty sure there will be some internal conferences at Google after this to make sure infrastructure problems can't propagate across the entire company and world at this rate even in the event of a sysop fatfingering a console...

Google Home and Chromecast also rendered useless. _Hey Google, how are you?_ leads to some very passive aggressive replies.

So happy that I didnt even notice this, because I stopped using Google's services :)

Is Google Auth down because of the SolarWinds Orion exploit?

https://cyber.dhs.gov/ed/21-01/

Wow, I'm glad there are a few things that I do that don't require Google auth.

Pretty much anything that requires login is broken.

YouTube Music failed in an interesting way. My music has been playing, but I started hearing ads (I'm on Premium), which prompted me to check what was going on.

edit: added details edit: redacted my phone number edit: big mistake to add phone number edit: I think illic is right, probably not me edit: removed details

You say "outage", Google will probably say "sunsetting" or "deprecated."

But why is HN so slow? Pages take like 30+ seconds to load for me (German vantage point, other sites are fine). Does it timeout on some Google dependency or try to use a Google submarine cable or something? Or is everyone just posting to HN about it?

Google login also seems to be affected, meaning all services using google account login are inaccessible if you aren't logged in.

Gmail, Calendar, Google Accounts down in Germany

As a consequence, many websites depending on Google services are stuck too. Like the nytimes.com.

Ah btw HN is being crushed by the load lol.

Seems like services does indeed work in incognito.

https://news.ycombinator.com/item?id=20132520 Maybe this beast kicked the bucket.

I happened to use Google Takeout to download everything I had last weekend (~200 Gb)... at first I thought my account was banned and that was the only solace I had, but this definitely is killing my productivity.

I really hope this is just a temporary issue and no accounts are going to be lost. If that happens, way too many people are going to feel the pain of thinking Google makes backups unnecessary - including me.

Google has now updated the status page, every single service has an outrage: https://www.google.com/appsstatus

And somehow Pokemon Go is down, too. They use GCP for hosting, but I'm not seeing other outages on GCP. Maybe they have deeper integration with Google's Cloud than publicly disclosed?

I feel kinda bad - I'm pretty sure I broke Google as I was uploading quite a few large files via Google Drive and it was chugging pretty hard.

And now because of that it seems HN is also hurting.

I'm sorry internet.

Maybe it’s time to go outside?

Seems like the auth solution that is down. Every service work in incognito/private. Shared drives in Docs/Drive does not work. Probably connected to the auth service?

Gmail service has already been restored for some users, and we expect a resolution for all users in the near future. Please note this time frame is an estimate and may change.

We've got about 3,000+ people a minute hitting Downforeveryoneorjustme.com, pretty crazy.

I can access youtube if I'm not logged in (Incognito)

The moment you realize that the only way you notice Google is down is through HN.

Cheers all you selfhosted-FOSS-alternatives! Time to bump up those Patreon contributions...

Hackernews is also having some kind of stability issues, either everyone is on here complaining about the google outages, or somehow they share infra.

Word from inside google is their auth is down, so anything that doesn't require auth should work fine. Log out of youtube, works fine.

Even Google Sign In service is down. Sites like Airbnb, Duolingo, etc that uses Google Sign In can't log in their users.

How's things like Stadia? Or is anybody using more exotic Google stuff? Like, how are their selfdriving cars doing?

HN is also down because everyone is checking whether Google is down

HN outage is soon to be the next one. The HN server is overwhelmed by the traffic due to Google’s outrage. :/

Google is scaring the shit out of us, we were already trying to move out of Google for some past bizarre outages, and now before they went down, they outright claimed business account didn't exist anymore, our business drive and all our IMAP e-mail clients got kicked out.

This is a great and interesting thread. But I believe nobody discussed here the opposite and postivie effects of this kind of golbal outages. There are funny calculations for example of having less spam mail open-clciks or things like that. So think in such massive otuage of millions of users and auatomated process what means in term to generate less CPU, RAM, I/O and as consecuence less Power/Cooling >> less CO2 and pollution... It's huge! and at the Earth's weather impacts is like charge trillions in advance that we''ll have to paid in a future not only with money... (Economy Degrowth)

Bummed to have not received an email about what's happening and why from the Google team.

Has anyone on a "platinum"/"enterprise" google workspace plan received any relevant communication about what's happening and ETA on uptime?

Looking at the other comments, it seems highly likely zanzibar[1] is down. Their auth service.

[1] PDF: https://research.google/pubs/pub48190.pdf

GCP console is not working too https://console.cloud.google.com/, however at least the services are still running :) phew

Yep.

Launched YouTube app on the Roku and was prompted to sign in. Opened browser on PC, entered "activation code" from YouTube app, after entering my Google account username on the login page, it presented me with the reassuring "Couldn't find your Google Account" message.

Tried logging in to Gmail directly with the same effect.

Thanks to the Twitters, I realized that Google hadn't canceled me, specifically (apparently they decided to temporarily cancel everyone!).

FWIW, I typically get directed to their Chicago datacenter(s).

(Note: 7 A.M. on the U.S. east coast on a Monday morning; at least they have impeccable timing!)

What do you think, how much money are they losing right now every second?

We use Google Cloud and Firebase at work and our application and servers are still up and running. Can't say the same about the Cloud Admin Console, tho.

Somebody threw a Spanner in the works is my guess. Global-scale database underpinning everything (or maybe the giant that is colossus is having an off-day).

"all green, which does not reflect reality for me (e.g. Gmail is down)"

This isn't true. Which make it more complicated as we try to figure out what systems are down -- since their status board is usually faked.

gcloud compute instance-groups list-instances [removed] --zone=us-central1-c | awk '{print $1}' | grep -v NAMEERROR: (gcloud.compute.instance-groups.list-instances) Some requests did not succeed: - Internal Error

I'm very interested in HN' crowd back-of-the-envelope estimations on how much this outage costs the world (when we know how long it lasted).

It's impressive how fast they are at fixing outages, barely a coffee break required. I'm almost starting to believe in this DevOps thing...

Interesting that I have been watching a long video (over an hour long) and the buffer is still loading while YouTube in general is down for me

Amazing how many products are clearly getting affected due to this. The top 18 products on downdetector.com are all Google-owned or related

is it me or I noticed that google search rarely goes down? is it somewhat in an isolated system or a fundamentally different architecture?

youtube-dl is still able to download videos, which is quite interesting ^^

Singalong: She broke the whoooole world, with her change, he broke the whole wide world, with his change, they broke the whole world with their chaaaaange!

I'm sure its stressful right now. But someday, these engineers will look back and retell the stories about how it happened and the lessons they learned. Hopefully with a laugh.

GOOGLE UPDATE YOUR GCP STATUS PAGE. Stop making us run around trying to figure out what systems are down.

Running: timeout 60 gcloud compute instance-groups list-instances [removed] --zone=us-central1-c | awk '{print $1}' | grep -v NAMEERROR: (gcloud.compute.instance-groups.list-instances) Some requests did not succeed:

- Internal Error

I’m seeing poor internet performance overall, even accessing HN...too many relying on Google DNS maybe? Even ISPs?

FYI if you have cronjobs using Google apis, disable them or delay them.

a bot is flagging ips as abusive. it should clear up later.

Gcloud/GKE command line utils are also having problems.

Was deploying some test applications and kubectl started complaining about gcloud auth helper throwing not zero errors. Trying to launch cloud shell from the website and nothing happened.

The web application which is a site that does not rely on any external API is running fine.

Google Meet and YouTube also 500 in Brazil

Not just google, hackernews was also displaying: "We're having some trouble serving your request. Sorry!", has it something to do with google architecture?

Btw, that's funny you listed all the google services exactly in the order I found them unavailable: first youtube, then gmail, then docs

Can I just say I'm remarkably relieved I have an excuse to kick back for the next hour and do nothing?

I hope it lasts a bit, not just five minutes. We might finally question our dependency on Google services.

It looks like the auth service is broken. Everything works when logged out. Attempting to log in gives "Account not found".

EDIT: Looks like they logged everyone out while they fix the auth issue, presumably so people can use YouTube and other stuff that doesn't necessarily need login?

Was wondering why youtube can't show videos from the search page , it even reverted my language settings.

South Africa

Seems like the incident lasted from 04:00 AM PST to 04:30 AM PST, at least for BigQuery (sorry, image timestamps are in JST, since I'm in Japan): https://imgur.com/a/kiHED6v

Can confirm Incognito Mode and Private Tabs are still working on YT. Seems to be something auth based.

All services are still down, https://www.google.com/appsstatus#hl=en&v=status

Crisis averted, was not locked out. Gotta take out all the data though when it gets back up.

Thank God! Google Cloud Print is still working! I'd be destroyed if that ever stopped working!

The worst outage I've ever seen with Gmail. I can't log in, getting "Couldn't find your Google account" message. Also, with the already logged in account, I can't browse anything but whatever is in cache. I'm definitely worried.

It's time to decentralized Gmail, Youtube, and Google Docs, maybe their Search Engine too.

Anyone experiencing lost emails? I'm not sure if it's actually me messing things up or because of the Gmail outage, I couldn't find an extremely important email and I've tried archive, trash, deleted, spam, sent boxes..

I did not realize how dependent I was to google/google services until 30 minutes ago.

This affects GCP as well as Google's consumer services. I'm here because I was paged about low traffic for a service. Turns out to just be that I can't access any metrics at all from Google Cloud Monitoring (i.e. Stackdriver).

So to easily fix such down time, i have alerts setup for website, server and API using https://simpleops.io/

After that I realised how frequently you can get down time

These rare events help make extremely visible how much of our general basic infrastructure we've farmed out to a small handful of companies, centralizing something that was supposed to avoid the problems of centralization.

Look the status page still works https://google.com/appsstatus#hl=da&v=status but it says everything is broke.

Luckily nobody is using Google services for anything critical. Or what, are you?

I feel like Google is having more outages recently. I worked there for 4 years and even in such a short time you really noticed the shift from engineering focus to business focus. But maybe it is just a coincidence.

Switzerland's down too. It works if you access it without session though.

Seems to also affect Google Cloud console to some extent. Dashboard data is not loading at the moment. Not sure what services exactly are affected there. Google Cloud SQL seems to be up and running so far.

youtube greece is down returns "oops" error but i can still playback a 2 hour video that I was already watching before it went down. Trying to open any other video or channel returns this error.

At this point Amazon and Google should just have a static page with green buttons for their status page.

They NEVER show red, or oddly even go down themselves during outages. ( Last time amazon went down springs to mind).

Yup, suffering it right now.

At times like this I am thankful of running a Google takeout periodically.

Not because I use it as a live failover for Google services, but because these outages are good reminders of the mortality of online services.

https://viewdns.info/ismysitedown/?domain=google.com points to 172.217.1.46

Google employees can't login to update their status board on https://www.google.com/appsstatus ?

Seems to be down in a bunch of places https://istheservicedown.co.uk/status/gmail

Good. Now, at least a few people would search Google alternatives in DDG

How coud all of googles websites be down this long tho. You would at least think google being the company they are would have some sort of like backup serevers or backup plan type stuff. Idk

Not a fan or even their user but kudos to Google for admitting the failure. Amazon (at least AWS) customers would have probably been shown a green dashboard and told to review their code.

It's back

Apparently Google "can't find" my account.

I was so scared for a while, then I found that all of my family members have the same issue. Then checked HN and breathed a sigh of relief :)

A good explainer video here: https://www.youtube.com/watch?v=5p8wTOr8AbU

YouTube is working in India. But can not log in to any Google Accounts. GMAIL and Drive not accessible via web. Drive sync is working though. Appstatus still showing green.

The cool thing is that newpipe is working while youtube is down.

Strangely everything seems to be little faster in incognito mode

Now we'll have a bit of outrage, talking about how dangerous it is to have the whole world impacted by one company's processes and forget about it in 2 days

HN is faster and more accurate than status pages of major service providers themselves. Special thanks to people working at those service providers and posting here!

Waiting for the post-mortem to see it was all because someone forgot to renew an SSL cert that nobody knew was a single point of failure for literally everything.

Seems that Google has no budget to pay their GCP instances

Google Search seemed to be also down: https://serpapi.com/status

Time to mention the alternative, a decentralized internet, e.g. https://zeronet.io

How much do you think Google can learn from this incident's postmortem? Writing down all the internal details probably amounts to a small book.

Gmail is currently returning "550 5.1.1 The email account that you tried to reach does not exist" for many valid gmail addresses.

In inkognito window (without Adblock) it works for me!

Google page is right: servers are up and running - but not reachable. Could be an infrastructure attack (DNS?) or someone just broke the cable :)

I wonder why services like Uber (which rely on Google Maps for everything are still working, with things like routing being completely okay).

Yeah weird it’s all green. I just noticed things down as well. Came to HN to see how quickly people posted. This is already #1 at 6:55 am EST

Appears to be only affecting pages that have an authenticated session; Youtube works in a private window, but not in window with cookies.

Also Google Voice. I ctrl+F'd for that one, and it's one thing I haven't found a way to easily degooglefy from. Also down.

The appstatus page from Google is a good indicator for me of which products are currently live from Google. Uhm, more of a sidenote :)

"Couldn't find your Google Account"

Seems to be auth related. Login does not work, and Youtube is also throwing Internal Server Errors, but only if I'm signed in.

Seems to work when you're not signed in. Won't help with gmail and docs, but maybe a workaround for YouTube and search.

"Sign in with Google" also fails. So this outage is taking quite a few otherwise unrelated third party apps with it.

Google Cloud is also down. Can not access either Cloud Console or any of our GKE clusters. Stackdriver, Cloud Auth also down.

Even my personal e-mail (using google workspace) is not accessible

Did some asshole forget to renew a certificate or run the cron job to renew?

Gmail, Chat and Meet down from Italy as well, was in a Meet call and got disconnected, only getting 500s across all three now

Allright, what did Gilfoyle do this time? ...

Jarring reminder of how addicted I am to YouTube... getting childishly irritated that I can't continue to binge lol

Discord also seems to have gone down for me. Also (probably unrelated?) HN is really slow compared to normal right now.

Youtube, Gmail and Drive pages open in incognito mode but can't sign in. The issue seems with the account manager.

HN also seems to be loading slow with sudden inflow of traffic. Would be funny if HN goes down because of Google outage

Migrated out od G infrastructure long time ago. The G Search working fine, G Maps down (continental EU). HN very laggy.

Seems the issue is related to the google authentication system since if logged out you can visit youtube without error

At least Google DNS (8.8.8.8) still works. So, the people who use it, still have access to the rest of the Internet.

Seems like some core service crashed since all of these sites are unavailable. However, google.com seems to be fine.

UK here, YouTube and Gmail are innacesible, search is working fine. Workspace Status Dashboard is showing all green.

Drive too. Status board still green though.

All down, but all green.

New norm, let's have status page, but make it green all the time :-). Surprisingly consistent practice.

Looks like the problem is with Accounts. Most Google apps work when not logged in, i.e. can view YouTube videos.

Same here in Germany. Gmail says that my account can not be found. Plus Google's support page is also down

Whoose laughing at self-hosters now huh ?

I think they are out of hard drive space.

My Gmail app on iOS keeps displaying a “No Connection” error even though my phone is connected to the Internet

The problems seem to have been (mostly) resolved over here, at least the authentication seems to work again.

It's also weird the page title with "Error 500 (Server Error)!!1" - Why !!1 at the end of it?

Everything down in Canada. At least everything that seems to require auth. Gmail, Gsuite, Youtube and more.

All G Suite down for me, says "Couldn't find your Google Account". Broke youtube mid-stream.

How does this incident affect IoT devices? I'd expect many of them to be using Firebase just like HN.

Looks like if google services are down for more than 45 minutes, Workspace customers gets 3 days added.

I wonder how on earth googlers are supposed to fix this when they can't log in to make changes...

YouTube work for me in Firefox just fine.

But I have it configured with some of the recommendations from Privacytools.io

Just use alternatives:

* DuckDuckGo instead of Google Search

* Bitchute / LBRY / Odyssee instead of YouTube

*Nextcloud instead of Google Drive

2000 points in an hour on HN. 120K upvotes on down detector. Trending worldwide with 500K tweets.

Google Cloud Platform is down in US.

Maps could be the scariest one! All those drivers wondering how to get where they're going.

This is a good time to think about what would happen if you lost access to your Google account.

Maps are partially affected, page review returns 500, now even location pages doesn't open

It is possible to reach youtube in incognito mode, so it seems to be a google accounts issue.

Transitioned to Google Workspace from Microsoft Office 365 last week and it’s a nice welcome.

All service down in Sénégal as well. I first noticed Youtube, then everything else followed.

Yes, I can confirm this. I got an error when accessing Google Docs, Google Meet, and Gmail.

Calculating cost of downtime ==>

World economic output is - $150mn/minute

Current world population is 7.8bn.

Google has 4.8bn users. That's approx 61% of the population.

Let's assume about 50% of the users were impacted = 2.4bn (which is 30% of the population).

So, the loss could have been about $50mn/minute.

This is without taking SLA into consideration. There will be losses incurred on that too, wouldn't it?

Looks like the authentication services are having problem. I have account not found error.

Gmail and Drive down in Thailand.

Blogger is also down, even for public site access so it's not only auth related.

Curious that minutes ago I was listening music on YT. BTW not all services are down.

Hungary - gmail down, youtube down if logged in, search works with 5s response time

Sweden. The same problem. Authorization doesn't work. Gmail doesn't work.

All down for me, Serbia. YouTube works for me when logged out (i.e. in Incognito).

Yep. Gmail redirects to Gsuite Status Control and shows everything as fine there.

And just on the day that electoral college certifies the election. Great timing!

Hey can someone explain to me how all google apps are down cause idk a lot and ik that at least a kid ddosing these websites is not a possiblity so what kind of attacks are going down here? And how come google doesnt have like backup servers to run there websites on or is that not a thing?

I hate the fact that the status page is not reflecting anything related to it.

In Japan Youtube is down too.

Google Cloud outage? Looks like Discord is down too. Might be a coincidence.

Back up for me, almost. Some warnings on gmail but the inbox is loading up.

Whataya know Youtube didn’t work and I came to HN to check what happened.

Down in india As well, Gsuite, GCP IAM, Youtube, Calendar, Drive et al.

Google Workplace (EU) is down and also Google Cloud (login at least)

Same here, can't access gmail (currently on a trip in Poland).

This morning also Microsoft O365 had problems with authentication

My service accounts for the play store are all returning err 500.

Anyone else has this in the technical details? Numeric Code: 7444

Is google's status page actually linked to anything?

Or is it ahem airgapped?

I'm able to access GMail and Maps but not Youtube or Drive.

HN is loading very slowly too for me. Related to Google outage?

HN is the fastest way to know about outages.

Youtube & Gmail are down for me.

seems to be working again, for me. except gmail says "Gmail is temporarily unable to access your Contacts. You may experience issues while this persists."

Seems to back up. Most services working outside of incognito

Can verify - India. Gmail, Drive, Docs, Youtube not working.

Alors, Search queries taking up to 3 seconds here in France.

Hungary - gmail loads, can't send email (error #2014)

Has google.com ever suffered such outages in recent past?

This is google's way of saying do not work from home

Confirmed from Spain. Google Workspace seems to be down.

Everything down in HK

Everything seems to be back to normal, for me at least.

Looks like some Google Cloud customers are offline too.

Can verify - Finland

>drive not working

Something something someone else's computer.

Can confirm from the Netherlands, Hangouts also down.

Only thing working for me is the Google search engine

Guys, chill out! We're DOS'ing HN already!

regular text search is working for me right now, but gmail and maps are totally broken. google drive/docs also totally non functional.

yup, all down in Vietnam too, this page took some time to appear too, so i suspect a lot of issues in the pipe (mostly via China i gather)

It would be so interesting to see the postmortem...

The console for Google Cloud Platform is also down.

Drive and Gmail seem to be back for me (12:31 GMT)

Can't access my files on Google Drive, shit.

Looks like Hacker News is having trouble as well

Can confirm, gmail, gdrive and youtube are down.

On reddit thread there are some really good jokes about this [1] related to their asinine interview questions. Like:

> Did they try to fix them by inverting a binary tree?

>> Yeah maybe implementing a quick LRU cache on the nearest whiteboard will help them out here

>> Did they try checking what shape their manhole cover is?

>> Dev ops was too busy out counting all the street lights in the United States

[1] https://www.reddit.com/r/programming/comments/kcwqij/every_s...

I was kicked out of a Google Meet call just now

Google meet/hangouts, appliances, the lot.

Youtube is up for me, but CANNOT login to Gmail

Yeah, drive and youtube seem to be down as well

Gmail, Docs and YT down for me too in Romania.

Youtube down in India, Docs Down, Mail working

Is it due to the SolarWinds Orion exploit?

https://cyber.dhs.gov/ed/21-01/

The Nest endpoints are returning 500 as well.

All Google services down in Sénégal as well.

Some stuff works if you are unauthenticated.

Google play web also down along with others.

Confirmed: all mentioned above down midwest!

Gmail and YouTube are down in India as well.

I think it's back. Youtube is loading.

Can confirm from Germany, youtube is broke.

Even HN is being hugged by death of Google

Youtube, docs down in India, Mail working

Yep, drive and gmail give me 500 errors.

YouTube shows a monkey for me in Israel

youtube works if you don't log in. Otherwise, gmail, cloud, youtube with accounts are all down.

Seems to be back up (Europe/NL).

Here in Brazil, YouTube has just up!

GSuite all down. Confirmed in Japan.

Most of google services are offline.

Guys, Orkut also seems to be down!

Not working for me either, Canada

It seems 2020 is not done yet :).

incognito youtube is working btw

Same situation here at Liman 1.

Meet is also down. From Sweden.

Yeah hangouts/meet as well

Google Login down in Australia

https://www.chrisranjana.com why

same just lost access to all services including Firebase here in Toronto Canada

Incognito seems to be working

GCP seems to be down as well.

Google classroom is also down

Google Home not working in UK

Australia, same problem here.

Pour one out for the SREs...

All services back up for me.

Seems like they are up again

Hmm. My magical spell works.

Hmmm google is back... wonder if this was an attack or if this was an internal problem with something?

[deleted]

Exciting!

Maps is affected as well

GMail IMAP is also down.

Google meet is also down

i wonder what % of the hn crowd cant work if google auth is down

It's restored now.

Can confirm from India.

Calendar is also down.

Going back to normal.

Firebase is down too.

It's up in Japan

Fastmail is still up!

Also down in Paraguay

Google home down too!

Down also from Italy

Cool way to end 2020.... Pandemic and now no more Google. Thank you God :)

just authentication services does not work in Turkey.

It's up again!

Reply on news.ycombinator.com