Our production servers were suspended by Google Cloud
onvoard.com
onvoard.com
Similar horror stories:
https://news.ycombinator.com/item?id=32547912
https://news.ycombinator.com/item?id=33737577
-----
[Update 1]
Yes I agree I have some fault for the initial suspension, but that's not the biggest issue here.
The biggest issue for me is how they "react" to help a customer get things back to normal when something goes wrong.
First, I had to wait for 8 hrs for their Trust and Safety team.
Then it was 24-48 hrs. See https://imgur.com/a/i0OHXcI
Until now I still haven't had a resolution.
And the most ridiculous part is they could still tell me "I just received an email from the Trust and Safety team and they mentioned that they have sent you a separate email." when I had replied 3 times within an hour after they requested for more details. See proof https://imgur.com/a/1ZSmGBH
Usually I don't care much for grammatical errors, as long as the point gets across clearly. In this case I don't know why it irks me.
But no, they insist on using every product the different cloud providers offers. This, IMO, is a recipe to lock yourself to vendors and make maintenance and interoperability a nightmare.
It feels like when devs felt they had to use all the gang of four design patterns.
- Solicit a hard limit when you open an account. People can only afford so much, don't bankrupt them.
- As soon as we see your account heading for a 10,000% increase on last month, we'll send a SMS, GIVE YOU A PHONE CALL, even send a letter and confirm everything is okay.
- If you're about to trigger a suspension, we'll send you an automated SMS, and if you're a paying customer, we'll do you a solid and spend $5 calling you to make sure you understand what's about to happen to your business.
- You get one 7-day extension. People get sick, go on holiday, have life in their way and often haven't figured out how to juggle business critical credentials amongst people.
Just imagine the quality of life improvements if people felt safe using a cloud provider. Instead we see stories like this and withdraw back to colocation and flat-rate VPSes.
if your spending maybe 4 but definitely 5 figures you should have an account manager. I would argue same on the publisher side.
-A hard limit? Why limit your income to $600 when your customers can accidentally rack up a $6000 bill? You can even be magnanimous and negotiate it down to a $2000 bill and everyone's happy.
-Why would we pay money in monitoring software, SMS & phone call fees to stop a 10,000% revenue increase?
-$5 to try to tell a customer not to give us more money? Are you crazy?
-People getting sick under capitalism are drags on the economy and should be derided and made as miserable as possible so it doesn't happen again.
> Just imagine the quality of life improvements if people felt safe using a cloud provider.
We're not here to make people feel safe. We're here to make incredibly rich people even richer. Get on board, man. You'll never operate a successful company with your approach, everyone else will beat you on price.
(And yes my satire detector is triggered, though one can never be sure)
AWS is pretty much a line of credit, unlimited credit actually.
When you open a credit card you are usually given a limit, say 10K.
If there is a (pretty obvious) suspicious activity on your account it gets disabled, the bank just won't let your card be used indefinitely to buy yachts and planes.
And when you contact the bank they won't ask you to go find the bad guys and stop them from using your card before they can help you.
That would make absolutely no sense but that's exactly what AWS did in our case.
As is said every time this gets proposed, this is not only wildly impractical to implement, but also dangerous from an availability. Why is it impractical? You are billed for active things (running databases, network traffic) but also passive ones such as storage. What do you want AWS to do when you go over the hard limit, delete everything? As for the availability, nobody wants to lose the production environment because nobody has updated the budget with more appropriate values in years. What AWS and GCP do is allow you to set a budget, with notifications however you want them when you approach/hit it. When you get a surprise bill due to a misconfiguration or whatever, you can dispute it and cloud providers are extremely generous with them and usually waive the first few errors.
I agree with the rest though.
They don't need to cut me off right away (unless I'm already outside my budget) but for most bitten people, getting a call the day after making a colossal mistake will save everybody a lot of money and angst. Neither of us needs to negotiate and I don't have to sell Grandma.
It's been a few years but the last time I tried AWS's Budgets, it took a hundred clicks or so per service to just get an email out of it. Emails get missed, life gets in the way, etc. The cost to a trillion dollar cloud host to call me up is far less than it would cost you to call me as they off-shore it all.
And I'm all for people actively choosing to play without limits. I just think a lot of people a) don't respect that from the off, and b) aren't notified in a timely fashion.
I had my own feud with AWS over a $60,000 bill and an enormous amount of hours spent by my team fighting crypto hackers after they took over one of our accounts and use very advanced scripts to create hundreds of large instances.
AWS has become very complicated to manage and at the same time the attacks to their infrastructure have increased exponentially.
If your company doesn't have a dedicated team to stay on top of AWS you'll be vulnerable to attacks like the one I experienced.
You simply can't count on the baseline configuration of your account and instances provided by AMS, you need advanced knowledge on how to secure your resources beyond basic stuff like MFA and monitoring.
And if you think AWS will have your back when shit hits the fan, lol good luck with that.
They will try to collect the bills while providing absolutely no support, reminding you about their Shared Responsibility Agreement.
They also expect you to fight the attacks.
I ended up spending months of work securing my services and cleaning my debt with AWS billing, to the point we almost went to court.
So again, you may have good reasons to migrate, just beware that AWS won't come free of hassle.
On HN the usual way to quote is like I have done above.
I too react like the poster above you and would probably not quote that way except to ridicule. Hopefully there aren't many of these across my posts.
(As for an almost green account commenting on HN standards I have been here is some form or another since around 2009 I think, I just once in a while create a new account so it won't be trivially simple to doxx me.)
Extraordinary claims require extraordinary evidence.
Edit:
By the way, what do you mean by threat level here? And FWIW, here I could have used quotes without being rude.
Also: if you read that sentence again you'll probably find that it wasn't just the use of quotes but the sentence that as a whole that made it stand out as rude.
Suggesting it is an extraordinary claim, and would suggest an extraordinary threat to all AWS accounts. This seems very unlikely.
The attack resembled a very aggressive, clever virus so the people behind it definitely had very advance knowledge of AWS.
They could enable regions we had disabled and recreate roles that were deleted in a matter of minutes, again MFA on, no keys.
AWS kept giving us conflicting instructions and refused to disable our account unless we stop the attack first.
We have been customers of AWS since 2014.
I believe during that time the Shared Responsibility Agreement did not exist, so AWS demanded we signed it now for them to even consider taking a look.
That's when I reached out to the Attorney General of my state who sent a letter to AWS and got things moving.
We were contacted by another team (two months later) in AWS that seemed much more knowledgeable and they gave us the scripts that eventually stopped the attack.
Our case is not that rare, if you do some research you'll fine plenty of stories like ours.
In what way did the attack resemble a very aggressive, clever virus?
Do you mean that it happened fast? How do you know it wasn't something simple like your credentials leaking?
What advanced knowledge are we talking about here?
When could you? when AWS was security by obscurity a decade ago.
When you create an account, one of the first things AWS has you acknowledge is the Shared Responsibility Model.
AWS has always given you the rope to hang yourself.
I was trying some VPN services at the time, and logged in to an AWS account with the VPN turned on, after which AWS suspended resources in the account. This took a two days to review and fix, though the only saving grace was that I did not have a paid offering running there.
When pressed as to why they'd flag my account, they'd would evade the question as to why my account was suspended, though I'm pretty sure it's the VPN. (And before you ask, no, my account was not compromised, as I had to review 90 days' Cloudtrail logs before AWS would budge on the issue).
What was very advanced about those scripts?
As an outsider, I'd think that perhaps a very simple script might be enough to create hundreds of large instances.
You would be absolute mad to use any other platform. AWS gets it right on the customer service front. Amazon, perhaps deeply less so... but AWS? amazing.
Tech support was always friendly, polite and apologetic, but could not provide help.
The attack was way beyond their level of skill and they insisted in providing instructions that were completely helpless, all manual actions to fight a full scale extremely vicious automated attack (change password, delete instances, disable regions, delete roles, etc).
AWS security got it done in one hour, but it took months to get to them.
That's not to say that Amazon is being altruistic here, I think it just makes good business sense. Foregoing a few thousand dollars for even a tiny chance that a student's future self decides to use AWS for a multi-million spend project down the line is worth it.
This has been my experience, 5+ years into our 'cloud journey'. For all those people we didn't have to have on board because cloud magically made them redundant, we've had to add almost as many people to ride herd on our relationship with the cloud providers.
TANSTAAFL
Was the 10 day countdown the first time Google mentioned "we'll suspend the account unless..."? Generally on these things - and unless it's an absolute emergency - you'd expect lots of prior warning and at least a couple of "YOUR SERVICES ARE DUE TO BE SUSPENDED" emails in the 48 hours before.
That said, I've rightly called my bank out for sending an email with the subject line "Your account was compromised" - and then the body began "...is what you could be reading if you didn't have adequate security", followed by a sales pitch for some cyber security software. When you get rubbish like this (or absolutely banal "important information regarding" emails) so often, it's hard to know when to bother.
AWS at least prefixes mails with "[ACTION REQUIRED]" if there _might_ be some action on your side required.
And by coubting the mails that are coming in today with "important information", I can fully understand if you'll miss one.
Until everyone uses the prefix everytime...
The industry has moved beyond depending on a single cloud computing vendor. It’s negligent to put all your eggs in one basket.
I would say don’t put your business in the hands of a single cloud vendor. Either exit the cloud or go multi cloud or run plain old Linux machines on hetzner or ionos. Design your systems to withstand your credit card expiring, or missing an email.
Given how much of the industry struggles when an AWS region goes down, this seems... unlikely.
(... yes, more cost-prohibitive than losing one or two days of business to a SNAFU).
The reality is that all serious outfits either host on AWS, or keep some level of AWS account at the ready, in addition to bare metal Linux servers in colos or hosting providers that they can move onto if needed.
Your statement is confusing simplicity and flexibility with multi-cloud. No one wants to deal with the confusing mix of random hosted services that 3 different cloud providers try to sell you, with one of them (Google Cloud) being this opaque horror story that the article is talking about.
Host on AWS, and mostly host your services on EC2. It's dead simple to migrate elsewhere if needed. You will likely never need to.
Don't host production stuff in there. And if you must, have a backup somewhere.
Mutlicloud is great but somewhat complicated to setup correctly.
The value add of AWS is that Amazon has unusually good customer service.
Amazon may not be particularly customer centric in terms of the products they allow on their market place, or AWS by the plate of spaghetti that is their various cloud offering, but their customer service division is great. You can get in contact with them in hours, and there's never a hesitation to give you refunds or credit if something goes wrong; something I can't say the same of when it comes to Microsoft or Google.
1. Excellent customer service, that tries to make the customer as happy as possible, within reason.
Amazon seems to be the big example of this one.
2. Terrible customer service, that has a hostile attitude towards the customer.
I'm not sure who's a good example here. Comcast?
3. Non-existent customer service.
This approach has been made famous by Google.
I know when I call my insurance or bank that I'll mostly get things sorted, however I need to jump through 101 hoops to get to the right person via a automated service. Stop being cheap and just hire some operators to wire you through to the right people. It's not terrible per-se, just some combination of good (helpful customer service reps) and bad (automated operator).
In that case, the vast majority of enterprises are negligent.
> In that case, the vast majority of enterprises are negligent.
Vast majority of enterprises are not mission critical and their products don't need guaranteed downtime lower than 1min per year. What they need is a lower price, because people don't care about quality in that case but about costs.
I’ve hosted on AWS for ages. They don’t do this. We didn’t pay the bill for two months once because the owner of the card left. No problems. This was a small spender outfit. They could have thrown us on the street as we were a rounding error. Now we’re a big spender and we still forgot to pay them sometimes (sorry Bezos)
But some of your point is worth considering. Design for portability should the worst happen (you are cut off due to geopolitics or other problems) or somewhere else turns out to be a better deal. Stick to IaaS and portable services only.
For the moment I trust AWS though.
But having this reputation in the tech bubble negatively effects things like google cloud that tech people get to make decisions on. This is one of the reasons that google cloud is so far behind AWS and Azure in adoption.
As a standalone product I'm not sure anyone would use it.
It also helps that there are multiple funding sources, so I can split the bill between multiple cards or accounts.
I travel the major part of the year, and it is quite common to get my cards lost (often at my own fault) or compromised (Mastercard tends to suspend cards when they detect charges from multiple countries in a short time frame). I can wait for the new cards to arrive worry-free that my cloud services don't take a hit like this.
DO, Vultr etc. would have the same issue depending on the timeframe they were given to prove compliance by the feds.
- 2023-03-04 "Your Project: nono is at risk of suspension"
- 2023-03-04 "Action required: your billing account 0028C9-xx-yy is past due or has invalid payment information"
- 2023-03-04 "Google Cloud Platform & APIs: Update your payment method to keep services"
- 2023-03-04 "Google Cloud Platform & APIs: Payment was declined for 0028C9-xx-yy"
Similarly, I got four emails from Digital Ocean, but only one from AWS.
There is no way a non-internet business would cut off a client of 4-years without actually talking (phone) to them.
I come from a healthcare and telephony background where "five nines" (roughly five minutes and 32 seconds of cumulative downtime in a year) is actually still a thing. I'm absolutely amazed at the number of companies that just accept each region/service/control plane/networking/etc outage in the big clouds. First you wait an extended period of time for them to even acknowledge the issue (rarely even updating their status dashboard) and then you stand around powerless waiting for your business to come back online. Time and time again.
Yes, multiple region deployments are encouraged (still not even five nines) but when AWS and others give you the ability to create really convoluted architectures (which they love and encourage - it's sticky) using at least a half dozen of their products (IAM, VPC, RDS, S3, etc) for the simplest solution that make it virtually impossible to deploy, debug, and maintain multi-region without a dedicated "AWS Architect" role. Downstream outages for solutions/companies building in clouds because of region or service issues demonstrate just how difficult this approach is in practice.
Then you sit back and wait for the almost inevitable "where did this $50k invoice come from", "we ran through our monthly budget overnight", "Oh they don't like us anymore and are closing our account", "What's the spend going to be this month - no one knows" and other scenarios discussed in this thread.
Can't figure out how to wrangle costs? Sick of the outages? Good luck getting your cloud Rube Goldberg machine migrated out (especially because by now you've hired based on cloud of choice).
Depending on a variety of factors any of these scenarios can kill a company.
This isn't normal. This is regressive. It is unacceptable.
The first rule of production on GCP is to get off credit card billing, and that has been pretty well-known for a while.
However, the law requires them to not do business with people on the US embargo list. And to verify that, they need your name.
That is probably why there was no way to ignore this paperwork, and why customer service couldn't override it.
At the very least they should try calling the account holder before suspending the account.
If they can't reach the account holder, they should re-enable the account very quickly after they reach them, and give them an extension with enough time to provide all the required information.
The 2010s dream of "megacorp serving everyone with centralised programming" ever more exposed as an unethical pipedream.
Many of the Google horror stories are less a failure of automation and more just a failure of well designed business processes. This same event could/would have occurred if it were humans sending the "additional information required" emails and also following a protocol of "shut them down in 10 days if they don't provide". The fact that this process was (most likely) automated is not the issue.
Honestly, as an outsider, it would appear that Google is a tech company that isn't actually very good at business. They do some smart tech things with their huge number of smart workers, but the business side seems to make a lot of very poor decisions (or simply doesn't even consider some part of the business or doesn't prioritize business processes well).
Would anyone imagine that the decision to cut off a customer entirely, is not checked by a clerk? by a lawyer? The defense “It was automatic” doesn’t even pass the basic sanity test.
Their primary product (ads) sells itself.
So as a company they haven't ever had to become good at business -- in the interacting-with-other-business-entities sense.
If it's business critical, talk to a human. All the cloud providers have account managers just waiting to talk to you. If this sort of thing happens and you have an account manager, they call you!
Even better, get a contract that prevents them from cutting you off like this, or sets an SLA for getting back online. Again, all the cloud providers will offer this, even for relatively low spends.
There are no shortages of incompetent people and institutions that will screw you. It's typically incompetence, not malice. But you will run into malicious actors as well.
The reason I sort of agree with you is that, being someone who is very comfortable with bare metal solutions and came from the 90s mentality of writing portable code, I have started to see "The Cloud" as a return to mainframe programming. There is nothing wrong with that, but you are programming for "a machine" that often leads to code that is exceedingly hard to port. "Cloud Agnosticism" is the industry buzzword for not locking yourself into one vendor, but you need to make that decision on day one and set engineering policies enforcing a cloud services abstraction layer to be maintained. Most startups in the process of scaling are not going to see the immediate business value in incurring that cost. (though, with more horror stories like this one maybe that will change... but I doubt it).
Have a falling out or big issue? No reputable colo provider is going to hold your hardware hostage (it's the physical world and legal and reputational issues heat up quickly). They'll be happy to ship it out (get rid of you) to another provider, who will be even happier to re-rack. Meanwhile if your architecture is even remotely well designed (load balancer, multiple facilities) you're never down and it's not even that much of a "fire drill". With the capabilities of modern hardware this is often a single 1U redundant HW machine.
All of this, by the way, is a fraction of the cost of cloud solution (a $25k bonkers 1U is ~$700 a month with a three year lease and 1U hosting + bandwidth + power is ~$200 a month). So for $2k a month (and it's actually predictable) you're up in two facilities and have capabilities an order of magnitude greater than a big cloud at equivalent spend.
Companies screaming on twitter and profusely apologizing to their customers due to the most recent "name a cloud" outage demonstrate that multi-cloud (and even multi-region) is extremely difficult if not impossible to do in practice.
You mean you have raw power an order of magnitude greater than a big cloud at equivalent spend. Capabilities are all DIY - if you want a database, object storage, replicated storage between your two locations, message queue, etc. etc. you have to set it all (and think of the backups, patching, monitoring, etc.) on your own. That's a non-insignificant costs that will easily dwarf that $2k. Source: I've done this at an MSP/MHP, handling a decent chunk of the infra side.
I considered a "hybrid" model where I would keep bare metal servers for distributing media but move compute and persistence to "the cloud" but just couldn't really justify it on cost alone. The replication and auto-scaling would have been nice, and were the major selling points, but wasn't a high enough ROI for me.
Postgres, minio, rabbit, whatever absolutely does not even come close for initial implementation and ongoing support. If you spend more than a tenth of that you’re doing it wrong.
Source: CTO who’s done all these things.
You missed a KYC e-mail and they locked your account. If you miss a "Important information about your bank account", what do you think your bank is going to do?
1. Send more than one email. Three would probably be reasonable for an account shutdown action maybe at 10, 4 and 2 days.
2. Use a better subject. That subject sounds like they are giving you information. At least something like "Required Important Information about your Bank Account". But something even better would probably be like "Must Provide Information about your Bank Account to avoid Account Closure" and maybe some escalating severity in the later warnings.
3. Do they have a phone number on file? Give that a call if there is no response to the original email in 3 business days. It could even be an automated message. "Hello, this is Google Cloud. We have sent you an email about required action on your account. Please read the information in that email and act appropriately otherwise your account will be closed. Remember that all Google Cloud emails come from @google.com addresses."
4. A big banner in the Cloud Console. After 4 business days show it to all users in the account.
We are talking account shutdown of a paying customer's likely production services. They should take a bit of care.
The OP certainly has some culpability but having bad experience of this process myself multiple times, it does feel as if Google designed it to be hostile — the emails being just one part of it.
Google Cloud is great in many ways, I love the product, but the billing is such a pain, top to bottom.
Haha yea. Like the post notes, the Data Processor notifications have gotten so irritating that I had to setup a filter to junk them immediately.
GDPR caused spam being foisted on the rest of us not in the EU.
I’m sure they will do better with AWS because AWS isn’t stupid enough to kill everything you have after a single email warning.
No, but Google pretending that email is reliable way to inform customers about such important thing is.
All businesses need to at least consider BC and DR, even if they eventually decide they have to risk it. There are loads of reasons why a dependency goes down which you have no control over so even basic spreading between providers or having a less-capable but simple DR system can go miles towards having a business not get killed in this way.
Any sort of weirdness from your server will also result in a very harsh email and a threat to suspend your services in way less than 10 days. I had the pleasure to experience their grumpy network admins when IPFS decided that node discovery included dialing 900+ RFC1918 IPs.
It's so easy, all you need is x. Then when you're far enough in they demand a through w or you're shutdown with extreme prejudice. Do they really need it? Is that what you agreed to? Did you negotiate the change in terms. "I am altering the deal, pray I do not alter the deal further..." They don't blush. They don't flinch they don't even blink. Good luck if you really don't have another choice.
Is the whole "cloud" just a con to trap people with predatory pricing into a massive overspend?
But the Feds are pretty solid on the whole "Aiding and and abetting terrorism is treason" thing.
TBF Google are big enough to display some spine if they'd "prefer not to" I'm pretty solid on that. So I flat out don't believe you when you say they'd prefer not to because if that were true, they wouldn't. Are you sure I'm wrong to think that way? Looks to me like they're quite happy to.
The Patriot Act doesn't work that way. Congress passed laws placing some reasonable minimum requirements on financial institutions that they have some sense of the actual identity of the people they are doing business with. Those are part of the "You're incorporated and do business in the financial sector" laws, so they're not a warrant kinda thing; a company's continued authorization to do business by law is contingent upon following them.
If you want to get into the philosophy of the Patriot Act, feel free, but if you're an individual US citizen, you have more power to change it than Google does. Google can push back, but that's a good way to get hauled in front of a Senate subcommittee or have their business practices curtailed unless the law itself changes.
Google could refuse and take that s&^t to the supreme court while funding the opponent of every politician who took them on for it. That's what they do to ensure they're not getting broken up like a monopoly should be, after all. But yeah the latter is their profit, the former is merely the right thing to do. They're motivated to enhance their profit at the expense of the country, which is what out sized market power /is/ and to ignore something where they might improve the country because there's no cash in it for them.
You kind of might want to revisit this part: "if you're an individual US citizen, you have more power to change it than Google does." Because it's clearly ridiculous on every level and comically so.
Did the law really change since these guys signed up with google or did google "overlook it" to sign them, then pull the switch later without any change in law between the two dates?
B) Google already has a PAC but PACs aren't votes, though the cynics would claim otherwise. If people want to repeal any part of the Patriot Act, they need to make it an issue. The kind of issue that shows up on the news.
C) It's hard to get purchase on changing this because know-your-customer is... Kind of a good requirement actually? Anonymity in business enables all manner of lawbreaking; sunlight is the best disinfectant, etc.
B) The WHOLE purpose of a PAC is to galvanise public opinion in ways that suit the donor and obfuscate and de-amplify things that don't. This incredible government overreach is not an issue for google's vast spending in washington. Why? Because it's your fault and if you cared you could move the needle and get it repealed? Seriously, come on. You can't fight google and if you try you'd need a google sized budget to garner support! There's cynical and there's wilfully naive and sometimes, maybe deliberately disingenuous.
> C) It's hard to get purchase on changing this because know-your-customer is... Kind of a good requirement actually? Anonymity in business enables all manner of lawbreaking; sunlight is the best disinfectant, etc.
Yet they skipped the requirements when signing up customers, deliberately, by policy. Lower friction. Get them in first for sure! "It's so easy, just go!" The bait. Then came the switch, the outages and "It's not our fault it's the law" The law which IS unconstitutional and which hasn't changed between the bait and the switch. Google support clearly sucks very, very hard and it's not easy at all.
I wonder if one day we'll look back at "cloud" and think, "what the actual fudge? How?"
B) You can't fight Google with that attitude, no. ;)
... but to be less flip: You're not fighting Google. Google isn't using its PAC to advocate for KYC law. So I'm not sure what you're arguing for here. If you think Google should change their behavior, the way to act on that opinion would be to... Organize Google users to put pressure on the company. So, power's back in our court again.
> I wonder if wone day we'll look back at "cloud" and think, "what the actual fudge? How?"
Oh, that I can help with. Maintaining one's own prem for anything that gets to be maintain-one's-own-prem-size is much worse. I've carried a pager for a cloud service, but I've never piled into the car at 2AM to drive half a state over to unlock my rented cage in a shared space with a physical link to a heavy-traffic backbone so I could power-cycle a machine. Nor have I put out a physical fire in a machine room, had a power supply blow up in my face, or driven up and down and up and down dirt roads to find the place where my fiber link is down because someone thought it'd be funny to take a shotgun to a junction box.
A lot of folks who ask "why Cloud" have nearly zero experience with what came before.
(TBF, for small businesses you can maybe handle your load with a few servers under a desk. Many people in this topic have explained how that goes though, and customers expect five-nines uptime these days.)
Best.
Even when they have issue over non payment, they wait at least 3 months, gives calls, sends follow up emails and for the actual shutdown they disable the network. the servers will continue to run for sometime. so if the payment is made it comes back without many issues.
Still, I'd bet even money Amazon does the messaging better than Google on something like that.
So I can fully understand people missing some "important" Google communication because of that.
Anything that is a countdown to doom should be a persistent banner on the app, reminding you of the importance and maybe even showing how many days left you have to deal with it.
I thanked them, hung up, and have since been slowly moving all my domains over to them.
I really appreciated that phone call.
But most people don't log in the Google cloud dashboard every day for fun.
For that reason, it isn't (at least, didn't used to be) blocking to roll a feature out for that pipeline to work, so a lot of things that should be top-banner messages weren't.
It's not that most CRUD apps could run on a single machine, it's that they probably shouldn't. Customers want redundancy, automated failover, etc. Datacenters are much better at offering these things than your office park.
- No one can see your code
- No one can shut you down
Are people spilling coffee on your servers at work?
Personally, I have found both Azure and AWS to be pretty responsive to Customer Service, although AWS can be quite blunt with their replies like when we tried to get them to authorise allowing us to use EC2 instances as mail relays. "No, use SES" was the long-and-short of it.
ymmv
Now they behave like this. I doubt even MS would have done a thing like this back in the day when they were the baddies.
Probably still face this same risk. You don’t own the hardware. You are just leasing it.
“Action required – Your AWS account is past due”
Common to any large company, if you want better service, support small business.
Same reason I've almost thrown out important communications from my credit card company; they send so much "important!" mail that's actually an offer to sign up for paid credit card monitoring it starts to be a spam flag.
have you actually checked this or are you just assuming it's the case?