Why you should not use Google Cloud
medium.com
medium.com
Not to mention the lack of visibility in changes - it seems like everything is constantly running at multiple versions that can change suddenly with no notice, and if that breaks your use case they don't really seem to know or care. It feels like there's miles of difference between the SRE book and how their cloud teams operate in practice.
I had an issue with my servers 2 days ago and I got a reply to my ticket within 1 minute. Follow-up replies were also very fast.
The person I was talking to was a system administrator who understood what I was talking about and could actually solve problems on the spot. He is actually the same person who answered my support requests last year. I don't know if that's a happy accident or if they try to keep the same support staff answering for the same clients. He was answering my requests consistently for 2 days this time.
I am not a big budget customer. AWS and GCP wouldn't think anything of me.
Thank you Vultr for supporting your product properly. And thanks Eric. You are very helpful!
Vultr looks like they provide more traditional services with a few extra niceties on top.
Within Google's infrastructure, I can deploy a new HTTPS REST endpoint with a .js file and 1 console command.
Could I set up an ecosystem on a Vultr VM to do the same? Sure, it isn't magic. But GCP's entire value prop is that they've already done a lot of that work for you, and as someone running a startup, I was able to go from "I have this idea" to "I have this REST endpoint" in couple of days, without worrying about managing infrastructure.
That said, articles like this always worry me. I've never seen an article that says "Wow Google's support team really helped out!"
There are ways to make the hosting relatively agnostic, but choosing a pub/sub solution (for example), that operates at 'web scale' will have a distinct impact on your solutions and force you into their ecosystem to maximize value. Why bother with BigCorps UltraResistant services if you're only going to use some small percentage of the capabilities?
I've made systems that abstract away the difference entirely, but I think the 'goldilocks zone' is abstracted core domain logic that will run on anything, and then going whole-hog on each individual environment. Accept that "cloud" is vendor lockin, and mitigate that threat at the level of deployment (multi-cloud, multi-stack), rather than just the application.
Downvotes can irritate, but there are at least two balms that don't involve adding noise to the threads. One is to remember that people sometimes simply misclick. The other is that unfairly downvoted comments mostly end up getting corrective upvotes from fair-minded community members (as happened with yours here).
Their sales staff is arrogant and has no idea how to sell into F500 type companies.
Source: 10+ meetings, with different clients, I attended where the Google sales pitch was basically "we are smarter than you, and you will succumb". The Borg approach. Someone needs to revamp the G sales and support approach if they want to grow in the cloud space.
The other funny thing is the package had a neoprene sleeve for a Chromebook. Eventually a sales person reached out via email assuming I owned a Chromebook and acted like I owed them a phone call because they gave me a neoprene sleeve I couldn’t use.
The entire package ended up going in the trash, which was an unfortunate waste of unrecyclable materials.
Well, that seems to be the approach at Google. Starting with hiring
Not surprising they end up with a hivemind that can't see past their mistakes.
And I've seen it cause them to lose at least 10 potentially good sales.
They have advantages but they're so arrogant that it puts people off.
It's more than 10 times or more people told me they prefer Google's solution to Microsoft or Amazon's but they're going with a competitor because they can't stand Google's arrogant attitude. It's close to laughable because of throwing money away just because they won't back off.
It blows my mind that GCloud, with arguably superior tech and performance compared to AWS/Azure, can't handle support. I have my own horror stories from 2 years ago, but still they haven't fixed it.
Google just doesn't seem to be able to focus on products that requires service and customer support. Maybe they just don't care about it while they have an infinite revenue stream from search and advertising. Whatever it is, they should be humiliated.
I love the tech, and the ux details like in browser SSH (AWS hasn't improved UX EVER) but they can't get support right? Amazing.
That's literally any product that people pay for (instead of viewing ads).
Customer support isn't and never has been in their DNA. It's often rage-inducing how hard it is to contact a human at Google.
They seem to think they can engineer products that don't need humans behind them.
Anecdotally: I've been an MS gold partner in a bunch of different contexts for years. The experiences I had as 'small fish' techie with AWS were on par or better. YMMV, of course, but I'd be more comfortable putting my Enterprise in the hands of AWS support than MS's (despite MS being really good in that space).
Prices seem high though: https://cloud.google.com/support/?options=premium-support#op...
So maybe think about which hosting provider to go with, don't get me wrong I like their tech. But their moderation does need a more human element, to be frank all their products do. Simply ceding control to algorithmic judgement just won't work in the short term if ever at all.
Still works out cheaper for workloads than AWS does even factoring staff in at this point.
AWS always turns into cost and administrative chaos as well unless it is tightly controlled which in itself is costly and difficult the moment you have more than one actor. GCP probably the same but I have no experience with that. Very much more difficult to do this when you have physical constraints.
Two man startup, perhaps but I think the transition should go:
VPS (linode etc) for MVP, colo half rack, active/active racks two sites then scale out however your workload requires.
One of the failure modes I see a lot is failing to factor in latency in distributed systems. Mainly because most systems don’t benefit at all from distribution and do benefit from simplification.
The assumption on here is that a product is going to service GitHub or stackoverflow class loads at least, but literally most aren’t. Even high profile sites and web applications I have worked on tend to run on much smaller workloads than people expect. Latency optimisation by flattening distribution and consolidating has higher benefits than adopting fleet management in the mid term of a product.
Kubernetes is one of those things you pick when you need it not before you need it. And then only if you can afford to burn time and money on it with a guaranteed ROI.
Basically, the LB decides which kubernetes cluster will serve your request and once you're in a k8s cluster, you stay there.
You don't have the control-plane that the federation provides and a bit of overhead managing clusters independently, but we have automated the majority of the process. On the other hand, debugging is way easier and we don't suffer from weird latencies between clusters (weird because sometimes a request will go to a different cluster without any apparent reason <-- I'm sure there's one, but none that you could see/expect, hence debugging).
My people's time is more important than your complex system.
If you run into an issue, send me a note and I will get someone to reply to your issue.
Maybe support just needed to satisfy their quota of kicked out companies for the month, who knows?
Can you elaborate on that? What do they monitor with the moderation bot?
Imagine you're running a cosplay community, and all of a sudden all your content is being deleted because the SESTA/FOSTA bill gets passed in a country where your "cloud" happens to reside in: https://hardware.slashdot.org/story/18/03/25/0614209/sex-wor...
"told us straight out that it would and we should move"
Sounds shady. I bet this would make more sense if OP explained what his company actually does.
I assumed they could tell that via CPU usage with they already monitor for quotas.
Good times.
By chance are you located out of the United States? These are not downtime issues, but anti-fraud prevention and finance issues.
Cryptocurrency?
The nature of the tech in question seems important in this story.
I hope there is a possibility to put a backup contact person / credit card so organisations can deal with people going on vacation or being sick or whatever.
IMHO this should be nicely documented as any other technical material you get to learn about the cloud product when you create an account (e.g. important steps to ensure your account remains open even in case of important security breaches, yadda yadda it's possible we'll need a way to prove that you are you yadda yadda, this can happen when yadda yadda, be prepared, do yadda yadda)
Erm ... no, evidently not?
A prudent person might consider a cloud provider to be a domain of failure and choose a multi-cloud option, which would probably be the correct way to address this resiliency issue. However, that's not really an appropriate approach for an early stage startup, where availability is generally not that much of a concern.
Is an exploding car safe because it is built by an early stage startup?
Just because you decide that implementing resiliency isn't a good business decision for some early stage startup, doesn't magically make the product resilient, it just isn't and that may be OK.
There are many options to choose from for implementing resiliency, it could be having multiple providers concurrently, it could be having a plan for restoring service with a different provider in case one provider fails, it could be by setting up a contract with a sufficiently solvent provider that they pay for your damages if they fail to implement the resiliency that you need, whatever. But if you fail to consider an obvious failure mode of a central component of your system in your planning, then you are obviously not building a resilient system.
Edit: One more thing:
> There's no indication anywhere from GCP themselves that a project could be a domain of failure. If asked, I doubt they would consider it as such.
Then you are asking wrong, which still is your failure if you are responsible for designing a resilient system.
If you ask them "Is a complete project expected to fail at once?", of course they will say "no".
That's why you ask them "Will you pay me 10 million bucks if my complete project goes offline with less than one month advance warning?", and you can be sure you will get the response to the problem that you are actually trying to solve.
If you replace "multi-cloud" with "multi-datacenter" (in the pre-cloud days), this premise is fairly unassailable. In those same days, applying it to "multi-ISP", it becomes more arguable.
Today, though, the incremental cost (money and cognitive) of the multi-cloud solution, even for an early startup, doesn't seem like it would be high enough to make the notion downright inappropriate to consider.
I'd even argue that if a cloud provider makes the lock-in so attractive or multi-cloud so difficult that that's a sign not to depend on those exclusive services.
The economics don’t work out if you are trying to do this with just vanilla VMs across AWS, GCP and Azure and managing yourself. You either do it the old fashioned way renting rack space and putting your own kit in, or you make full use of the managed services at which point - by design - you are locked in.
Usually this can get handled after a few days of aggravating emails back and forth, we get our client to ban the affiliate in question, and move on with our days with no downtime. But a few weeks ago my coworker came in to find our server taken offline, because AWS emailed him about a spam complaint on a Friday night, and they hadn't gotten a response by Sunday. It'd been down for hours before he realized.
They'd just null terminated the IP of the server, so he updated IPs in DNS real quick, but he then spent half a day both resolving the complaint, and then getting someone at AWS to say it wouldn't happen again. They supposedly put a flag on his account requiring upper management approval to disable something again, but we'll see if that works when it comes up again.
If and when you do, give serious consideration to how you handle DNS.
For that, you should consider setting up multiple accounts to isolate those services from the portable ones.
Ansible helps deploy software, but deploying software is the smallest problem of going multi-cloud.
Disclosure: I'm one of the founders.
See: Route53 and Dyn outages in the past couple years.
Forgive my ignorance but that seems like a weird choice rather than cutting access to the servers or in some more formal ways for copyright...
Also kinda concernit that multiple departments can take enforcement type action and others not know it. That seems way disorganized / recipe for diasater.
Why not Azure? They have a solid platform and (at least for a MSFT partner) their support is top-notch.
Yet their support didn't ever solve a problem within their SLA's and sometimes critical level tickets were hanging for months.
Plus my impression is that whereas AWS (and possibly Google) clouds are built by engineers using best practices and logic, Azure products felt always very much marketing driven e.g. marketing gave engineering a list of features to launch and engineering did the minimum effort possible to have the corresponding box ticked. I absolutely hated working on Azure and now won't accept any contract on it.
Documentation is horrible or non-existing, things just don't work, have weird limitations or transient deployment errors, super weird architectural and implementation choices + you never escape the clunkyness of the MS legacy with for example AD.
What does this mean?
This is not true at all. Once you start spending real money on GCP or AWS, they will reach out to you. You will probably sign a support contract and have an account manager at that point. Or you might go with enterprise support where you have dedicated technical assets within the company that can help with case escalation, architecture review, billing optimization, etc.
> What if the card holder is on leave and is unreachable for three days? We would have lost everything — years of work — millions of dollars in lost revenue.
Cards get skimmed all the time. When a card gets skimmed, the issuer informs everyone who is making recurring purchases with that card "Hey, this card was skimmed, it's dead".
If someone has a recurring charge attached to that account, the recurring charge will go bad. If this is an appreciable number of cloud services which are billed by the second, this can happen very, very quickly and without you knowing. Remember, sometimes the issuer informs you that the card was skimmed, which you will receive after all the automated systems have been told.
So, the cloud provider gets the cancel, and terminates the card. It then looks around sees the recurring charge, takes a look at your servers racking up $$ they can't recoup and the system goes "we don't know this person, they buy stuff from us, but we haven't analysed their credit. Are they good for the debt? We've never given them credit before. Better cut them off until they get in touch."
If only they had signed an enterprise agreement and gotten credit terms. It could still be paid with a credit card, but the supplier would say "They're good for $X, let it ride and tell them they'll be cut off soon". They can even attach multiple methods of payment to the account, where, for example, a second card with a different bank is used as a backup. Having a single card is a single point of failure in the system!
In closing, imagine you're a cryptocoin miner who uses stolen cards to mine on cloud services. What does that look like to the cloud provider?
Yep, someone signs up for cloud services, starts racking up large bills and then the card is flagged as stolen.
Would be interested to see what would've happened if they would've used a business account.
We're a health-care startup, and this past Saturday I got an email saying that due to the nature of our business, we were prohibited from using their payment platform (Credit Card companies have different risk profiles and charge accordingly--see Patereon v Adult Content Creators).
Rather than pull the plug immediately, they offered us a 5-day wind down period, and provided information on a competitor that takes on high-risk services.
Fortunately, the classification of our business was incorrect (we do not offer treatment nor perscription/pharma services), and after contacting their support via Email & Twitter, we resolved the issue in less than 24-hours.
So major kudos to Stripe for protecting their platform, WHILE also trying to do the right thing for the customers who run astray from the service agreement.
This seems like a billing issue. If they had offline billing and monthly invoicing (enterprise agreement) I do not believe this issue would have happened.
If you are running an enterprise business and do not have enterprise support and an enterprise relationship with the provider, you may be doing something wrong on your end. It sounds like the author of this post does not have an account team and hasn't take the appropriate steps to establish an enterprise relationship with their provider. They are running a consumer account which is fine in many many cases, but may not be fine for a company that requires absolutely no service interruptions.
IMO, the time this issue was resolved by the automated process (20 mins) is not too bad for consumer cloud services. Most likely this issue could have been avoided if the customer had an enterprise relationship (offline billing/invoicing, support, TAM, account flagging, etc, etc) with Google Cloud.
Paying bills offline via invoice establishes a enterprise agreement with cloud providers. It does in fact change everything with the issue discussed here. They wouldn't be taken offline due to an issue with the credit card payment.
We have significant spend on CC with GCP and we’re not “consumers”. Our account manager has no issue with this. If they did we’d move somewhere else.
> What if the card holder is on leave and is unreachable for three days? We would have lost everything — years of work — millions of dollars in lost revenue.
You should never be in this position. If this were to happen to us we would be able to create a new project with a different payment instrument, and provision it from the ground up with terraform, puppet and helm scripts. The only thing we would have to fix up manually are some DNS records and we could probably have everything back up in a few hours. Eventually when we have moved all of our services to k8s I would expect to be able to do this even on a different cloud provider if that were necessary.
This is one of the reasons I always implore people to have a backup of their data with another provider or at least under a different account. That protects against all kinds of accidents but also against malice.
If OP happened to me, sure yes I could have my entire infra on AWS/Azure/whatever else Terraform supports in an hour, maybe more to replace some of the tiny cloud specific features we use. But if it takes me a day to me to just move the data into Azure, thats an entire lost business day of productivity.
It's a simple truth that even if you are at the millions of dollars point, there is a data size at which you are basically all-in with whatever solution you've chosen, and having a secondary site even for a billion dollar company can be exceptionally difficult and cost prohibitive to move that sort of data around, again especially when you're heavily dependent on a specific service provider.
Yes, the blame in part lies with making the decision to rely on such a provider. At the same time, there are compelling arguments for using an existing infrastructure instead of working on the upkeep of your own for data and compute time at that scale. Redundancy is built into such infrastructures, and perhaps it should take a little more evidence for the provider to decide to kill access to everything without hard and reviewed evidence.
They suspect you are a fraudster because you are using a stolen card.
Either sign a proper SLA agreement with Google (which gives you 30 days to pay their bills by any form, and therefore you get 30 days notice before they pull the plug), or have two forms of payment on file. Preferably, don't use your GCP credit card at dodgy online retailers too...
While you make sense from Google's PoV, it doesn't from the customer's PoV. As google is a big corp, it's IMHO better to side with the customer here, as next time it might be you who's getting screwed over by Google/other corp.
Or at gas stations, ATMs, or any other place where someone can install a skimmer.
How do you do that?
It seemed very Kafkaesque to me, getting tried and convicted without any mention of the crime or charge. I think the author is justified in his disapproval.
The worst part is that when we did post morterm and asked Google why the support resolution was so slow despite being "the privileged" customer, their answer was that the P1 SLA was only to respond within 15 minutes there is no SLA for resolution. Most of the "response" that were getting was that a new support guy has taken over in a new time zone which is the most useless information for us.
We are seriously thinking of moving to another cloud vendor.
AWS would never admit that anything is wrong from their side.
Grace periods that respect your business should be a standard that all service providers hold themselves to
As far as I know, Mozilla has no business relationship with extension developers, so I would actually be very concerned if their first action wasn't to cut you off.
There is nothing dodgy about the extension. Mozilla was just being ridiculous.
And had they converted their project to monthly invoicing: https://cloud.google.com/billing/docs/how-to/invoiced-billin...
* When you have invoicing setup, the above shouldn't happen. You need to keep a payment method in good standing, but you have something like 10 days to pay your bill. -- They do a little bit more vetting (KYC) on the invoice path, and that effectively gets you out of dodge.
* Without paying for premium support, there's effectively no support.
I think if someone didn't pay their bill on time, you might shut off their service too, wouldn't you?
What does that have to do with anything? The account was not shut down for non-payment, it was shut down because of unspecified "suspicious activity."
But even in case of non-payment I would not shut down the account without any warning. Not if I wanted to keep my customers.
Then they unplug the Ethernet cable and wait a week or two.
But as you said, this isn’t about non-payment.
A hypothetical but not unheard of scenario in which immediate shutdown might be warranted.
It's a rough world and different providers have optimised for different threat models. AWS wants to keep customers hooked; GCP wants to prevent abuse, Digital Ocean wants to show it's as capable as anyone else.
If you can afford it, you build resilient multicloud infrastructure. If you can't yet do that; at the very least ensure that you have off-site backups of critical data. Cloud providers are not magic; they can fail in bizarre ways that are difficult to remedy. If you value your company you will ensure that your eggs are replicated to more than one basket and you will test your failover operations regularly. Having every deploy include failing over from one provider to another may or may not fit your comfort level; but it can be done.
Not without warning, no. It is possible that the customer intended to start a CPU-intensive process and fully intended to pay for it.
Send a warning first with a specific description of the "suspicious activity" and give the customer a chance to do something about it. Don't just pull the plug with no warning.
Yes, there's nothing wrong with that. You have their credit card and can even authorize certain amounts ahead of time to make sure it can be charged.
There's a degree of complexity that comes with multi-cloud that's ill-suited for most early stage companies. Especially in the age of "serverless" that has folks thinking they don't need people to worry about infrastructure.
My point is that the calculus has more to it than just money. The prudent response, of course, is to do as you described. Have a plan for your provider to go away.
Offsite backups and the necessary config management to bring up similar infra in another region/provider is likely sufficient for most.
Perhaps we'll start seeing a new crop of post-mortems from the "fail fast" type of startups failing due to cloud over-dependency issues. They're (presumably rare) edge cases, but easily fatal to an early enough startup.
I just heard a dozen founders sit up and think "Market Opportunity" in glowing letters.
CockroachDB has a strong offering.
But multi-cloud need not be complicated in implementation.
A few ansible scripts and some fancy footwork with static filesystem synchronization and you too can be moving services from place to place with a clear chain of data custody.
And if it was that critical it should have support and a SLA contract, and you know, backups.
The costs of Google may be comparable or lower than other services, but they don't seem to get that risk is a cost. Risk can be your biggest cost. And they've amplified that risk unnecessarily and shifted it to the customer. Fatal, as I said.
You might never see this happen to your GCP account in it’s lifetime.
But even if you assign blame to the OP for not expecting this, it doesn't look good, because the lesson here is "you shouldn't use google and if you do, expect them to fuck you over, for no reason, at any time".
It reminds me of AWS's opacity-as-antidote-to-worry with respect to hardware failures. If the underlying hardware fails, the EC2 instance on it just disappears (I've heard GCP handles this better, and AWS might now, as well). I like to point out that this doesn't differ much from the situation of running physical hardware (while ignoring the hardware monitoring), both from a "worry" burden perspective and from a "downtime from hardware failure" perspective.
We've gone through several account teams of our own that seem to be eager to help only to turn into radio silence once we actually need something. We have already moved mission-critical services to AWS and Azure, with GCP only running K8S and VMs for better pricing and performance.
GCP has good leadership now but it's clearly taking longer than it should to improve unfortunately.
It may not be expensive to Google to lose the business, but it's very expensive for the customer. Google's offering is now non-competitive if you aren't doing things at enterprise scale. Of course many of Google's best clients will start out as small ones. The metrics won't capture the long-term repetitional damage that's being done by these policies.
Maybe for critical accounts you need to have a backup visa on file with Google cloud with in case the first dies for security reasons.
A single visa is a single point of failure in an otherwise redundant systemm
After reading this article, I am probably going to do this.
https://cloud.google.com/support/?options=premium-support#op...
Paid support tiers (as far as I know) are for deeper system level diagnostics.
You heard me right, I prepaid them so that my account can be restored.
I miss AWS sometimes...
I'm curious though, did you have multiple credit cards on file with your Google billing account? I'm under the impression that this is part of their intended strategy for avoiding service interruption, but I'd like to know if it actually works that way.
(I took this as a reminder to add a second card to my account)
You should always have at least two payment methods on file with them for anything important.
That way if one gets flagged for fraud, services won't be suspended.
If something like this ever happens in AWS, they email you, call you, give you a grace period, and generally, do their best to avoid affecting your infrastructure.
GCP is getting better, but it's not ready for anything other than science fair experiments.
If you're doing something this mission critical, you have to have an SLA with your cloud provider.
I've been running three providers as peers (DO, Linode, Vultr) as a one-man shop for years, and I sleep better at night knowing that no one intern can fatfinger code that takes me offline.
Ignoring tweaks for international customers, each proxy is in an A record round-robin with health checks via Route 53. US-based requests get forwarded to one of the pods, and the proxy either handles the request with local servers, or points to servers in another pod if it has servers that are down. If any pod has a power outage or goes down for any reason, Route 53 automatically pulls the entire pod out of the rotation. If an entire provider goes dark, all of the pods get pulled out of the rotation, but all the others keep running.
Then they really escheated my earning to some a nation government after 30 days. However when I requested them for the escheatment ID so that I can contact that nation government to find my money back, they said they don't have the ID! They eascheated my money without any recording! Which is almost the same as throwing my money in ocean.
Last year my team migrated most of our infrastructure out of AWS into GCP. We'd been running k8s with kops in AWS and really liked using GKE. We also developed a bizdev arrangement.
As I was scaling up my spend in GCP, I began purchasing committed use discounts. Roughly similar to reserved instances in AWS. I'd already made one purchase for around $5k a month and these are tied to a specific region and can't be moved. I went to purchase a second $5k block, and typo'd the request ending up with $5k worth of infra in us-central rather than us-west. The purchase doesn't go into effect until the following day and showed as "pending" in the console. No big deal, I thought, I'll just contact support and they'll fix it right away, I'm sure. I had this preconceived notion based on my experiences with AWS. I open a support request and about an hour later I get a response that basically tells me that once I've clicked the thing, there's no undoing it and have a nice day.
I've literally just erroneously spent $5k for infra in us-central that I can't use and their response was basically, "tough". $5k is a sufficiently large loss that I'd be inclined to burn a few hours of my legal team's time dealing with this issue, something I shared with the support person. After much hemming and hawing over the course of a few days, they eventually fixed the issue.
More recently, I've been dealing with an issue that is apparently called a "stockout".
Unlike AWS, GCP does not randomize their AZ names for each customer. This means that for any given region, the "A" availability zone is where most of the new infrastructure is going to land by default. Some time in May, we started seeing resource requests for persistent SSD failing in us-west1-a. The assumption is that it would clear up pretty quickly after an hour or two, but persisted. After about a day of this, we opened up a support case asking what was going on and explaining the need for some kind of visibility or metrics for this kind of issue. The response we received was that this issue was "exceedingly rare" which was why there was no visibility and that it would be rectified shortly, but couldn't be given any specific timeline.
I followed up with my account rep, he escalated to a "customer engineer" who read the support engineers notes and elaborated how "rare" this event was and how unlikely it was to recur. Again, I contacted my account rep and explained my unhappiness with the qualitative responses from "engineers" and that I needed quite a bit more information on which to act. He was sympathetic throughout this whole process escalating inside the organization as needed and shared with me some fairly candid information that I couldn't get from anyone else.
Apparently, the issue is called "stockout" and us-west1-a had a long history of them. The issues had been getting progressively worse from the beginning of the year and at this time, this AZ was the most notorious of any in any region. Basically, the support engineer either patently lied or was just making stuff up. Also, I shared with my GCP rep how AWS load-balances across AZs. He promised to pass that along.
The moral of the story is that if you want to be in us-west, then maybe try us-west1-c. Also, GCP is a relatively young arm of a well-established company that has a terrible reputation of being able to communicate with consumers. They'll eventually figure it out, it will just take some time.
Just the way it is unless it is nuclear plant probably.
Total clown show.
Take this as a lesson in risk management and don't fuck it up again.
First, we sincerely apologize for the inconvenience caused by this issue. Protecting our customers and systems is top priority. This incident is a good example where we didn’t do a good job. We know we must do better. And to answer OP’s final message: GCP Team is listening.
Our team has been in touch with OP over what happened and will continue digging into the issue. We will be doing a full review in the coming days and make improvements not only to detection but to communications for when these incidents do occur.
Feel free to budget that into your choices on where to host, but getting into any arrangement where you rely on whatever free tier of support is nonsensical once you’re making any kind of money.
We’re not a big Google customer, couple thousand a month, but when we migrated we instantly reached out to account reps and have regular quarterly check ins.
Even still, I feel like it's still reasonably likely that the robot would shut down a project for "suspicious activity".
I'm not sure if it's the culture of secrecy or sheer cluelessness, but it's pretty bad.
I'm not trying to slam them either, I still really like their products and use them.
Enterprise Google support: one day after your stuff goes down, you get a phone call saying the algorithms decided you're fraudulent.
I do, but that wasn't my point.
from below: I work here in Google Cloud Platform Support. First, we sincerely apologize for the inconvenience caused by this issue. Protecting our customers and systems is top priority. This incident is a good example where we didn’t do a good job. We know we must do better. And to answer OP’s final message: GCP Team is listening.
Our team has been in touch with OP over what happened and will continue digging into the issue. We will be doing a full review in the coming days and make improvements not only to detection but to communications for when these incidents do occur.
Stunning that Google cloud can get away with this.
What is the activity which got flagged as suspicious?
Luckily, with AWS and Azure there is great competition to move to.
They provide no explanation as to why the accounts are closed, and provide only a single form to appeal the situation.
In my case they refused to release the locks on their services so all information, all contacts, all files, all histories, all YouTube data, and everything else they store is now effectively lost with no means of getting the data back.
This was done without warning, and the account was locked due to "suspicious activity on YouTube".
Through Experimentation with a new account I found that the "suspicious activity" they were referring to was Criticism of the Trump Administration policy of kidnapping children from their parents.
Posting such criticism to the threads that follow stories by MSNBC and other news sources triggered Google to block YouTube and all other services they provide and to do so without warning or any explanation.
If your stuff relies on "cloud" you are on your own.
I personally prefer using their API and tools over AWS. They seem more sensible. Azure is horrible.
GCP offers inter-region networking and more sensible firewalls and routers.
Pricing is pretty comparable to AWS, slightly better for instances, slightly worse for storage.
Storage performance for persistent SSD seems a little better than EBS GP2.
Cheap local disks on any kind of instance.
I, for one, wouldn't ever want to operate a firewall with enough rules where I'd have to be concerned about such a thing. Eek.
For just VMs with the fastest CPU, storage, networking and the easiest billing compared to other major clouds, GCP wins.
"We have temporarily closed your Alibaba Cloud account due to suspicious activity. Please provide the following information within [24 hours] by email to compliance_support@aliyun.com in order to reopen your account: ... If you fail to provide this information your account will be permanently closed, and we may take other appropriate and lawful measurers. Best regards, Alibaba Cloud Customer Service Center "
I provided the documents in ~30 hours because that's when I saw the email. There was no further communication from Alibaba. I assumed everything is ok, but in 2 weeks my account was terminated.
[1] https://landing.google.com/sre/book.html [2] https://landing.google.com/sre/
Okay that sounds like a greater systemic problem.
You should be have been able to deploy your git repository on another system pretty quickly, as well as have your own backups of your database.
The most time consuming thing should be setting up the environment variables.
Let me see, what else would be tricky: if you are using google analytics that data might be gone, but your other metrics package should have had many snapshots of that data too
The only way to use their product safely is to engineer your entire business so that cloud providers are completely interchangeable.
Forcing the entire industry to pay the cost of transparently switching upfront completely commoditizes cloud providers, which means they’ll no longer be able to charge a sustainable markup for their offerings.
This is fiscally negligent. Upper management should be fired.
However, it’s great for the rest of the industry — Google nukes a few random startups from orbit, some VCs take a bath, early and mid-range adopters bleed money engineering open source workarounds, and everone else’s cloud costs drop to the marginal costs of electricity and silicon.
the shareholders should be proud that such naivete towards vendor lock is still rampant