Removing data transfer fees when moving off Google Cloud
cloud.google.com
cloud.google.com
Which highlighted "Egress fees harm competition by creating barriers to switching and multi-cloud leading to cloud service providers entrenching their position" [2].
It's also interesting that they are calling out problems with software licensing, as that is another thing the CMA is investigating in their cloud market review.
[1] https://www.gov.uk/cma-cases/cloud-services-market-investiga...
[2] https://assets.publishing.service.gov.uk/media/652e958b69726...
as expected the hyperscalers refuse to acknowledge that the free ingress and expensive egress is a lock-in mechanism, and their smaller competitors complain bitterly about this
the hyperscalers say they have to charge egress fees to pay for the costs in building their networks, but for some reason doesn't apply to ingress (which they're silent on)
if they want to play this game then the CMA should simply make them charge the same for ingress and egress
that way they can "fund their network costs" without issue, and if they want to make them both free then that's their decision
This doesn’t pass the red face test IMO. The hyperscaler networks are indeed very expensive, but that’s because they need to provide non-blocking or near non-blocking performance within the availability zone, and the clouds don’t charge for this service.
The Internet egress part ought to be straightforward on top of this: plug as much bandwidth worth of connections into the aforementioned extremely fancy internal network. Configure routes accordingly.
It’s worth noting that the big clouds will sell you private links to your own facilities, and they charge for these links (which makes sense), but then they charge you for the traffic you send from inside the cloud to these links, which is absurd since they don’t charge for comparable traffic from inside the cloud to other systems in the same AZ.
I see you've never tried GCP.
FWIW, Google has been working on these fancy nonblocking networks for a very long time. They’re very proud of them. Maybe they don’t actually use them for GCP, but Google definitely cares about network performance for their own purposes.
Your message seems to imply that these datacenter networks experience very little loss, and this is observably far from reality. In GCP you will observe levels of frame drops that a corporate datacenter architect would consider catastrophic.
What google decided long ago is that for their traffic patterns, it makes the most sense to adopt clos-like topologies (with packet switching in most cases), and not attempt to make a fully non-blocking single crossbar switch (it's too expensive for the port counts). More hops, but no blocking.
Scaling that got very difficult and so now many systems at Google use physical mirrors to establish circuit-like paths for packet-like flows.
GCP is effectively an application that runs on top of google's infrastructure (I believe you already worked there and are likely to know how it works) that adds all sorts of extra details to the networking stack. For some time the network as a user-space Java application that had no end of performance problems.
> With ethernet you put a frame on the wire and hope.
This is not really true. With Ethernet, applications and network stacks (the usual kind — see below) ought to do their best to control congestion, and, subject to congestion control, they put frames on the wire and hope. But network operators read specs and choose and configure hardware to achieve a given level of performance, and they expect their hardware to perform as specified.
But increasingly you can get guaranteed performance on Ethernet even outside a fully non-blocking context or even performance exceeding merely “non-blocking”. You are fairly likely to have been on an airplane with controls over Ethernet. Well, at least something with a strong resemblance to Ethernet:
https://en.m.wikipedia.org/wiki/Avionics_Full-Duplex_Switche...
There are increasing efforts to operate safety critical industrial systems over Ethernet. I recall seeing a system to allow electrical stations to reliably open relays controlled over Ethernet. Those frames are not at all fire-and-hope — unless there is an actual failure, they arrive, and the networks are carefully arranged so that they will still arrive even if any single piece of hardware along the way fails completely.
Here’s a rather less safety critical example of better-than-transmit-and-hope performance over genuine Ethernet:
https://en.m.wikipedia.org/wiki/Audio_Video_Bridging
(Although I find AVB rather bizarre. Unless you need extremely tight latency control, Dirac seems just fine, and Dirac doesn’t need any of the fancy switch features that AVB wants. Audio has both low bandwidth and quite loose latency requirements compared to the speed of modern networks.)
Exactly. Network endpoints infer things about how to behave optimally. Then they put their frame on the wire and hope. The things that make it possible to use those networks at high load ratios are in the smart endpoints: pacing, entropy, flow rate control, etc. It has nothing at all to do with the network itself. The network is not gold plated, it's very basic.
A ratio has a numerator and a denominator. I can run some fancy software and get (data rate / nominal network bandwidth) to be some value. But the denominator is a function of the network. I can get a few tens of 10Gbps links all connected together with a non-gold-plated nonblocking switch that’s fairly inexpensive (and even cheaper in the secondary market!), and each node can get, say, 50% of 10Gbps out as long as no node gets more than, say, 50% of 10Gbps in. That’s the load ratio.
Or I can build a non-gold-plated network where each rack is like this and each rack has a 100Gbps uplink to a rather more expensive big switch for the whole pile of racks, and it works until I run out of ports on that big switch, as long as each rack doesn’t try to exceed the load ratio times 100Gbps in aggregate. Maybe this is okay for some use case, but maybe not. Netflix would certainly not be happy with this for their content streaming nodes.
But this all starts to break down a bit with really large networks, because no one makes switches with enough ports. So you can build a gold plated network that actually gets each node its full 10Gbps, or you can choose not to. Regardless, this isn’t cheap at AWS scale. (But it may well be cheap at AWS scale, amortized per node.)
And my point is that egress does not add meaningful cost here. A 10Gbps egress link costs exactly as much, in internal network terms, as a compute or object storage node with a 10Gbps link. For an egress node, it costs whatever 10Gbps of egress transit or peering costs plus the amortized internal network cost, and a compute node costs whatever the cost of the node, space, power, cooling, maintenance etc costs, plus that internal network link.
So I think there is no legitimate justification for charging drastically more for egress than for internal bandwidth, especially if the cost is blamed on the network cost within the cloud. And the costs of actual egress in a non-hyperscaler facility aren’t particularly secret, and they are vastly less than what the major clouds charge.
if the world was bellhead, ATM would have won.
The industry standard for peering is paying the 95th percentile of egress or ingress depending on whichever is greater. Ingress is free for these clouds because egress > ingress overall.
I've seen this at other providers. We did a competitive pricing exercise at my last company, and our overall cost went down, but the mechanism was per hosts costs went down significantly and egress costs went up significantly, and the per host cost decrease outweighed the egress cost increase.
It still doesn't make sense to charge for ingress, because everybody knows that should be free, unless you're a residential ISP.
bandwidth from cogent at whatever colo you can cross connect to them is wildly cheaper than "i need bandwith to everywhere" bandwidth.
Pfft...
> for peering is paying the 95th percentile of egress or ingress depending on whichever is greater. Ingress is free for these clouds because egress > ingress overall
How about customers pay for actual usage, rather than some [fake] averaged-across-all-customers usage?
It seems entirely reasonable to look more skeptically at cloud providers' exact charges vs cost for egress, particularly when high egress fees might contribute to lock-in, and when the public price sheet vs the preferred customer pricing might differ radically. But asking them to totally restructure the charges, inventing a charge for ingress when their actual total ingress cost is zero and, short of major industry-wide usage pattern changes, will remain zero? Why would you do that?
because ingress traffic volume is a fraction (in my experience, in website hosting of a well known household brand, barely 1%!) of egress traffic volume, and most peering connections are 1:1 in ingress/egress bandwidth so the egress bandwidth cost sets the price.
Many people have a x00 Mbps or even x Gbps downstream, but most have no more than x0 Mbps upstream. Literally their ability to pull traffic from websites is 50X in some cases than to push information out. Going beyond that (greater uploads) often costs significantly more.
Whether or not these two are actually related isn't clear to me, but it is interesting.
This is like Comcast giving you the initial installation for free but charging you an arm, a leg, and a year of your life to allow you to leave because "the process is costly". Both practices are scummy and abusive towards the customer.
This seems to me more of a "try us out for free" play. Bring your big data here, if you end up not liking it, we won't penalize you for taking your data out. Given that GCP is running at a very distant 3rd, they need to make plays like this.
If it doesn't work - GCP looks a little better compared to AWS
If it does work, AWS users will have an easier time extricating themselves from the platform, and possibly going to GCP.
If they remove the fees, then competition might be pressured to do so (as a marketing response). Thus making it easier for people to switch to Google.
If this move were really about acting in customers' best interests, they would reduce the fees for everyone. Doing this only for departing customers feels performative.
Does it lose files? Fails to write but gives an error? Fails to read sometimes? Silently fails?
But seriously, it seems like a fair-ish compromise. The 60 day limit is tough though. As long as you’re continuing your usage and not using offsite backups, ingress/egress isn’t probably too problematic. It’s only problematic when you try to suddenly egress all data you’ve ever stored, which you probably wouldn’t do unless you’re migrating away.
It's problematic to me. I run a system partly on GCP and partly on another provider. I'd like to move some of the stuff that's on the other provider over to GCP VMs, but I am prevented from doing so solely by the GCP egress fees.
My general complaint is that high egress fees prevent users from developing hybrid cloud solutions which mix components from different cloud vendors. Instead, you are forced to choose a platform, and then you're locked into it. That seems textbook anti-competitive. We shouldn't be grateful for the opportunity to switch vendors, we should be angry that "choosing a single cloud vendor" is a thing at all.
They can't really shut Google Cloud down and still be charging people to exit.
And if they had suddenly seen the light on egress fees then surely they would have cut egress fees everywhere..... the fact that it's only on account closure is kinda suspicious.
I'm not quite sure about this. Obviously they don't do it now, but I wouldn't have put it beyond them.
All over the Internet people are slapping google on the back for..... pretty much nothing. In fact much of the praise seems to be worded as though Google had dropped all egress fees.
Google continues to charge the nothing-short-of-highway-robbery 12 cents per gigabyte unless you are in Australia in which case its 19 cents per gigabyte. This is astoundingly bad value.
What are people hailing Google's "free to exit" in such glowing terms. Even herein the comments people are cheering for Google.
Worth noting at this point that Cloudflare R2 charges 0 cents per gigabyte egress.
I like to see them publicly call out Microsoft and Oracle.
I've worked at some orgs where to either move their data out of S3 would cost $20k, or to even delete it would cost thousands in API calls.
raises eyebrow
It's a well-known trick (proposed by AWS Support as well) to set S3 lifecycle rules to empty buckets with too many objects to cycle through via List calls. Doesn't cost anything.
That, and any of the smaller “clouds” with egress/delete fees that need to be considered when leaving. Seems massively disingenuous given that until this announcement they also had such fees (“look, those people try rip you just like we were doing until five minutes ago!”) but that is pretty standard for marketing materials.
This makes them a better option as the first cloud provider to try, other things being equal, because leaving (back to on-prem or to another provider) is easier. I assume they are trying to remove a little of the huge the distance between them and the two leading players by reducing concerns that might add on-boarding friction.
Google should have made the same announcement without the snarky mean-spirited bitterness.
https://cloud.google.com/exit-cloud
* Free data transfers related to Google Cloud Exit are available on Premium Tier Network Service Tier
* Only data residing in Google Cloud data storage and data management products are covered
* You must report any changes to your migration timeline set out in your request form to the Google Cloud Support team
* You must submit your free data transfer request prior to the termination of your Google Cloud agreement
* Google Cloud reserves the right to audit movement of customers' data away from Google Cloud for compliance with program terms and conditions
Is requiring a Google committee review/application process a new trend with Google products? I recently was denied on another application through Google for API access to get one businesses GMB reviews, and it's frustrating because there's no recourse. Google is so opaque now.
Let's just hope this isn't step 1 for their plan to do exactly that.
*piecemeal
I still feel that the egress fees charged by the big 3 are way too high.
Why not:
1. Migrate your data out.
2. Close your account.
3. You will automatically be refunded all network egress fees incurred in the final 60 days, capped at the number of gigabytes you had stored in our products in the preceding 60 days.
The TLDR is that when you tell them you want to cancel, you have 60 days to do so and during that time you'll have no egress fees. Makes sense.
Biggest problem though is... if you have a substantial amount of data or need to do a complex and seamless transition - this probably won't work for you. I would have to be on the DevOps team that's told to move a complicated and data-heavy application, and they only had 60 days to do so. Also the bulk of the data movement is, in my experience, one of the first steps of migration - not the last.
My hope is that, if nothing else, this will spur similar behaviors in other large cloud providers ::cough::aws::cough::.
In real life it's probably extremely unusual for any company to altogether cancel their Google Cloud contract. More likely is the scenario where you move the bulk of your cloud usage to a new provider, but still have various straggler infrastructure on the old one, which is not worth the effort to clean up. Or, you go to a multi-cloud strategy so you want to move half your data off Google but keep the other half around. Google's egress fees are still standing in the way of these cases.
They know this and this is mainly marketing, I think.
Couldn't you wait to tell Google until you've more or less figured out the logistics, then tell Google that you're leaving, fees for egress gets disabled and you initiate the move. Then you have 60 days to complete the move.
But yeah, if the move takes 30 days because you have a ton of data, and you figure out after the move is complete, that you missed 10%, you only have 30 days to figure out how to get that out too.
This is fine issue of its own. But the way it is laced up here muddies the waters around this actual news.
Google was doing the wrong thing and is changing that while not really taking responsibility for doing the wrong thing.
But to make that less obvious, this other concern is brought into the story creating this high ground on a separate topic.
The strategic reframing of corporate communications is tiring.
calling it now
Put differently, if you can get over the principled nerd stuff like "everything must be open" and break out that wallet, you can stand to make a lot of money without as much headache as the guy who decided to roll his own IdP.
Getting "cost sensitive" on Azure is how you lose track of the rabbit. The whole point in my view is to trade money for reduced complexity and time. It lets you focus on solving really hard business that others simply can't seem to find the time to.
This is a smart play that costs them basically nothing. Remember that egressing data costs customers at worst 3x the monthly cost of storing it. Nobody is avoiding leaving because of the egress fee.
What this does do is assuage the “lock-in” fear common in cloud-reluctant customers, while presents them as being forward thinking.
I’ve never heard of a cloud migration where the data egress wasn’t at least an order of magnitude less than the engineering cost of the migration itself.
The parent poster https://twitter.com/QuinnyPig is one of my favorite Twitter accounts. Every day he gives me renewed hope that, no matter how much I wish I had more time to devote to developer experience for people using the internal tools and APIs I design on the startup side of things... at least it'll be better than the DX and customer service provided by the biggest players providing infrastructure to our entire industry :)
But all cloud providers leverage The Principle of data locality or data gravity, which states that compute benefits from being close to the stored data. If a customer moves the data elsewhere it follows that the compute will soon leave too.