Spotify moves its back end to Google Cloud | Hacker News Reader

Spotify moves its back end to Google Cloud

news.spotify.com

715 pointsdmichel381 comments

What Google Cloud needs is a better sales story. AWS consistently beats out Google in this respect. This story is welcomed and there should be more success stories published like this.

My business is a heavy GC user (AppEngine, Datastore, CloudStorage, BQ, ComputeEngine, ManagedVM) and I couldn't imagine achieving what we have achieved in such a short amount of time on any other platform.

We (myself and another guy) started at zero and shipped our product in 3 months on what is effectively an infinitely auto-scalable platform where we don't have to do any devops or carry a pager. I'd much rather work on features than chores. =) 4 months later and we've had profitable growth and zero downtime (knock on wood) and we're hiring.

Thanks Google!

One of the reasons I avoid Google is because they have taken lot of pain to make their customer care totally suck. I mean even IRS and Comcast support does better compared to Google.

I published a very critical release to my Android app few weeks back and realized that it was not getting published at all. It took me a month to get the problem resolved.

I don't want to buy a cloud service where the support might be nothing more than bunch of "Help yourself" pages.

They have a paid support service that is actually pretty good.

It sounds like you're avoiding the generic Google in the same way you'd not buy something from Amazon vs. using AWS.

As I said Google's reputation is what made me not even bother to try. We all know that AWS and AZURE both are state of art and very competitively priced. If Google wants to beat them they have to win the perception war too. (OR reduce prices substantially).

I don't agree that AWS and Azure are actually competitively priced with Compute Engine anymore. Following our price cut last May and our custom VM shapes, we're often 40% cheaper:

http://googlecloudplatform.blogspot.com/2016/01/Happy-New-Ye...

that's a massive lead (and it's sadly not commonly known). So I agree with you, we need to do better in the perception war, but Compute Engine is hands down the leader in cloud pricing.

Disclosure: I work on Compute Engine (and care a lot about our prices).

GCE pricing is great but GCS is not. It is much more expensive than S3 or cloudfront (2.5x - 1.3x) when you consider per request costs. I have brought this up a couple of times on HN to Google cloud employees but nothing has changed.

1. https://news.ycombinator.com/item?id=10442654 2. https://news.ycombinator.com/item?id=10187600

Other problem areas for GCS compared to S3/Cloudfront

1. SSL support for custom domains in GCS is lacking. 2. Also SSL does not work for domain-named bucket when using <bucket>.storage.googleapis.com/<object> form. You can only use https://storage.googleapis.com/<bucket>/ This makes using SSL for a static site hosted in SSL very difficult because you can't use relative URLs. Cloudfront has solved it by using URLs of the form http(s)://xxxxxxx.cloudfront.net/<object>

That's fair, our native egress pricing is higher. We've chosen to partner with folks like CloudFlare for what we call CDN Interconnect (https://cloud.google.com/interconnect/cdn-interconnect) although this was a somewhat recent announcement.

As far as GCS pricing goes (rather than egress), I think we've been more cautious on the "cost per op" but really led again on cost per byte. This was especially true when we launched Nearline, triggering AWS to launch S3-IA (but with lots of caveats like small files and still a 25% higher base rate).

While not integrated directly into GCS / GCP, does going through CloudFlare alleviate some of your concern?

CDN interconnect is just a bonus :). I prefer GCS having competitive pricing and features since it will be one fewer service to manage.

Also cloudflare is a bit confusing. We only want to serve assets and we serve 3TB per month and I'm not sure if cloudflare is OK with that usage in its free tier option.

I think CloudFlare, Fastly, et al. are all happy to have you serve 3 TiB per month. Their hope is that you grow and upgrade to their paid plans one day.

So what you are saying is basically that you are not doing your market research properly, instead depending on irrelevant anecdata?

Disclaimer: I work on GCP. AWS and Azure are not price competitive with GCP, we're routinely seeing a a 40% advantage. Check out HTTP://cloud.Google.com/pricing/tco

Seriously, go and talk to your marketing guys ... Everyone comes out better if there is a third horse in this race, but Google needs to step up your marketing etc. AWS has a massive presence in terms of conferences, documentation and mind share. Azure has a massive sales team that are offering huge discounts to buy business. Google has ... what exactly?

Yes, compare this to a typical Amazon high quality blog post: http://googlecloudplatform.blogspot.com/2016/02/Google-and-R...

You have no idea what is actually being offered after reading that. Is it a managed service, a recommended configuration of what.

Almost every Amazon blog post includes a walk through on how you actually use whatever they are talking about. To be fair, this post does link to more documentation, but it's general purpose and describes a variety of solutions.

AWS is often competitively priced, but Azure is usually around twice as expensive.

Our Google hosting costs are a rounding error compared with our revenue. That's all that matters. Build business that make money.

We offer paid support for Cloud (http://cloud.google.com/support) that let you trade off responsive time (e.g., 4 hours vs 1) for price.

The support personnel (and engineering!) also do best effort scanning of Stack Overflow, Server Fault, and per-product mailing lists. You can get links to those here: https://cloud.google.com/support/#community.

Disclaimer: I work on Compute Engine, but not in Support (though I do jump in and support customers!)

What kind of support am I going to get if I'm just tinkering with the platform? Maybe I'm just used to AWS but I always found the GCE interface (at least for launching VMs) a lot more difficult to use.

Sorry to hear that. Is there something specific about our current console you find hard to use? Perhaps I'm used to both (and I'm biased) but I like our create instance page just fine.

Like I mentioned above, we do have active communities on Stack Overflow, per-product mailing lists (in this case https://groups.google.com/forum/#!forum/gce-discussion) and Server Fault.

I'll have to try again. The last time I tried to use GCE I had to create some kind of linked project before I could even start to create a VM.

Did not know about this. To me, Google has a terrible reputation when it comes to support and keeping products with large numbers of users running.

Edit: (posted too soon) my main concerns with putting anything on GC are around support, continued support / long-term commitment to the platform, ease of switching (App Engine was a very bad on boarding experience for me when I tried it a few years ago and required lots of GCE-specific ways of doing things in the code), and pricing differences that might make it cost way more in subtle ways to run a thing on GC.

I think it'd be good for Google to start fixing their story around support and commitment to products with everything else under the Google brand before people will be wanting to risk their infrastructure with that brand.

So, you ask your paying customers to pay extra – not to get support, but to increase the chance of getting support in time?

That’s... an interesting business model.

Paying more for more support cases or more urgent responses is standard practice for support plans. For example, here's the AWS Support comparison page:

https://aws.amazon.com/premiumsupport/compare-plans/

which has similar tiers of support.

I agree that Google Cloud offering different response times for different support tiers is consistent with the industry at large.

That being said, Google Cloud support is overly expensive. $150/month Vs. $49/month for AWS Vs. $29/month ($174 minimum commitment over six months) for Azure.

I wish all them had a "per issue" tier. I want to pay for a single ticket/resolution. But at least Google Cloud and AWS don't have a minimum commitment period like Azure (even if Azure is cheap per month).

I'd likely rank them:

- AWS is best (alright price with no commitment).

- Azure is next (amazing price but lame commitment).

- Google Cloud is worst (high price, almost as much as six months at Azure!).

And before you tell me that they offer different things at those price points I DON'T CARE. All I care about is the minimum amount of money I have to pay someone to look at a technical support ticket for me. I don't care if it takes 1 or 48 hours, I just don't want to pay a lot. If I cared about responsiveness I'd be buying the gold plated support with telephone agents.

Google Cloud's $150 to look at one ticket thing really puts me off the entire platform.

> That being said, Google Cloud support is overly expensive. $150/month Vs. $49/month for AWS Vs. $29/month

Your basis for ordering is their support prices which is just a tiny tiny fraction of Cloud infrastructure pricing. Google Cloud is 50% cheaper compared to AWS. Which is huge. Also, thanks to ease of use (Cloud Shell, SSH from browser ..), Google Cloud will save you lot of man hours.

Also talking about commitment, you need to commit for 1-year / 3-year usage of resources and pay partial or full bill in upfront for discounts. Google beats those prices without any commitment.

Let's compare the Cloud providers on these factors.

Let me get this straight, OVH – which is known for bad support – provides the same service for free that AWS provides at 49$/month?

That’s surprising.

Spotifier here. One of the big risks that we identified with the Google partnership was exactly this: Google isn't exactly known for awesome customer service.

We've been very pleasantly surprised. The cloud team has been pretty awesome to work with, including lots of engineer<->engineer contact, walk-throughs of systems and code for critical dependencies, and solid support and collaboration. Exceeded our expectations.

I am not surprised. Spotify is a big company. The question is will it work exactly same for Joe's Inc.

I think latchkey at the top of this thread is precisely "Joe's Inc".

Yes, that is the way we started out, but at this rate, if we do it right, we will be a moderately big company by the end of the year. =)

We are much smaller, but also use the Google paid support.

I'd echo the sentiment here: my experience of the paid support for GCP has been universally excellent across Compute Engine, App Engine, BigQuery and Cloud Storage. This is across 30ish tickets, some of them quite complex.

I use the lowest level of paid support, I think it is the silver plan, and have observed similarly high quality levels of support. Sometimes you get a lower level support dude and have to convince him that your issue is real, but overall the support is good.

I've had problems with their customer support. It got to the point, I just gave up, and left my videos on YouTube. (My problem was that slick move they made when they bought YouTube, and messed with the login names.)

I just try to forget they are up there. I got to one person at Google, and when they realized I didn't want to advertise, it was "Go to the help boards. I don't deal with that stuff."

Do I still use their products--yes, but don't trust them like I used to. I usually avoid customer service like the flu virus, but thought Google would be different?

I guess they are big enough to not care?

(Tedious disclaimer: my opinions only, not representing anybody else or my employer. I work at Google, not on YouTube)

There is a big difference between paying for one of the support offerings, versus being a free user. The status of free users is approximately: free access to the product but no support. If you want customer service then you need to open your wallet.

It's important to separate "I think the support was bad", which would be very interesting to a lot of people who work here, from "I didn't pay for the product so I didn't receive it". Did you pay for support?

You seem to have forgotten to plug your product.

I'm sure you can figure it out. =)

I think he was ironic?

> infinitely auto-scalable platform where we don't have to do any devops or carry a pager

You never know what you don't measure and test

Are you implying that we don't measure and test? We do and I'm very happy with the results. =) StackDriver, Google Cloud console, Intercom, Mixpanel and Pingdom are all being used as well.

If that's the case then you might actually be "doing devops". If pingdom is notifying you via sms then you may also be carrying a pager.

Touché -- That said, we don't have pinggom tied to SMS, just email. =) Pingom is just an external tool for us to graph uptime from other parts of the world.

If we're down there is plenty of other alerting to notify us (mostly through Slack). That said, there isn't much we can do other than wait for Google to fix it.

It is part of the trade off that we have to consider, I'm paying Google for devops instead of paying a team to do it for me. If you've ever had to hire a 24/7 support/devops staff, it is a much easier hiring proposition to just rely on Google to do that for us.

> That said, there isn't much we can do other than wait for Google to fix it.

IE. "Cloud ops"

Indeed, I do wish we'd stop with all of the verbal gymnastics and just say we're doing ops correctly. "We don't do ops" usually means "We don't find ops to be troublesome because we do it correctly". No group of people needs dedicated janitorial staff if everyone picks up and dusts occasionally. Of course, you know what this implies.

One of the big advantages of Google Cloud platform is "Scalability".

Load balancers: Scale to millions of users seamlessly. No warming up, no tickets, ...

PubSub: You can send the whole Internet 10 times in a day through it. Google does that every day

Big Query: Its equivalent to spinning up a 100+ node cluster in a matter of seconds and it can process data at speeds of GB's of data per sec, I heard couple of use cases where the user was able to process at ~ 50 GB/s

Kubernetes: 1000's of nodes running 100's of thousands of containers.

Can't beat that!

Sounds expensive when no one is looking at how the magic works.

Speaking for latchkey here (I'm the other guy), we routinely run several-thousand-qps batch jobs. Scalability is not an issue.

> You never know what you don't measure and test

Funny you should mention that. :) I work at Google on PerfKit Benchmarker (https://github.com/GoogleCloudPlatform/PerfKitBenchmarker). Not only can you measure and test GCP (and other clouds), but we're trying to make it very easy for you to do so!

(PKB doesn't have a benchmark for App Engine yet, though. Sorry.)

True, while other providers focus on DevOps, Google Cloud strives for NoOps. They should do more marketing on that.

Our company motto is that we will strive to never hire a devops role and will work to design and engineer our platform in a way that backs up that claim.

You also forgot to include a link to your hiring page in your website, too bad you're not hiring remotely https://angel.co/gearlaunch/jobs

I want to see your lovely face in the office every day. =)

Man, you should consider remote options at least for the Customer Success given how low you're paying (is $50K really livable in San Francisco?).

Depends on your standard of living. A studio apartment there goes for around $2500/month these days with no parking or utilities. Agreed that remote would be a nice option.

Paying $2500/mo for rent alone on a $50k salary is crazy.

That's why no one does it. They have roommates

and/or they live in the East Bay, where rent is still much more affordable.

Roommates in a "studio apartment" (i.e. one big room)? That's... not great.

As a grown adult, having roommates is not really an acceptable situation.

> As a grown adult, having roommates is not really an acceptable situation.

I know plenty of grown adults that have roomates, because they prefer the quality of accommodations they can afford that way to what they would be able to afford without them. (And not even in places with Bay Area-level high cost of housing.)

If it is unacceptable to you, that's fine, but don't pretend it has something to do with being "a grown adult".

What do you count as an adult? I was the weird one by having my own place. Every 'adult' I know had roommates until they found an SO and moved in together and/or got married. These people all made enough money to live alone, but preferred spending money on other things. I also do not live in a super expensive place. A 1br apt. goes for ~1200 in the trendy spots.

Sure ... but hopefully not in a studio apartment (as per OP).

It's a real stretch to say 50k is a livable salary in most of the bay area.

Is $1,500 much better?

Um yes? It's $1,000/month better

Much, much better.

A reasonable approximation is to target no more than 25% of your take home for housing, but you can probably survive closer to 50% for a while if you must. So for 50k, you're looking at something like $800 - $1600 being manageable. You'd be much better off near the other end of the range.

This may shock you, but yes, it's livable. Many people live on $50,000 or less everyday in San Francisco. It may not be a standard of living that we as engineers are familiar with, but it is not poverty either. It's not a salary you'd want as a household income with dependents, but for an individual it is fine.

I live in Spain and am very frugal, so I haven't experienced first hand the high cost of life in SF, but if someone has to spend $1000 on the rent, there won't be much left from the salary received.

Also the job in question is "Customer Success", and consists in interacting with customers "through phone, email, and live chat" (from the job description). Those can very well be done remotely.

You can't afford rent in San Francisco at $50,000 a year, period. The going rate for a single bedroom (usually with no tenant's rights) is somewhere around $1,200 a month, well over 30% of your monthly take-home at $50,000/year.

If you're going to force people to move to the Bay Area to work for you, you need to pay them enough to afford it. Good luck hiring though.

just playing devil's advocate here but since when do you NEED to spend less than 30% of your take home on rent to survive?

There are people out there paying 50% or more of their take home on their rent/mortgage I imagine - it's just not really considered to be wise move in most cases. 1200/mo for a 1 bed doesn't even sound that bad for what's supposed to be the most expensive city to live in America. I know people in DC who spend 1800/mo on a studio and that's pretty average. You could easily share a room and cut your rent in half if you were smart/willing.

> since when do you NEED to spend less than 30% of your take home on rent to survive?

Yeah, that's the same question every scumbag landlord in America started asking themselves a few years ago when demand for rental units started going up.

You don't "need" to spend less than 30% of takehome on rent. If you assume you're going to be running in a treadmill your whole life, and you're satisfied working for subsistence only, then sure, go for it, blow half (or more) of your paycheck on rent. Lots of people in San Francisco and other expensive cities do this.

If you have life goals that don't involve being stapled to a desk forever, it's probably not such a good idea.

The discussion is living on entry level job salaries. Life goals shouldn't come in to the equation because one of those life goals should be "don't have an entry level job for your entire life"

just playing devil's advocate here but since when do you NEED to spend less than 30% of your take home on rent to survive?

That seems to be one of the common and very popular requirements to even have your lease application put aside for consideration, that being income must be 3x the ask for rent. At least this was the case in Austin proper (i.e. inside the city limits, and inside the I-35, Mopac, 183 and Hwy 71 box), I saw similar requirements when helping a friend scout for a place in SFO.

When I was looking for a place I found a landlord willing to negotiate and haggle on the deposit since my income wasn't 3x the rent ask. They took a larger deposit, I signed a lease, wasn't late once.

As with everything, YMMV.

>since when do you NEED to spend less than 30% of your take home on rent to survive?

While on one side the 3x rent rule might be just "good sense", most places won't consider you if your salary (or the salary of all tenants) isn't 3x your rent.

That's $1200 for a bed in a shared house, not a 1 bed apartment, which goes for about $3300.

Wow, that's expensive.

You can quite comfortably live on $35k a year after rent, if you're a single person supporting just yourself.

I managed to spend 3 years at university living on less than that. I've just spent the last 3 months interning for NZ$20 an hour (~$40k a year), much of that was spent paying about $260 a week for rent (paying rent for 2 places due to contracts). I managed to get along just fine.

Cost of living-wise, where I am isn't radically different from SF.

Christchurch?

My rent in Los Angeles for my studio apartment is about 43% of my monthly take-home, and that's sub-$1000. I'm not saying it's easy, but it's definitely liveable. I don't often have to /worry/ about money... I just am also not saving anything.

So you're deferring your worry?

I really don't see how you can't live for $4160 - $1200 = $2960 a month? Heck, I could live on $500/month easily (after rent and taxes), if I wanted (as far as I'm aware, living costs are a lot cheaper in the US than were I'm from). That would make my yearly salary $20.400 (ignoring taxes, since I don't know what the rate for US is).

> Many people live on $50,000 or less everyday in San Francisco.

And many more people live on $50,000 or less everyday working in San Francisco, but commuting from somewhere else in the Bay Area with lower costs.

I survive on ~$35k post-tax in SF. Though I make more than that.

...and I'm assuming you've got some killer rent control, or some other non-scalable, non-replicable housing situation. That's neat for you, but it's meaningless to anybody else.

I signed the lease in the past two years. So rent control has kicked in once. I have a room in a shared apartment. There are plenty of open rooms on Craigslist for similar rents. I'm not saying life will be great but it's entirely possible to live in the city on $50k/yr.

Most adults do not consider having a "room in a shared apartment" to be adequate. A living wage allows you to rent/mortgage a place of your own. What I'm hearing is that $50k is not a living wage.

You're out of touch. It might not be common for families, but it's very common for young adults to share an apartment. And you're trivializing what activists are trying to do when they talk about making sure people have a "living wage."

As an only slightly older adult, let me assure you that after years of petty arguments, no-fault evictions, playing Craigslist roulette for new roommates, sitting in massive group interviews to compete for a room, other people's animals ruining your shit, etc. ad nauseum, "a room in a shared apartment" will stop being an acceptable option. I never thought I would get there either. Just give it time.

If you think it's hard to live on $50K salary in San Francisco you should think about how everyone else in the service industry supporting all those tech workers manage to live on minimum wage or barely above that. Think about the people serving lunch, or making coffee, or cleaning offices, etc. Where do they live and how can they afford to be in the city every day at less than half of $50K per year? Working remotely isn't an option for them either.

If it's truly prohibitive for a single person to live in SF for $50K per year then we should be really concerned about the families that earn $50K from two minimum wage jobs, and need to support 1 to 2 children on top of it.

Did you see: http://uk.businessinsider.com/talia-jane-fired-yelp-eat24-2-... ?

$50k is livable, but on the other hand you have to live In a city full of people who don't shut up about the Internet. I'd say that's a definite net loss.

net loss -- see what you did there?

I've been relying on multiple Cloud offerings (including PaaS, IaaS etc) outside Google for many years, but for some unclear reason, I cannot manage to consider Google a reliable partner for a type of hosting or another, at this point.

I would love to be proven wrong really, because I often look at their GC offering, and I /really/ want to dive in, yet I feel that I would risk having to migrate quickly in 2 weeks at some point because they are changing things significantly (either shutting down or other major change).

So to my question: for how long have these services been around in a fashion that is stable (feature-wise, quality-wise etc)? Is there anyone here with a longer history of using Google Cloud offering, able to comment on this?

EDIT: thanks to everyone who replied below. I think I'll give it a try :-)

I should add, I've been on top of AppEngine for about 4 years now. This is my second business on it.

I have been running my website on AppEngine for 6+ years now. No problems at all.

Thank you obulpathi and glad to hear this. Can you also take a look at our new docs here:

cloud.google.com/appengine/docs/

and let me know what you think? This is an iterative effort of course, so more changes to come. Thanks again!

For Google Cloud products that have made it to General Availability (GA), we have a one-year deprecation policy (https://cloud.google.com/terms/, Section 7.2). That is, if we're going to make some big change, we give you a 1 year heads up.

Disclaimer: I work on Compute Engine, but I'm not a lawyer ;).

This is good, but I wish Google offered the same policy for most of their other general cloud services.

Promises to do something for a year which you can change or withdraw after 30 days are really promises for 30 days:

"Google may make changes to this Agreement, including pricing (and any linked documents) from time to time. Unless otherwise noted by Google, material changes to the Agreement will become effective 30 days after they are posted, except if the changes apply to new functionality in which case they will be effective immediately. If Customer does not agree to the revised Agreement, please stop using the Services." https://cloud.google.com/terms/, Section 1.7b

Also, Sections 13.1-13.3 means even though Google promises nt to do something, you really don't have any recourse if they do it anyway and cause you damage.

Although in the grand scheme of things, not the worst terms of service document I have read. The mere fact the drafters addressed the issue is worth ... something.

Disclaimer: IAMALBIANYL.

> Disclaimer: IAMALBIANYL.

I like that one. :)

I have a question though, is the prevalence of taglines in any way related to legal requirements? Do you have to stipulate that you aren't a lawyer when stating things because otherwise people to construe it as legal advice, or is it just a fun little identifier of how valid we should consider information to be (like it is in other acronyms of the same type)?

I think it may stem from laws and rules related to practicing law, but there is certainly not any "requirement" to stipulate you are not a lawyer when speaking about legal things. However there are a few things to be mindful of: In the (all of the) United States (and many other countries) the profession of lawyer or attorney is regulated and requires a license to practice as such. In many places to practice law without a license is a criminal offense. Obviously this doesn't contemplate criminal charges against a group of people spouting off about something legal related on the Internet (otherwise we would have to jail perhaps 80% of reddit), but providing legal advice and interpreting the legal effect of contracts, statutes, or regulations is part of the practice of law.

The reason we heavily regulate the practice of law is similar to the reason we heavily regulate the practice of medicine. If you screw something up, the consequences can be bad. If someone takes your advice thinking you know what you are doing, and you are wrong, that person can end up a lot poorer or even in prison. (Though some will argue we regulate it so lawyers can be the only ones charging high fees for legal advice!) Just like one probably shouldn't go around telling folks not to worry about their incredible chest pain and shooting arm pain unless they are a trained medical doctor, the same could be said for those that go around offering home spun legal advice.. ("Yeah man, if you ask the undercover cop if he is law enforcement, he HAS to tell you! Otherwise it is entrapment man...")

Also the reason you always see lawyers preface everything by saying "I'm not your lawyer.. This isn't legal advice... Yadda yaddda yadda" is because once there is an Attorney-Client relationship a lot of stuff happens. The lawyer has a host of duties to the client, from confidentiality, to competence, and many others. That obligation isn't taken lightly and many times if there is any doubt whatsoever whether or not there is an attorney-client relationship, the law will find in favor of one because --- well -- the lawyer should know better. Therefore lawyers will make it annoying clear they aren't your lawyer before spouting off some deep legal thoughts about whatever the topic of discussion is.

That lines up with what I assumed, and the last paragraph is specifically what I was asking about (not the IANAL case so much as the opposite). Thanks!

Great answer. Has the acronym "IANAL" ever been tested in court in such a way, to your knowledge? Now you have me curious, and that'd honestly be quite amusing.

Ha! Interesting question. I just ran an all state/federal Westlaw search for "IANAL" and didn't hit any case law. Two or three briefs, a couple of scholarly articles, but no court opinions. I also searched for "I am not a lawyer", but that brought up about 130 caselaw hits which seemed to primarily be references to various filings by pro se parties actually saying in a court filing "I am not a lawyer".

It may have started as an ass-covering move, but I also think of it as a common courtesy for letting people know that you're offering non-specialist advice and to take it with a grain of salt.

I kind of wish people did that for more things: "I am not theologian/windows user/empathetic person, but..."

That's pretty eye opening. Curious if you have looked at AWS with a similar regard, are they any better?

Not any better, arguably worse:

https://aws.amazon.com/agreement/

"We may change, discontinue, or deprecate any of the Service Offerings (including the Service Offerings as a whole) or change or remove features or functionality of the Service Offerings from time to time. We will notify you of any material change to or discontinuation of the Service Offerings."

"We may terminate this Agreement for any reason by providing you 30 days advance notice."

"We may modify this Agreement (including any Policies) at any time by posting a revised version on the AWS Site or by otherwise notifying you in accordance with Section 13.7..."

Ha, yikes! Thanks for the highlights. I'm guessing Azure has similar (or any IaaS company) leaving us to basically go on trust of the company to support their offerings long term.

Is this really how it works? I guess you have to go to court to find out, but I always thought of it this way:

If I have the same policy (support for 1 year) and I change the contract - the change will only take actual effect after 1 year.

I mean, I'm contractually bound today to provide support for 1 year from today, no? Am I just rambling, or does it make sense?

Google seems to be facing a trust problem here that its competitors don't, due to its history of shutting down popular services, even those that were being used internally and even those that it had been recently hyping to the press. Due to this unusual context, perhaps it would help if Google were to increase this promise from 1 year to 10 years (or 5 years or 3 years).

On the flipside, API's like maps have had stable versions that are deprecated for years.

That's one of the few examples that hasn't burned developer trust. But it's still not a shining example -- there were times of price changes and rate limit changes.

Yep. Their trust problem is also very well earned both from the APIs they've killed/changed and from the long list of Google products in the graveyard.

They've killed APIs/Products, but they've almost always given a very long amount of time to transition.

Also have killed APIs/Products that are extremely difficult to transition from with short notice. For example, they gave ~4 months for the Google Wallet for the web transition, and no data portability for businesses with users on subscriptions.

Stripe, for example, has a data portability clause that allows you to move card data to another payments processor that meets some compliance standards.

https://support.google.com/payments/merchant/answer/6107573?...

Regardless of if you get 4 months notice or 1 year notice you still have to do the work to migrate and you're still stuck with the sunk cost of investing in/learning a doomed platform.

That's why I wouldn't build a business on Google, because they have a long history of killing things when they aren't wildly profitable/successful. A $10m/yr profit product is considered a distraction of valuable engineer time unless it has some ulterior goal for the company.

And I'm not saying they should change -- they do what's right for them. One engineer working on some distraction project could instead be moved to ads quality and end up making a change worth anywhere from hundreds of millions/yr to billions.

But they can't have their cake and eat it.

Google App Engine comes to mind. I'm amazed they would kill of such a product at all.

Google App Engine isn't killed. It's still alive and new features are being added. What are you talking about?

The GP was probably referring to the huge price change that effectively killed it for many people.

Trust is hard to earn and easy to lose. No matter how much power/money you have at disposal.

Anecdotal: AppEngine deprecated their Master/Slave datastore[0] in April 2012, and it was actually shutdown on July 6, 2015. So that's 3+ years to move to newer tech.

[0] http://googleappengine.blogspot.com/2012/04/masterslave-data...

Also worth noting that Google built an automated tool to handle the M/S to HRD datastore migration for you. It was remarkably painless.

Painless eventually; the first versions didn't work at all, at least on the project I was involved with. Fortunately by the time you had to switch, it worked very well.

Being first to migrate to the hot new infrastructure has never worked out for me in any project.

I was mildly annoyed that I had to spend an afternoon "migrating" my app from the M/S datastore, even though I never used it in the first place.

I really struggled around 12 months ago. Lots of weird errors and it took down the service unexpectedly when trying to stage it.

Really made me regret going with app engine a few years ago.

I wonder how big you'd have to be to get a real SLA from Google, Azure, or AWS. I imagine 25% off your hosting bill doesn't really cut it for those companies running millions of dollars in revenue through it.

I've had admin on an account that did about half a million USD on AWS monthly in the past; we cut it down a lot by expanding physically and going in hard on reserved instances, but my view into things started around there. I'm unsure if we ever attempted to negotiate a SLA, but we paid for premium support just like you (despite having a dedicated, and great, sales rep) and got kickback credits from the general SLA policy[0] just like you. Nothing special.

If anybody has a special deal, it's probably Netflix. I suspect even half a million monthly is a smaller account to Amazon.

[0]: https://aws.amazon.com/ec2/sla/historical/ specifically, not the current (better) one

The problem (I suffer from it as well), is that people naturally feel that the corporate offerings from Google will mirror the experience that they have with the consumer offerings from Google. And the big issue with Google's consumer offerings is that they seem to have limited to no support and that there is a very real chance that Google will just shut them down all of a sudden. This is the exact opposite of what you want in the enterprise space, where Service & Stability are paramount. Even though I know that Google's corporate offerings are treated differently, that little niggling doubt remains. Plus, Google really needs to hire some damn marketing experts to sell the features & benefits of their offerings and make them cohesive - they just have their stuff plastered everywhere.

That's pretty much exactly what my perception is. Add to that, you're account gets shut down and/or frozen without warning and only a kafka-esque system to find out why you were shut down.

It makes sense that isn't true but I have to admit it is what comes to mind.

> And the big issue with Google's consumer offerings is that they seem to have limited to no support and that there is a very real chance that Google will just shut them down all of a sudden.

Express has, IME, excellent support, and I haven't seen any Google product (consumer or otherwise) shutdown "all of a sudden".

The ones people point to as examples of Google products being shut down were shut down with very long notice.

I do think that there is a very loud group of people who like to raise the "Google's just gonna shut it down tomorrow" line every time a Google service is mentioned, but it doesn't seem to be based on any real history of Google being any more prone than any other provider to shut down services (consumer or otherwise) on short notice or without a migration path. Maybe that loud group represents a real and commercially-significant feeling about Google in the market and not just a small, loud group that doesn't real effect the market for Google products. But I don't know what you can really do with that, even if it does.

This is also my impression of Google compute offerings, but it's also my impression of AWS offerings.

I mean, when people think of AWS services, do "service and stability" really come to mind? IME it's impossible to reach a person unless you're a multi-million-dollar org, and Amazon itself tells customers that AWS services can go down at any time and it's up to the customer to architect the application to deal with it.

If I had to peg a reason AWS gets treated differently in the marketplace, I think it's probably some combination of a) AWS was first to market, and b) AWS revenue seems materially important to Amazon, while Google Compute revenue is a rounding error to Alphabet. Therefore, Amazon has a lot more incentive to keep, maintain, and grow AWS.

> IME it's impossible to reach a person unless you're a multi-million-dollar org

I am sorry to hear that, but I can assure you it's not the case. I'm an engineer on a newer (and therefore smaller) AWS service, and I help investigate customer complaints, from customers of all sizes, all the time. I'll also note that not only do we handle cases that come in via our paid support, we also do deep dives for customers who post to our service's forums and indicate issues.

My experience with AWS has been exactly this. I had a Cognito question/issue on IOS that I posted to stack-overflow and got a quick response from an engineer at Amazon. Also I posted a support ticket (zero paid support) to increase my sns API calls limit (which didn't end up being required because I was doing it wrong) and they got back to me over a few days to a week.

On the other hand, Google takes my Android app off the play store with next to zero details on the reason why and I have next to zero methods to contact them. The only way I was able to get my app back was to remove features via trial and error to determine what part was infringing. Why would i partner with Google again for hosting?

You are comparing 2 very different products. Now if you were comparing play store support to Apple App Store support they would be more in line with each other. Or Googles cloud offerings to AWS.

Play is a low margin business per "customer" compared to cloud. This has direct impact on the type of support you can expect.

They have reasonably priced support plans that give you access to a person: https://aws.amazon.com/premiumsupport/

I can't comment on Google, but we pay for AWS business support ($100 per month currently) and we can raise as many support tickets as we like and get nearly instant access to an engineer. Also, the quality of the support has been amazing. They have often gone out of their way to solve problems that are not strictly AWS problems (e.g. MySQL tuning, network issues, VPN questions, etc) without ever questioning it. Our account manager is easy to reach and through him I can get access to solution architects who are happy to advise on pretty much anything. There are also frequent invitations to AWS events, where SAs and account managers are usually keen to meet up and discuss projects. I know this is enlightened self-interest on their part rather than altruism, but it's definitely to our mutual benefit. When pushing to trial AWS I often encountered this view within my org (that is was large, faceless, and inhuman with poor support) but the experience we've had has dispelled that myth.

You could have done the same on AWS (source: I have worked at AWS for 6 years, and know GCP very well)

That is true. And, as Spotify found out, what Spotify does can also happen in its own Data Centers :)

Google does give an additional level of comfort - seamless scaling, No-Ops, powerful Big Data tools, flexibility.

I'm going to shamelessly promote one of my opinions on this topic when it comes to BigQuery:

https://cloud.google.com/blog/big-data/2016/02/visualizing-t...

> Google does give an additional level of comfort - seamless scaling, No-Ops, powerful Big Data tools, flexibility.

I'm going to bookmark this comment and come back to it when Google decides its getting out of the "cloud" compute business.

Hmm, sounds interesting.

Why do you think they will do so?

Its not their core business. Google constantly exits businesses unless they explode in profitability. As someone who currently uses AWS, it would take an act of god for me to move over to Google Cloud, even if the tools were better and even with the cost of some services (nearline storage) cheaper than AWS.

I'm not just paying for a service. I'm paying because I know AWS is in it for the long haul.

This is great for Spotify, just as AWS was great for Netflix (considering both services are terrible at representing true workloads; they're both just control planes serving static content, videos or music). I don't see it dragging a ton of business into Google, just making their acquisitions easier for people who decide to start on GCP or migrate in.

Just my two cents.

> Its not their core business.

Its the heart of their core competency. Its not their principal mechanism for monetizing that, though outside of the core advertising business its also an avenue that they seem to have made one of the most significant, longest push to make a route to monetization, even in the presence of large, dedicated competitors (behind Apps for Work.)

Obviously, any insufficiently profitable business from any vendor will eventually be closed down, either voluntarily or by business failure, but Google Cloud Platform doesn't seem like a particularly likely candidate for closure on any timescale on which any existing cloud platform would be reliable.

> Its the heart of their core competency.

For "Google Scale" computing tasks. Not someone with tens or hundreds of virtual machines.

Would you have said the same thing about Amazon when they started AWS? "Data centers aren't their core business: retail is".

Yes. But, Amazon is known for their customer service, even in the AWS space. Google is not.

If Google wants to succeed in certain spaces, they need to commit to them for a duration, and evangelize them. I don't see them doing that anywhere as well as Amazon does with AWS.

Are they known for the customer service with AWS?

Anyways both companies provide SLAs, phone and email support, and consultation for a price.

As someone who's been on Google Cloud for 2.5 years now and has seen the growth and the transformation firsthand, I welcome this bet.

AWS doesn't have AppEngine or the Datastore. Yes, there is similar products, but they are absolutely not 'the same'.

Is datastore tied to appengine? some parts of the docs say "datastore is for appengine" and other parts don't mention appengine at all.

It used to be but not anymore; you can use it separately now.

Are you happy with the performance of AppEngine?

I have tried it last summer and was quite disappointed. CPU-bound programs that completed in a few seconds on my own computer took about ten times as long on AppEngine despite similar nominal specs (GHz and RAM).

App Engine expects your web apps to return almost immediately (in a matter of few ms). For anything that takes longer, you need to use task queues in App Engine. Or maybe try AppEngine Managed VM's?

App Engine requests time out after 60 seconds (that's 60000 ms).

https://cloud.google.com/appengine/articles/deadlineexceeded...

(I work at Google, but not on App Engine)

They don't have to if you use Basic or Manual Scaling (Then they can run up to 24 hours).

https://cloud.google.com/appengine/docs/java/modules/#Java_I...

If you run multiple modules at once (some serving the frontend, some for longer tasks w/task queues), you can have the best of both worlds and use different scaling options for each.

(I also work at Google, but also not on App Engine)

If you like the App Engine model, but want a "full computer worth of horsepower", you might want to consider our "Managed VM" environment built on top of Compute Engine: https://cloud.google.com/appengine/docs/managed-vms/

Disclaimer: I work on Compute Engine.

Managed VM is still beta, I think only brave one could try beta for production.

We have been using Managed VMs in production for while now, integrated in a micro-services architecture, for non-essential, non-frontend services and are quite happy with it. Using custom VMs (Docker based) and work like a breeze.

I love GCE. I really would like to see Node on it natively, without the need for ManagedVM. I tried it a while ago and did not want to deal with all of the Docker stuff. It was especially a pain on Windows. :(

Default AppEngine instance is only running at 600MHz. You can choose to pay for a faster instance:

https://cloud.google.com/appengine/docs/python/modules/#Pyth...

Traditional App Engine is not particularly cost-effective for compute-intensive loads, but it works great for basic web workloads. If you have special needs, Managed VMs have the performance characteristics of the underlying Compute Engine node. We've found that App Engine is not "all or nothing"; it's the backbone of our app but we run services where they make the most sense (usually GCE).

In addition to all the excellent comments here. Let's not forget that the massive massive Snapchat runs almost entirely on Google AppEngine. So if AppEngine can handle their scale, it can handle most others :)

No longer true. It's all GCE now last I heard.

Where'd you hear that?

Not without risk of giving an argument from authority, your sources are inaccurate. As late as November of last year Snapchat was on stage talking about their broad AppEngine use. (I work on Google Cloud).

App Engine Product Manager here

The main reason I tend to shy away from GC is that it's blocked in China. AWS isn't (which also means Heroku isn't blocked since Heroku runs on AWS, so I use AWS for IaaS and Heroku for PaaS).

If you plan to offer multi-lingual content, anticipate having viewers in that part of the world, and your content isn't something they would be interested in blocking, it's just a practicality worth factoring in.

Another reason is that I find the GC console a mess. I had to simultaneously use two versions of the console to get different tasks done; they didn't have one console that had all features of the system. Heroku was a lot cleaner and easier to navigate in this respect.

On the contrary, GC is NOT entire blocked in China. Some of the Google services (including GC console / dashboard) are indeed blocked but GC's IP address pool and the applications running on GC does seem to be accessible in China.

My company is running mostly on Google Cloud and I had just returned from a trip in China where I've verified it.

In my experience (3 months in Shanghai at the end of last year) AWS does occasionally get blocked or severely slowed down in China. I'd say it was inaccessible about 5% of the time (for significant periods of time).

You can of course run in the AWS Beijing region which requires you to have a Chinese business presence with a ICP license.

It'll be interesting to see what happens now that Google is expanding its presence in China [1]. I ran into some Googlers on the Maps team while in Shanghai - not likely a coincidence.

[1]: http://www.bbc.com/news/technology-34698642

The old App Engine / API / Admin console era is over. All tasks have been migrated to https://console.cloud.google.com/ including some of the last App Engine specific administration things.

Slightly off-topic, but not too off-topic question: Does GCE have a text format for its configuration?

My pet peeve with cloud services and their web-based admin GUIs is that you have to use the GUI or the APIs to do anything. You can't capture the current configuration and then replay it if anything is misconfigured. You can't save or restore the config. You can't diff the config across versions. Most SaaSes don't even have audit logs, or if they do, they're too minimalistic to provide the ability to get a lot of detail or roll back anything.

My wish list item is a kind of "Puppet for GCE". I write my manifest, push them to the cloud, and stuff happens. If something isn't mentioned in my manifest, it gets deleted. Truthfully, I don't want to use a web GUI at all.

At the very least there should be a way to do an inventory of an account. Recently I needed to take over an older AWS account, and I wanted to find out what was in it — instances, load balancers, buckets, and so on. It's virtually impossible to do this with the management console since the ontology of possibly configurable objects is so huge and complicated, and pretty much everything has a computer-generated hash instead of a human name, so I was surprised when I couldn't find anything.

Perhaps you want to look at something like Hashicorp's Terraform (https://www.terraform.io/) which will do most, if not all, of what you're looking for.

That's exactly what I was looking for, thanks! Only feature missing is that it apparently can't dump an existing setup to file; looks like you have to write them from scratch.

Check out Deployment Manager...I think it has what you're looking for.

(I work at Google but not on Deployment Manager)

https://cloud.google.com/deployment-manager/overview

In (most?) places throughout the web-based console you can click a little link to show you the REST API equivalent of your resource (whether it's an instance, a disk, or a command you're about to do). You might be able to make the job of "exporting your infrastructure" somewhat straightforward from those, though I think I'd personally just write against the handful of APIs in Go. All of our newer offerings (meaning GCS, GCE, etc.) have really sane object models that just map directly to a fairly obvious, autogenerated struct in the Go client library. Here's the struct for an Instance:

  https://godoc.org/google.golang.org/api/compute/v1#Instance

Thanks, being able to see the API call for a resource is a nice convenience.

But part of my point was that while you can code together something with APIs, you still need to write the diffing and converging logic to accomplish what I suggested. For example, if someone has gone through the console and added something, so that you now have a conflict, you want to be able to resolve the conflict. First by either accepting or rejecting the conflicting change, and then, if you rejected it, remove it as part of the next push.

I'm on mobile so this initial response will be short:

AWS' core product is AWS. They do a damn fine job of providing this.

GC's core is "who knows what" - they do a damn poor job of even making an offering if switch thousands and thousands of servers to - let alone thousands and thousands of man hours.

Ironically, Google has massive domain knowledge of how to run infra... Yet clueless how to adopt and support and foster companies other than google.

It sucks.

But I've been in the business of fork lifting many people to aws, for all the reasons you'd expect:

Find me a single ops eng on the market who has done scale in GC? Nope.

EDIT: may I please have a rebuttal?

> Find me a single ops eng on the market who has done scale in GC?

Every user of appengine, for a start. You need to code to their datastore, but apart from that scaling is practically infinite - and completely invisible. We don't need to configure, provision or even * understand * load balancers or geographically redundant servers.

Concrete examples? The royal wedding website was served from appengine a few years back; I imagine you'd be impressed with how that scaled from nothing to phenomenal traffic loads and back again. And Khan Academy, Snapchat, and now Spotify ought to meet your definition of "done scale".

Wonderful response!!! Thank you, I'm not trying to be obtuse, I do want to understand who is doing what and where.

I also want to understand the cost doffs as a Well.

I hope that you see the irony of your argument in this thread :)

Honest request:! Please enlighten me..

What's your product by the way?

According to his github it's https://www.gearlaunch.com/

Which still isn't very descriptive, sorry. We are working on a website.

Angel might be a bit more of a description. https://angel.co/gearlaunch

Anything more, feel free to contact me privately. =)

We are not stealth, but we also aren't the type of company that goes around pushing ourselves down people's throats. The people who are primarily interested in our product know about us through other channels.

That said, we are building the website out more really more from a hiring perspective. It is hard to hire people when all they get is a logo on a homepage. For now, just know that we are a startup that is actually profitable, building a great product that people want and we're growing like mad.

My experience with upper management is that Google isn't "serious" about their cloud offerings based on how much of Amazon's profit margin is tied up in AWS. It's an enragingly pointyhaired sentiment to deal with.

Who's upper management? As I mentioned in another thread, we acquired Diane Greene's company so that she would run Cloud and Apps at Google, back in November:

http://www.nytimes.com/2015/11/20/technology/google-picks-di...

and Sundar (CEO) and Ruth (CFO) mentioned Cloud a lot on Alphabet's most recent earnings call. I'm not disagreeing that AWS is massively important to Amazon, but to say that Google (and Alphabet) aren't serious about Cloud is incorrect.

My clients' upper management. It's a common sentiment as you can see from other threads in this post. Google has a persistent credibility problem in the cloud services market. It drives me crazy (full disc: I'm a GCP certified engineer and I work 100% in AWS. I really miss AppEngine, BigQuery and Datastore.)

I assume your business is GearLaunch? If so, why do I need to enable javascript just to see the landing page, which is just a logo...? I'm curious if this was intended, or a consequence of the framework you're using (e.g. Meteor).

Edit: It seems to be on purpose, with the use of noscript tags.

It is 2016 and ok to run JavaScript, the Internet works much better with it. I promise not to hack your flip phone. =)

I run three browsers with varying levels of "openness". It just happened that I visited your website while using NoScript. Turned it off for your site, but I was only greeted by a logo. So my interest was genuine - not trying to be snarky. Why do you force JS to show a logo? And I also don't really know what your company does...

It's 2016 and Hacker News, this very site, works fine w/o JS (except for search). That's how I'm using it right now. That's how I've always used it.

I can access most of the sites I routinely visit without JS. E.g. The New York Times works better w/o JS because it doesn't block articles or count how many I've read.

And I don't really care if you you will or will not try to hack me if I visit your site with JS on. There are countless other sites who will try exactly that, so why shouldn't my default be "NO"? And if it's not the sites themselves, it's the shitty ad networks they all use; I don't trust those at all.

IMO it's the epitome of hipster arrogance to display nothing useful to people w/o JS.

"hipster arrogance." Ha. I'm 42, got my first email address in 1991 and hardly a hipster. I own a fixie because I used to race on velodromes.

What is the word for people who use the internet in some purposely crippled fashion, just so that they can complain about things not working for them? =)

Your website is just a logo and nothing else, how exactly are you growing. What is your business model?

GearLaunch provides merchants with software that allows them to build and run online storefronts, and also manages production, logistics and customer service for all products sold. GearLaunch is the only e-commerce software provider to cover the entire value chain, enabling marketers to focus on marketing.

https://angel.co/gearlaunch

AWS has Werner Vogels who is not a CTO but a tech salesman on every conference.

The only thing that is stopping us from having Google Cloud is lack of an Australian data centre. I'd love to be on Google Cloud, but requirements prevent us from doing so unfortunately :-/

Legal requirements? Internal company ones? Latency? ;)

Latency is a huge one for AU. With AWS in SYD you can easily cover BNE, SYD, MEL and even AKL. The difference from SIN or TPE is very noticeable. SYD to PER and SIN to PER is negligible.

"profitable growth"?

:puzzled:

Would you mind sharing with us what product you have shipped?

I just like to see what other people can achieve in 3 months of time, it's kind of a motivation for me :)

Better sales story AND marketing.

Look at the names themselves CloudStorage Vs Redshift/s3, ComputeEngine Vs EC2. Cookies vs. Oreo.

AWS service names are a brand in themselves while Google service names are too generic.

We internally use our three letter acronyms all the time: GCS, GCE, GAE, GKE (which itself is a pretty clever joke about the K being for Kubernetes).

I want to commend some of the Google Compute Platform employees that commented in this thread.

I was at AWS for 6 years (left 2 years ago... today!), and I've always been a proponent of being more open and communicative with developers, but it rarely happened - I guess that AWS' PR policy and such are a big showstopper for these kind of discussions. Although, some individuals did their best (e.g. Jeff Barr) to try to share as much as possible.

It seems that the Google guys know how to do it.

Keep doing it. It will help your business a lot.

Thx boss!

AWS employees may not speak out very often, but they hardly need to. AWS is a marketing machine, from Jeff's regular blog posts, to the volume of free information they provide and to the conferences that they are running everywhere. I think Microsoft is starting to catch on with Azure promotions & conferences, but it's like Google didn't even know that the there was a race .... and they left their running gear at home.

This might be true but I think the issue is that many AWS employees actually want to speak out or respond to issues personally.

I used to work at AWS and the things we could say were very controlled. It could just be my team/managers though.

No, I agree with your last statement - at AWS you always feel the invisible shadow of PR :)

For me personally[0], the difference isn't being able to speak in terms of marketing, but in terms of engaging with users who have problems. It may not translate to much in terms of run rate, but I like to think (or at least hope) the presence of GCP staff on HN, Stack Overflow, and in any other community we happen to be part of will in the long run pay off in terms of goodwill.

I am free to comment on posts about GCP here on HN. If there are workarounds, I can post them. If it's a bug I can comment as such and file it internally. I can express opinions as long as they're clearly marked as my own. I didn't have to take any training or end up on a whitelist; Google expressed their faith in my good judgement when they hired me. If nothing else, it was a morale boost to feel trusted to act responsibly.

At the other end of the spectrum, GCP has started building out in terms of large-scale engagement. Cloud has been a big part of Google I/O; now there's GCP NEXT as a Cloud-specific user conference: https://cloudplatformonline.com/NEXT2016.html. Building more in this space seems easier to me than a cultural change which encourages (rather than prohibits) customer and community engagement on the individual level, but perhaps I'm being naïve.

[0]: I currently work at Google on Compute Engine; I formerly worked at Amazon, although not on AWS.

The Google Cloud guys are always all over Hacker News threads whenever it comes up. I wish they had the same exuberance about their support tickets :(

> I wish they had the same exuberance about their support tickets :(

Is there a particular ticket (or tickets) you ran into trouble with? I can try to follow up.

My issue was resolved eventually (took about a month with one week lag in their replies and it was me who figured out what was going on, not the support).

It looks like the App Engine documentation was finally fixed. It used to say "If you do not specify a class, F1 is assigned by default." as recently as a few months ago. For a long time the default was actually F1. Then one day my costs went up unexpectedly. The default instance class had changed without notice (at least none that I ever saw). I didn't know that's what had changed though and I emailed billing support just asking what happened. Canned reply email after canned reply email with requests for tons of information they already have (including information I typed into the original support request form). They also had me take screenshots of my console (can they really not look up my information?). And sending me the canned emails takes a week for some reason.

I should probably point out this was billing support. I'm not sure if that's a different department or what but I'm pretty sure they have no idea what they are doing over there (or, at best, just don't have the time or interest to help smaller paying customers).

Thank you for taking the time to write up the experience. I can think of a few metrics that directly relate to what made it suboptimal. Google is pretty good at improving things when we have metrics we can measure directly.

That said, this falls into that category of things where I can tell you "I'll follow up with the teams involved" but it's unlikely I'll be able to say more than that (unless any result makes it into a public blog post or something).

I will do what I can, though.

I've made feedback on AWS docs, and every time I get a followup from the person responsible for that documentation (different section each time).

I tried Google Container Engine (GKE) and really liked it - it's the best cloud solution for deploying Docker to production in my opinion, mainly due to its use of Kubernetes. Unfortunately in my Web apps I make heavy use of Postgres-specific features, and since Cloud SQL only supports MySQL, Google Cloud is a total non-starter for me.

So for now I'm on AWS, using Postgres on RDS and deploying containers with ECS. ECS is a lot simpler than Kubernetes, but since my apps are pretty simple (a half dozen task definitions), it's not a big deal. I really hope Google adds Postgres to Cloud SQL at some point.

Is there any reason you can't mix and match Google Container Engine and Amazon RDS?

Latency

And bandwidth charges

Got it, thanks! What about running a postgres container on GKE?

There's nothing stopping anybody for sure, but it's definitely not going to be as hands-off for getting to high availability, backups, scaling well, etc would be with managed SQL as a service.

Nice writeup on that here: https://blog.oestrich.org/2015/08/running-postgres-inside-ku...

Lots of reasons not to. You'd be in charge of setting up backups (e.g. setting up WAL-E + an S3 bucket and monitoring that - and you'd have to add WAL-E support to your Docker image, which is a PITA), have to perform database upgrades yourself, have to monitor resource usage (CPU, memory, and disk), have to worry about how to manage the Docker volume so that write performance doesn't suffer (Docker defaults to copy-on-write - very bad for databases), you'd have to manage a Docker image and how to make it configurable (how do you enable Postgres extensions if you end up needing one?), you'd have to worry about how the GKE/Kubernetes scheduler will allocate the pod and what other resources on that node might affect it (and how that will affect the resource assumptions in your Postgres config), any Docker updates will require restarting the container and thus downtime (unless you have some kind of replication setup), all kinds of things.

In general I wouldn't want to run a relational database in Docker, especially not as some sort of generic cluster of containers; I'd want to give it its own VM with Postgres configuration settings specific to its resources (e.g. using pgtune).

Ahh totally makes sense. Thanks a ton for the insight, it's super helpful!

I'm tentatively happy with our move to GCE as well, although we need to live with it more. What really delights me is the quality of the web dashboard, command line tool, and the ability to open web based SSH windows with a single click.

> I really hope Google adds Postgres to Cloud SQL at some point.

I too hope so.

There are vendors running managed Postgres services on Google Cloud Platform, like ElephantSQL [1] and Aiven [2]. And you can of course run your own on GCE, even to the extent of 24/7 commercial support - you can run EnterpriseDB from Cloud Launcher [3].

And, you can also run Kubernetes on AWS - we have a group focused on making sure it's an excellent experience.

I work for Google Cloud Platform; ping me if you'd like more help with either option.

[1] https://www.elephantsql.com/blog/2014-11-17-google-compute-e... [2] https://aiven.io/ [3] https://cloud.google.com/launcher/solution/public-edb-ppas/e...

You don't have contact info in your profile.

Fixed, thanks!

> There are vendors running managed Postgres services on Google Cloud Platform, like ElephantSQL

I did check out ElephantSQL but my pricing needs are somewhere between their $100 and $20 plans and there seems to be a lack of configurability compared to RDS's parameter groups (e.g. enabling extensions).

> you can also run Kubernetes on AWS

I've had success turning up Kubernetes clusters on AWS for demo purposes, but I really don't want to manage a k8s cluster myself (anecdotes I've read about etcd failures / partitions especially scare me). Also I use Terraform for provisioning, and kube-up.sh is not something that fits into that paradigm. I've also made the mistake running kube-up.sh with the wrong arguments after a previous invocation that had created a cluster, which caused it to try and create a new cluster, which wiped the local cache of the previous cluster I had made, making kube-down.sh unable to automatically clean up the old cluster (so I had to do it manually in the AWS console).

The other thing I tried was the kube-aws CoreOS tool, which is nice, but it comes baked with a 90-day expiration due to TLS certificate expiration, so I'd have to set up some sort of PKI process to make that production-ready. All in all just too much work for a single person trying to deploy a small number of containers for small to medium sized projects; if I was a medium-sized company with hundreds of containers and some dedicated DevOps resources maybe it would be worth it, but for myself I'd prefer a turn-key solution like ECS or GKE.

We also just had a user check in a cloud formation template for Kubernetes on AWS - would that meet your needs? (No kube-up.sh required)

Disclosure: I work on GKE and Kubernetes at Google

You can't compare that to postgresql on aws. especially not when it comes to pricing.

IMO one of the benefits of a platform service is that you get the whole platform from one vendor, so if you're having some problem, you can work with one vendor to sort it out. Trying to get Google support to work with a 3rd-party vendor and my hypothetical company on an issue sounds like a nightmare. There are already many other options for platform services which provide e.g. Everything you'd need to run a Rails app in dev and prod, so that's where the bar is.

I sincerely hope you will add PostgreSQL support to CloudSQL soon.

The current Postgres offerings are not great. ElephantSQL is extremely expensive compared to their offering: 4 cores, 15GB RAM, 1TB data for $1,000/mo. CloudSQL (2nd gen) with the exact same specs would be $370/mo. Aiven doesn't advertise exact prices until you sign up, so I can't compare, but I see that the number of instances (max 3 nodes) is very small, so not really an option.

Google Cloud SQL is no replacement for Amazon RDS in my opinion. Cloud SQL instances run outside of your project's private network, so connections from Google Compute Engine have to be over a Public IP. This means accepting connections from any host (insecure) or whitelisting each Google Compute Engine VM's IP address (pain in the ass).

I've resorted to running my own MySQL instance inside of Google Compute Engine and setting up replication and off-site backups myself. It's definitely not as convenient as Amazon RDS, but the rest of Google Cloud has some great features like Google Container Engine.

Would love to see an RDS-like solution from Google that runs in a project's private network and supports more than just MySQL.

So, basically something like AWS's VPC? Given that it took them a few years to get that right even with tons of people asking for it, I'm not super optimistic that Google could match it any time soon. I'd love to be wrong!

We've recently released Subnetworks [1], which let you segment your network in a similar way to AWS's VPC. We've been working closely with customers to ensure that we're building out the platform in a way that works for everyone, up to very large, technically adept, customers like Spotify.

(Many of the other features of VPC, such as a worldwide network rather than a regional one, were built in from day 1.)

https://cloud.google.com/compute/docs/subnetworks

GCE networking was always equivalent to AWS VPC, but there's new functionality in stuff like Cloud Router, subnetworks, and other beta features to expand that even more. Here's the Network Services section from a guide explaining GCP for people who are used to AWS:

> The differences between AWS networking and Google Cloud networking are significant. This due to the nature of how these services were designed. Google Cloud Platform treats networking as something that spans all services, not just compute services. It is based on Google’s Andromeda software-defined networking architecture, which allows for creating networking elements at any level with software. As a result, Cloud Platform can create a network that fits Google's needs exactly—for example, create secure firewalls for virtual machines in Google Compute Engine, allow for fast connections between database nodes in Cloud Bigtable, or deliver query results quickly in BigQuery.

> To create an instance in Google Compute Engine, you need a network. In Google Cloud Platform, we create a default network for you automatically, and you can create more as needed. Unlike AWS, there is no choice of a public network like Elastic Compute Cloud-Classic. In all cases, you create a private network, much like Elastic Compute Cloud-VPC. Unlike Elastic Compute Cloud-VPC, Google Networking does not have sub-networking, but it does have firewall rules, routing, and VPN. These prerequisites are not necessarily required for all Google Cloud Platform services. Google BigQuery, for example, does not require a network because it is a managed service.

> Most of the networking entities in Google Cloud Platform, such as load balancers, firewall rules and routing tables, have global scope. More importantly, networks themselves have a global scope. This means that you can create a single private IP space that is global, without having to connect multiple private networks, with the operational overhead of having to manage those spaces separately. Due to this single, global network, all of your instances are addressable within your network by both IP address and name.

> Another major difference between Google Cloud Platform networking and Elastic Compute Cloud-VPC is the concept of Live Migration. Under normal circumstances, all hardware in any data center—including Google—will eventually need either maintenance or replacement. There are also unforeseen circumstances that can happen to hardware that can cause it to fail in any number of ways. When these events happen at Google, Cloud Platform has the ability to transparently move virtual machines from affected hardware to hardware that is working normally. This is done without any interaction from the customer.

From: https://cloud.google.com/docs/google-cloud-platform-for-aws-...

I've always thought that live migration feature is incredible.

I am sure Google Cloud has VPC stuff for all the components, including Cloud SQL. It lacks the variety (No PostgreSQL, Oracle, MSSQL, or MariaDB). All traffic is encrypted by default on all posts (Google Cloud does this for free) too.

Look no further (for item 1 of 2 anyway) —

http://googlecloudplatform.blogspot.co.nz/2015/12/the-next-g...

Can't emphasize enough how much I wish GCE had PostgreSQL support. Managing and maintaining our database is something I'd love to hand off to my cloud provider. MySQL simply doesn't cut it these days, and RDS has made me happy and lazy.

can't agree enough. MySQL's quirks make me want to gouge my brain, and it's not going to solidify into a competitive db engine any Time soon. Let's say mariadb added something like hstore--would I ever get that patch?

I believe parent was commenting about ability to run a CloudSQL instance with an internal IP in the same/different subnet as rest of the Compute Engine instance. I'm drawing that conclusion because I've had the exact same use case and I ended up using SSL certs used by Cloud SQL instances to secure communication between my apps running on compute engine and CloudSQL.

With RDS one can create a postgres/mysql instance that shares the same internal subnet thus negating need to open it up to external IPs, something I had to do after much deliberation because whitelist individual instance IP is just too much pain.

[edit: I looked at the link, and created a test instance to make sure I wasn't missing on a config setting, still unable to use private network to communicate with Cloud SQL AFAIK]

> Google has long been a thought-leader in this space, and this shows in the sophistication and quality of its data offerings. From traditional batch processing with Dataproc, to rock-solid event delivery with Pub/Sub to the nearly magical abilities of BigQuery, building on Google’s data infrastructure provides us with a significant advantage where it matters the most.

Big Data is the core strength of Google Cloud. Good to see this move by Spotify!

> What really tipped the scales towards Google for us, however, has been our experience with Google’s data platform and tools. Good infrastructure isn’t just about keeping things up and running, it’s about making all of our teams more efficient and more effective, and Google’s data stack does that for us in spades.

What I really really liked about Google Cloud is the ease of use. Spin up a VM, start Cloud shell, SSH into your instance, install a bunch of software and you will know what I mean. It's "Quality" Cloud.

At work we moved to GCE at the beginning of this year, from Linode after they were having stability issues over the christmas break. No complaints from us. So far have been very happy with it. We were considering moving to AWS, but to realize the same pricing as GCE we'd have to purchase reserved instances - the sustained usage discounts have been huge for us.

Additional things we liked: - gcs is way more responsive than S3. also, was fairly painless to migrate our S3 buckets to GCS via the web console. - peering with cloudflare lets us save on bandwidth costs! - network load balancer has shown itself to be very reliable and solid for holding open A LOT sustained websocket connections. - http load balancer has shown itself to be very capable of ssl termination & routing (love that we can route /api to our API servers, and the rest to our static servers). additionally, no pre-warming was required. when we did the datacenter switch, it was just 5 mins of downtime. didn't have to worry about pre-warming for our production load like we would have on ELB.

Things we didn't like: - salt-cloud's driver for GCE is still lacking. Can't specify disk size for provisioned storage. Can't specify local SSD storage either. Also parallel provisioning didn't work. Something about PyCrypto not liking the way it forked. Not GCE's fault - billing support non-existent unless you purchase a support plan. - documentation still needs some work.

Billing support is provided for free to all customers. If you had trouble with a particular ticket, please feel free to ping me.

Disclaimer: I work in Cloud Support.

Thanks for the reply! I don't remember the specific issue, but we had a billing inquiry and were told we had to upgrade to a paid support plan to get an answer.

Please feel free to forward the ticket to me at jani at google, and I'll look into it.

I tried to sign up for trial. Entered my credit card info. All cleared. Try to use bigquery and got some confusing error about billing not being set. Contacted support a couple of days ago. It said response time is 2-3 hours. Haven't heard since... Anecdotal? Yes. Makes me re-consider even trying google cloud offering? You bet.

GCP has locally attached SSDs on almost all machine types and up to 64 TiB of block storage, configurable. What features were you missing?

It's a shortcoming in the GCE salt-cloud driver not being able to specify those in a deployment configuration. Sorry if this was unclear!

Aha roger!

The big point of this move is the hit it will have on on-premises Hadoop offerings: Cloudera, Hortonworks, MapR. It's a massive vote of no-confidence in their offerings. Spotify are effectively ditching a 2000 server, 90 PB Hadoop cluster to go GCE. As Spotify say themselves, that's big news....

Spotifier here. This is an important point. I have nothing bad to say about Cloudera or HWX (disclaimer: we're an HWX customer - we've had a pretty good experience), but I don't really see a compelling reason at this stage to manage your own cluster(s) (HIPAA/regulatory constraints, maybe?)

Getting shared-storage and indepedently operated/scaled compute clusters on top of that storage isn't easily achievable with the standard Hadoop stack, and building that on top of HDFS is non-trivial.

In fact, I don't think large orgs like you (Spotify) really want independently operated clusters. That prevents easy sharing of data, causing data silos to appear. You really want to have true multi-tenancy, which isn't in Hadoop yet. Hadoop has worked more on Kerberos support at the cost of features like easy-to-use access control - Apache Ranger or Sentry anybody!?!?

Yahoo's Hadoop clusters operate in a multi tenant fashion for precisely this reason: to ease sharing of data between groups.

When did Spotify abandon their original peer-to-peer architecture?[0]

[0]http://www.csc.kth.se/~gkreitz/spotify-p2p10/spotify-p2p10.p...

Quite a while ago. Not sure on the exact time, but judging from this article, I'd guess sometime in 2014 https://torrentfreak.com/spotify-starts-shutting-down-its-ma...

Seems a great growth hack. Too bad it's not more.

I'd be curious if anyone knows the details about why: complexity, customer complaints, reliability?

Too much complexity for a world that's moving to mobile clients that can't be active peers of the network.

Seems to be a pattern: wimpy mobile endpoint devices drive everything to the cloud.

If so this is perhaps a dated phenomenon. Moore's Law is still a thing and today's generation of mobile devices are getting fatter and fatter.

Moore's Law has not been improving batteries, which is the real problem. Mobile P2P nodes need to be awake most/all the time, which is a real battery killer. Modern devices have improved battery life by being asleep almost all the time.

Depends on what those p2p nodes are doing. We do p2p on mobile but it's to allow the mobile device to act as a client to access an IoT device, not to force the mobile device to act as a CDN node. Spotify was doing the latter, which is more problematic.

Moore's Law eventually will matter, since faster and more efficient chips equal the ability to do more with lower power consumption. Batteries aren't getting better fast enough, but power consumption per fixed unit of compute is falling like a rock. As a result "effective battery life" for a given fixed workload is subject to Moore's Law.

The only reason mobile clients can't be peers is due to the metered network usage.

data transmission from mobile is energy-hungry. More so than compute.

Regardless of why Spotify phased it out, I think peer-to-peer is fundamentally the wrong way to deliver that kind of paid service. If I'm paying Spotify for the service, then Spotify should bear the cost of delivering it. Not me, and not other customers, in the form of increased bandwidth and power consumption.

Fewer than a quarter of Spotify users are paid users. The P2P model makes a lot of sense for a data-rich service like streaming music, where most of your users do not pay anything.

We host a lot of our applications on Google Compute Engine but have had many issues with even simple features available from other cloud providers. For example, Google's HTTP(S) Load Balancer offering does not support SNI [1], HTTP to HTTPS forwarding [2], or sticky sessions [3], making it unusable for us. The support we pay for has been pretty helpful, but all we hear is "we have a feature request for this but I cannot provide an ETA on when and if this will be implemented". Many of the issues in the Compute Engine public issue tracker [4] haven't been updated in many months or over a year in some cases. Additionally, it can be very confusing finding out which beta services can only be accessed via the gcloud tool and not the web console. Without the ability to even schedule a window for maintenance on our VMs, we often feel like we do not have the control we need over our hosting environment. Please increase your transparency around addressing Google Cloud issues, all the Docker updates have been great, but many of these issues are over a year old and still show-stoppers for some users. Thanks again for churning out new features/services constantly, but please help close out some of these older feature requests!

[1] https://code.google.com/p/google-compute-engine/issues/detai... [2] https://code.google.com/p/google-compute-engine/issues/detai... [3] https://code.google.com/p/google-compute-engine/issues/detai... [4] https://code.google.com/p/google-compute-engine/issues/list

We (Sojern) moved to 100% GCP as well this month, admittedly not nearly at the same scale as Spotify, but we did. Other than some growing pains - that GCP team is already aware of - this has been a very smooth migration for us.

We use Google Cloud {Storage, BigTable, Container Engine, BigQuery, Monitoring, DNS, Compute} and may be some services I'm missing. I'd be happy to answer any questions.

This was mentioned somewhere deep in a thread, but Spotify (and others!) will be speaking at our upcoming customer conference (GCP Next) in a few weeks:

https://cloudplatformonline.com/NEXT2016-schedule.html

You'll be able to livestream the sessions here:

https://cloudplatformonline.com/NEXT2016-Live.html

Disclosure: I work on Compute Engine (and I'll be at Next).

The GCE model is the future. Google knows this because they've already discovered it internally by running their own services. They know containers are he future and they built a lot of the LXC kernel updates that makes things like docker work.

Most people don't need the future yet. Really people are just trying to get out of the business of managing and scaling their hardware with as little work as possible. Containerization is much farther down the line.

So I posit the future GCE customers are not people "moving to the cloud" like AWS, but rather people "moving to the small, contained, service" -- indeed, it's current AWS customers.

Also on this: http://www.wired.co.uk/news/archive/2016-02/23/spotify-move-... and http://googlecloudplatform.blogspot.com/2016/02/Spotify-choo...

I notice App Engine isn't mentioned in their stack. I'm a bit worried that Google abandons it once Snapchat moves off it. It has already been in maintenance mode for some years now (save for Managed VMs). The writing was on the wall when Guido left the team. Apparently Google still uses it internally (as the recent switch to Monorail made evident), so perhaps it's safe for a few more years.

With Lambda entering the space, I don't think they will remove app engine. Sounds like they should work hard on improving it.

Have you seen Google Cloud Functions? https://cloud.google.com/functions/docs

Disclaimer: Google Cloud Pub/Sub engineer here.

While Google could potentially abandon App Engine, it's much more likely it will simply remain unchanged for a number of years (as indeed it has roughly been for the past few years, save for Managed VMs).

GCP is now offering an increasingly larger percentage of what App Engine used to offer, so really the future may look a lot more like just build your apps on GCE + PubSub + Cloud Datastore and don't even mess with App Engine at all.

Hi pdknsk,

I'm a Product Manager on App Engine and wanted to reach out and say that 100% we are invested in app engine and are working to make it your favorite platform. For example, we heard that GAE documentation needs enhancements and we've kicked off a large project to do just this. It's an ongoing effort as you can imagine, but please take a look at the new landing page and managed VM content as well as the left-hand nav reorg we did. Here is the link. More to come of course.

https://cloud.google.com/appengine/docs

Please give the new set of docs a try and please give us feedback when you see errors or generally when the content is not up to par with what you expect. There is a feedback tool on the top right hand side of every page.

Thank you! Amir

Do you have any plans to launch a data center in India? AWS is launching one this year, and I imagine this will be a huge boon to the startup industry here.

As of 2016, AWS is the better bet.

True that Google stills uses it internally, but external usage exists way beyond Snapchat (although they are a fun customer). App Engine hasn't been in maintenance mode, although we have set aside engineering cycles to improve the reliability and stability of the service in order to ensure the highest level of support for production-grade customers. In tandem, we've some large features such Go, PHP, full-text search, and Cloud Storage support. We've added integration with the Cloud Console as well as Cloud SDK to allow developers to build hybrid solutions that span, for example, App Engine and Compute Engine. And, yes, we've invested in the App Engine Managed VM environment with a host of new runtimes including Java 8, Python 3, Go, PHP, and Node.js. We have more coming there over the next 2-3 months. Guido leaving was tough, I used to share an office with him. That said, I'm not sure that his exit was the writing on the wall.

Disclaimer: I'm the lead for the App Engine product team.

Not sure if this has changed in the last few years, but Khan Academy heavily invested in App Engine: http://highscalability.com/blog/2013/4/1/khan-academy-checkb...

To your first point though, streaming services typically are more focused on storage and egress (network) optimizations. App Engine has integration points with each, but its core workloads fall more in the dynamic web app and mobile backend realm. Within Spotify's architecture, there are places where App Engine makes sense (e.g. hosting their frontend APIs). That said, this is small relative to the rest of their architecture and understandably isn't the first area of focus when you're making a move to a new vendor.

>"...feature testing and more intelligent user-facing features," the Google staffer wrote.

I have to wonder if Spotify is doing this so that they can (eventually) use Google's deep learning technology in improving their recommendation engine. They purchased Echo Nest in 2014 to achieve this.

I have some questions based on previous experience many moons ago experience (3 or 4 yrs).

When I tried to port some of our platform to AppEngine I had some issues with various Java libraries (writing tmp files, opening sockets, etc). Once I sort of got past that and used some of Googles own APIs (which is sort of annoying that I had to couple my stuff) I had timeout issues. See we have to integrate with all these enterprise third parties (SOAP/REST) and these guys can be really slow. We also couldn't use our own pub/sub (AMQP/RabbitMQ) so that was going to have to be ported as well.

Things must have changed because its hard for me to imagine Spotify being able to get all their needs met on a rather coupling platform.

* How does one write code for Google Cloud while being agnostic of Google Cloud?

* Maybe appengine supports AMQP?

* Or maybe the container engine fixes these problems? (ie need something more custom run it in a docker image).

Our current platform runs on Rackspace and Digital Ocean. These guys are IaaS so your basically doing DevOps which is a pain... but Rackspace does have kick ass customer service.

AppEngine is Google Cloud Platform's PaaS offering. Its the oldest product on Google Cloud Platform, but its not one of the ones mentioned that Spotify is using (the Spotify post doesn't talk much about the products, but Google's [0] does.)

The Google Cloud Platform services they are identified as using are: Cloud Storage, Compute Engine, Cloud Datastore, Cloud Bigtable, Direct Peering, Cloud VPN, Cloud Router, Cloud Pub/Sub, Cloud Dataflow, BigQuery, and Cloud Dataproc.

So, its not surprising that they might be doing something that Google AppEngine doesn't support particularly well.

[0] http://googlecloudplatform.blogspot.com/2016/02/Spotify-choo...

I guess they are not entirely on Google Cloud but rather their data warehousing / big data stuff is now?/going? to be on GCP.

I don't really call big data processing "backend" but rather the microservices the backend. Its not clear to me if those are going to be ported over or if they are already.

Does spotify plan on porting everything over or just the heavy computing? A detailed techy case study I hope will follow soon as I am interested (the DevOps pain is very real).

> I guess they are not entirely on Google Cloud but rather their data warehousing / big data stuff is now?/going? to be on GCP.

I don't know why you would guess that: the Google announcement indicates that the migration has both a "services" track and a "data" track. And Spotify's announcement certainly seems to indicate that they are moving off of their own datacenters, onto Google Cloud Platform, and not in a limited way.

So... basically nothing is live yet? Are you from Google? Is anything live. I admit I got confused by the OP title: "Spotify moves its back end to Google Cloud".

Adding to what dragonwriter said, we now have an IaaS offering (Compute Engine) that launched publicly in 2013. You get raw virtual machines letting you be agnostic of Google Cloud's particular offerings.

If you're looking for "just run my app" but with a bit more customization / flexibility than App Engine (like running your own RabbitMQ) then yes, Container Engine (https://cloud.google.com/container-engine) is our hosted Kubernetes offering. You don't need to manage with the VMs yourself, just choose a cluster size and go.

Disclaimer: I work on Compute Engine, not really App Engine or Container Engine (but I'm familiar with them all).

This reads like they've been acquired by Google and are integrating. Now Google just has to write the check and own music streaming.

Hmmm...interesting point but Google already has an awful lot of music infrastructure in the form of Google Music and YouTube does it not? Would it really make sense for them to purchase Spotify too?

Buying brand loyalty, perhaps?

They'd be buying the users.

That's about the only real value. But then what? Do they run the two entities separately? Or merge Google's offering with Spotify and risk pissing customers off who are loyal to the Spotify brand and app?

No idea whatsoever. Maybe merge Spotify into Google Fiber and include it as part of a content offering?

I use Google's offering and it's probably pretty solid with YouTube Red, but I'm guessing at this point they're third behind Spotify and Apple?

Sure, one less competitor in the market.

I've only really looked at Spotify and Tidal, but Spotify is really good. There's more to streaming than content.

Makes me curious about how good Google's recommendation engine is.

As a long time Spotify user who switched to Google a few months ago: I really like the playlists and I've found a lot of new music that I like. The recommendations have worked really well. I'm not looking back.

I've also found that Google's radio stations are more diverse and less repetitive.

Just checked out Google Music on Wikipedia. No native desktop clients. Not even for Windows, and I use Linux. They can't be serious. Total dealbreaker.

I tend to always have a browser open so a desktop client doesn't matter to me. I keep Play Music open in a tab. Then I use an extension[0] to add the other controls I normally use.

[0] https://chrome.google.com/webstore/detail/prime-player-for-g...

I like it more than Spotify's. I found the repetition was much higher with Spotify, and never felt that it was "learning" what I liked. I don't know how many Mumford & Sons songs I thumbed down, but it would find progressively more obscure songs from them to slip into my Alt playlists.

Even when Spotify wasn't repeating individual songs, it repeated artists a good deal. This was over a period of six months until I got frustrated and switched to Google Play Music (what a clumsy name this is, though!).

With Google Play Music (GPM?), it throws a lot of different things at me in their generated playlists. I've had great success with the "I'm feeling lucky Radio", as well. I don't want to stop and think about what to listen to during a work day, so it's valuable to me to be able to start it in the morning and just listen in.

I thought this as well. They have. This is Androids answer to Apple Music.

edit: This also seemingly posted from a shelf account. Front page on HN, account 530+ days old, 0 comments, 0 posts.

Reads like an acquisition. I am not sure what to make of this as it could be a total coincidence that after a year and a half dmichel was really interested in spotify's backend migration, but either way I wouldn't be surprised if google is going to purchase or bid for it.

Maybe spotify moved unilaterally and is trying to position itself, could be a thouyght as well.

Do you have any source that Spotify has been acquired by Google? It sounds like baseless rumormongering to me. This wouldn't be a good way to announce such an acquisition anyway.

Sorry it was a flippant guess, which frankly I still stand behind as likely to be correct, but if unclear it was a guess.

Just an employee of spotify excited about the transition (I usually just lurk)

hahaha awesome! Glad you spoke up, exciting news. Just started using spotify a few weeks ago, but it is an awesome product. Cheers.

Why shell out for that effort before you get bought, though? Seems counterintuitive

Congrats to the sales team at Google Cloud. I wonder how much Google's new AI services played a role in convincing Spotify to use them.

Hey guys it's Tino. When will you contribute to our Big Data blog with your excellent ideas? :)

https://cloud.google.com/blog/big-data/

I am more than happy to contribute but I have a feeling you've mistaken me with someone else? :)

yup sorry posted in wrong thread.. but please do :)

I would be interested to get some more technical info. OK so you use GC, but how? What products do you use, how do you build your scalable infrastructure so anywhere in the world I can have all the music in the world? I love reading those type of posts from Netflix and was hoping to find something similar here

As I mentioned in a top-level comment (https://news.ycombinator.com/item?id=11159840), they will be speaking at GCP Next (which you can livestream) and should have some more detail to share.

Today: Highly publicized move onto Google Cloud

Later: Silent, gradual move away from Google Cloud after experiencing a number of issues and bugs, constant connectivity issues with services like Cloud Storage, silent failures that do not return error messages on a random basis, detailed billing info only available through arcane APIs and not the administrative UI, broken functionality and rigid inflexibility of BigQuery, and other non-obvious negative behaviors

I was wondering about Spotify's contributions to Hadoop stack such as Snakebite and Luigi. What will happen to these projects if they're moving to BigQuery?

Google Cloud has a managed Hadoop service called Dataproc: https://cloud.google.com/dataproc/

Spotify are not moving to BigQuery alone. They are also using Dataproc, Google's managed Hadoop service (which co-incidentally went GA yesterday), which lets them get value out of all the orchestration tools they already use, like Luigi. (They are also adopting Cloud Dataflow, Cloud Pub/Sub and other parts of our Big Data stack.)

Luigi has had BigQuery support for some time: https://github.com/spotify/luigi/pull/1002

Spotify will be talking more about the specifics of their data pipelines at GCP Next in SF in March, - I would expect to see a lot more on the new Google Cloud Big Data blog at https://cloud.google.com/blog/big-data/ too.

[Disclosure, GCP guy, etc]

Nothing in this move should affect contributions of Snakebite and Luigi. If anything, it will just make them easier to use with cloud environments in addition to bare metal.

With over 20K jobs/day, Hadoop will be a part of Spotify's data processing stack for quite a while. BigQuery is just a (awesome) piece of the full puzzle.

[spotifier]

Om Malik sees it as a first step toward acquisition by Google (or rather, Alphabet).

https://twitter.com/om/status/702194246489546752

This has been rumored before. E.g here's a headline from 2014: http://www.wsj.com/articles/google-considers-buying-spotify-...

Great now they can free up some engineers to bring their Linux client out of beta, which has been in beta since dinosaurs.

Or work on a successor to libspotify. Ha! Who am I kidding?

I've run a few small side projects on GAE (Google App Engine). I use it as a very simple and fast way to get an MVP off. My major issue is that their documentation tends to lag (sometimes way behind) when they change stuff, especially things that are integrated to other services they provide. For example after dropping support for OpenID 2.0, the documentation for 'Authentication Type' on Google App Engine was not updated to reflect this, neither was the documentation for securing your urls. This cost me considerable time and effort.

Hi NearAP,

I'm a Product Manager on App Engine and I truly apologize for the trouble you went through when you tried the platform. We have kicked off a large initiative to update and enhance our docs, and we just recently launched our new App Engine docs, an ongoing project of course. You can check them out here:

https://cloud.google.com/appengine/docs

Please give the new set of docs a try and please give us feedback when you see errors or generally when the content is not up to par with what you expect. There is a tool on the top right hand side of every page that explains this.

Thank you! Amir

Now how about Spotify fixing their front end so that Premium customers on Android can just hit "play" to play the album they want to play, in the right order? And how about an app in the Windows Store so that you can use it on a touchscreen? And how about some improved metadata that work for classical music? And a way to report corrupt or mistagged tracks and get them fixed?

/rant

I'm on mobile so can't directly link you but there's a support form for getting songs fixed.

I have never found a form, but I have found forums where people complain about persistent issues and nothing gets done. They seem to go out of their way to make it difficult for users to report issues. In addition they seem to have contempt for user experience, or they're incompetent at it, or they are under weird nonsensical licensing pressures that require them to make things difficult for their users. Their front ends need an overhaul and they could easily introduce a "flag this track as having a problem" feature.

I had no clue spotify operated their own data centers.

Does anyone have a link to that compares google cloud vs aws on a per price + per service basis ?

Last year, we commissioned an independent 3rd-party vendor to prepare a whitepaper comparing GCP pricing with AWS: https://cloud.google.com/files/esg-whitepaper.pdf

We also have a handy guide for AWS professionals who are looking at Google's cloud: https://cloud.google.com/docs/google-cloud-platform-for-aws-...

[Disclosure, I work for GCP, but I love all TLAs equally]

There are some half-truths in that whitepaper, like "reserved instances cannot be resized" (They can, within the same "class" of instances)

there are a few resources out there, the most recent one: http://www.zdnet.com/article/what-google-says-to-aws-price-c...

But GCP also has a pretty nice pricing calculator: https://cloud.google.com/pricing/tco/

Cheers,

I wonder if this has to do with Amazon's streaming music competitor being more of a threat to Spotify than Google's?

Amazon's streaming music competitor is awful. The apps are bad, the web UI is bad, the selection isn't as big, and the audio quality is middle-of-the-pack. In general, it's a "oh right, I have that Amazon service because I have Prime"; a similar comparison would be Prime Video to Netflix.

Spotify is better off in most ways (their desktop app still sucks, but the web UI is better).

I wouldn't judge a competitive threat solely on a V1 implementation, given it's from a company with engineering chops like Amazon.

The short and sweet of my user story as a hobbyist fullstack developer and wannabe data analyst is AWS has standalone services for your apps that either don't exist on GCP or are better than what GCP offers.

I'm talking about pay-as-you-go compute functions with Lambda (I'm aware that GC Functions are on the way, Lambda has been out for more than a year already), managed Postgres with RDS, and managed task queues with SQS. But GCP's console UX is killer compared to AWS, and using BigQuery feels too good to be true compared to how difficult it is to accomplish the same thing with Hadoop/Spark.

A little birdy told me that if you find a bug in GCE's live migration they'll buy you a bottle of scotch. Apparently the team gets blamed a lot for problems that turn out not to something else. :)

I'm curious regarding the economics of this but could anyone approximate the cost associated with this move or expenses going forward. It sounds like this would be a pretty big account for Google.

You can bet your bottom dollar Spotify are getting heavy, heavy discounts. Google are ramping up here in Stockholm, partly to get this account. This has been in the works since before last summer, so they've taken time before announcing it.

The precedent for Spotify was switching from Cloudera to Hortonworks a couple of years ago. Rumours are that they got their licenses for close to nothing from Hortonworks, they were so desparate to close them. Same rumours going around vis-a-vi GCE...

Just with respect to some details: Spotify changed from being a self-supporting (free) user of Cloudera's open source platform to being a paid support customer of Hortonworks. So, it was never actually a Cloudera customer (despite what was widely "reported").

Disclaimer: Cloudera emp here.

Nice, reinforces the point that Spotify likes and Spotify gets discounts on infrastructure :)

Still no IPv6 on Google's cloud... why?

Has anyone seen stats published on how big of an infrastructure Spotify runs?

Or how it compares to Netflix, which runs on AWS?

Sorry but spotify's infrastructure is not a shiny beacon of correctness.

They've compounded mistake upon "Oooshiny lets use that"

here is a post where someone dissects spotify's infrastructure bit by bit: http://www.secretbatcave.co.uk/software/docker-and-you/

This isn't an anti google compute, its got some really great features. But, spotify's backend infrastructure ain't one

Yes, I appreciate that that post is from a while ago, and they might have completely paid back that massive technical debt/innovation token debt.

To be fair, the presentation that that article is using as a reference for Spotify's infrastructure was from 3 years ago, and during most of the presentation I talked about what was wrong with how we were doing things.

How we do things is quite different now. Check out my DockerCon 2014 presentation or our tech blog for details.

Spotifier here. For the record, I hate puppet with the fiery intensity of a thousand suns. It's also pretty hard to magically make go away. I could probably do a whole talk on why puppet is difficult to kill, it's the kudzu of config management. I think it is the worst.

We've got our warts and a pile of tech debt, and I wouldn't want anyone to think otherwise. Containers are a part of a long-term strategy to get away from puppet and onto more idempotent units of deployment, and move a lot of what is considered to be "configuration" back into the build process where it belongs.

Is there any public available information where we can read about deployment process in the company?

Does Google Cloud offer an equivalent to AWS Direct Connect?

Yes. We have both "Carrier Interconnect" (https://cloud.google.com/interconnect/) and "Direct Peering" (https://cloud.google.com/interconnect/direct-peering).

Isn’t it risky to move to competition's infrastructure?

IIRC, Netflix predominantly hosted on AWS. It's definitely a consideration, but focusing on core competency isn't bad either.

Not really - many people rely on Google Cloud and if Google made it harder for Spotify to compete (via, say, raising prices for Google Cloud for them), they'd have to be pretty specific to Spotify to not cause other customers to move to AWS or Azure.

Spotify has much bigger risks in their licensing deals.

cost it not a problem and google wouldn't do it. I'm talking about stealing recommendation algorithms and models. Have a peek new features before release etc.

Seems googlers are down voting massively.

Recommendation algorithms and models don't do you any good if you don't have users.

Google Music has a lot of users?

(Serious question) Isn't it risky to rely on any Google product as a foundation for your business?

They've shut down more popular services that people depend on than Google Cloud. They tend to enter a market, undercut all the competitors, and then once they are dominant in the niche they terminate or change the service in a way that makes it not fulfill the original promise.

Well Amazon killed the Fire Phone though ;-) https://fortune.com/2015/09/09/amazon-killing-fire-phone/

Jokes aside, probably you are thinking about free products like Google Reader, because this:

> once they are dominant in the niche they terminate or change the service

would not make any sense if the product in question makes $$$

I think I could try moving to GAE if they had Postgresql offering for database storage.

I wonder if they are still using Cassandra or moved that over as well.

Why google over aws? Why isn't this man answering the real question?

I always used to think Facebook would make a move for Spotify..

-1 :(

Zuckerberg used to wax lyrically about Spotify and I always assumed Facebook would one day like to move to the subscription streaming market. A barrier to entry for Spotify to stream movies I always suspected would be cash.

Moving to google cloud won't stop a Spotify sale but it may discourage some?

Smells like a puff-piece

A puff piece would be if some tech reporter wrote about the wonders of GCP.

This is the Spotify team, and they've clearly put their money where their mouth is by actually making the move.

Marketing blog spam is the new puff piece.

Would you prefer testaments like these from Spotify engineers?

https://twitter.com/plamere/status/702168809445134336

https://twitter.com/plamere/status/702169733454495745

https://twitter.com/mpjme/status/702167231623516160

Kind of. We all know that companies cut deals with providers in exchange for blog posts, tweets, and other sorts of promotion. I'm always wary of posts like these, especially when they lack detail or awareness of competing products.

I think if Spotify dressed down Google Cloud Platform competition, no matter the merit, there would be many more complaints (ex. see comments on this technical post https://news.ycombinator.com/item?id=11029032).

Spotify did say that they really enjoy specific aspects of Google Cloud Platform, through their own engineering blog. Spotify engineers have also been independently vocal on Twitter.

Netflix has been very similarly vocal of their support of AWS.

I'd love to hear your thoughts on a better mindshare model, perhaps we can all learn to do things a bit better.

Right, I don't think they have to directly attack anyone. It's just a matter of tradeoffs - here are the things the Google Cloud gets us, here's a specific example that hones it in, and here's why that's critical to our business.

At the moment, the blog post just says, "Yay we moved to the cloud" and threw out some marketing buzzwords. It should be beyond a shadow of doubt that they got a lower price as a result. That's ok - now give us the details! On a product level, what made Google Cloud so compelling that you were willing to move your entire infrastructure to it? That question really wasn't answered, and is really the biggest thing, as a customer, I'd want.

Right now, I'd have to call Spotify as a reference in order to do to get that answer, which again allows them bring the price lower. Not necessarily democratize the information.

Spotifier here. Frankly, price is not the biggest factor in a decision like this. If we were going for the lowest cost cloud option, it probably wouldn't be either AWS or Google - there are other providers who are hungrier for business that would be willing to do deep cuts at our scale.

The way we think about this is that there are basically two classes of cloud services: commodities and differentiated services. Commodities are storage/network/compute, and the big players are going to compete on price and quality on these for the foreseeable future (as with most commodities).

The differentiated services stuff is a bit more interesting. Different players have different strengths and weaknesses here - AWS has way, way better capabilities when it comes to administration and access control and identity management, for example (which is actually pretty important when trying to do this in a large org). The places were Google is strong (data platform) are the places that are most important for us as a business.

Compelling: dataproc+gcs, bigquery, pubsub, dataflow Made it safe: high-enough quality, cheap enough.

What more would you like to know?

Hey nrh,

Nice! Did you have the data tooling built out before you went to Google Cloud? If you did, I could imagine the migration was pretty hard as well.

Also, all of those seem relatively possible with AWS Redshift, Kinesis, and Data Pipelines. I'm interested what Google Cloud had to offer, spec-wise.

I would categorize data tooling as a moving target - we have some, it's never enough, it probably won't ever be enough. It's a moving target (p.s. obligatory we're hiring!).

I think this Quora post does a good job of redshift vs. bigquery: https://www.quora.com/How-good-is-Googles-BigQuery-as-compar...

More generally speaking, (excuse me for being a little hand-wavey here) many of the AWS offerings feel like polished, managed versions of familiar tools. Redshift, for example, feels a bit like "hey we figured out how to abstract away a bunch of mysql instances to feel like a big processing cluster". That's not a bad thing, necessarily. The google stuff feels much more intentional - "we need to solve the problem of doing these sorts of queries at scale" vs. "we need to solve the problem of scaling mysql to solve these types of queries"

Maybe they're just better at abstraction, but whatever - that works for me!

So it sounds like BigQuery was the deciding factor in this case?

I'd say the data platform overall, bigquery is certainly great . So are some other bits.

Awesome, good to know. Thanks!

Are you dropping Cassandra?

rolls eyes I have been playing with their stuff internally for a while now and no one told me that we would get a discount for a tweet. I tweet about shit I like. There is no giant conspiracy.

http://www.paulgraham.com/submarine.html

In the same way that facebook shut down parse, I am afraid that google is going to shut down google cloud. And reluctantly have become okay with relying on AWS.

Google Cloud interacts with the core of Google's business (web services that needs scalable data infrastructure) in a way Parse never did for Facebook.

Google's core business is ads. Parse makes a lot of sense strategically for facebook, a big moat that they have built is around being your identity on the internet. Parse's platform encourages app developers to use facebook login as a core component of their apps. So even despite being ancillary to their business, they shut it down. Building a technology company around an ecosystem is potentially a lot riskier than people take it as. Especially one with a lot of proprietary bells and whistles..

Google's core business is providing web services and putting ads on them. That core business means Google focuses a lot on running those web services effectively, which is why they are now focusing hard on the business angle of running other people's web services.

Google spends $2+B per quarter on capital expenditures; where do you think that money is going if not datacenters?

Using Facebook as a default login was never core to Parse (e.g. PFUser: https://parse.com/docs/osx/api/Classes/PFUser.html) nor is Parse nearly on the same scale as Google's investment in its datacenters / web software efficiency.

And at some point your company might decide that it's already its biggest customer for its cloud products and close it to the public.. or it might not.. but it's up for me to decide where I host :)

Except that in November, we got Diane to come take over all of our Cloud and Apps business:

http://www.nytimes.com/2015/11/20/technology/google-picks-di...

Edit: Formatting (didn't mean to quote the url)

Reply on news.ycombinator.com