LinkedIn shelved plan to migrate to Microsoft Azure cloud
cnbc.com
cnbc.com
There was no attempt to "lift-and-shift" anything. There are technologies that overlap and technologies that conflict and technologies that compliment one another. As with any huge layered stack, you have to figure out which from the "LinkedIn" column marry well with those in the "Azure" column.
I personally appreciate LI management's ability to be clear-eyed about whether the ROI was there.
Then came the move to AWS and Azure, again because the business case made sense. A big part of this was Microsoft being good at selling their products in a bundle, so that getting into Azure sort of just made sense because of how well the IT-operations side benefitted from moving into things like Azure AD (now Entra ID for whatever reason), Azure Automation instead of scheduled tasks and so on. If your IT-operations infrastructure is already largely in Azure then your development will also be in Azure because it’s going to be such a tiny blip on your Azure bill that nobody will ever notice.
With the price hikes, and also just how complicated it is to actually run and maintain IT-Operations in Azure (I can’t speak for AWS) we’re now in a world where more and more organisations here in Denmark are looking to get out of Azure/AWS to reduce cost, and many have already made the move.
Of course no one is leaving Office365, so many of us are mostly just trying to broker better deals with Microsoft.
It's an endless loop, I think. I've observed a very different behaviour in other parts of the world, where companies which either migrated away from the cloud already, or companies which never moved to the cloud to begin with, are now moving things into the cloud and other managed services in order to better focus on their core competencies.
Dropbox, for instance, shifted everything out of AWS in 2014 -- but are now moving a lot of their corporate infra back into AWS in order to focus on effective and efficient storage, which is what they're good at, without having to build everything else themselves. Likewise hedge funds - if you look at job postings for a lot of systematic trading firms (e.g. Jane Street, HRT, 2Sigma, etc.), you'll find loads of them are hiring extensively for people with cloud experience, because there's been a sort of wake-up moment where companies are breaking free of the idea that everyone can be Google, and realising that you can't build everything in-house if you only have 1000 employees.
So, interesting to hear of a move in the opposite direction in Denmark. I'm sure that ten years down the line, a new generation of engineers will be coming in and asking, "Why in the world are we managing all of this infrastructure on our own?"
So anyway, more like the reverse: cloud is colocation but expensive.
You can certainly use colo to provide the facilities most need from cloud infrastructure.
Just stuffing a server in someone's rack isn't cloud, but larger installations are often managed in similar ways.
The questions become:
1. Does it make sense to dump our special features in the stack, or move them to a higher level in the stack? 2. Does Azure have comparable capabilities for the LinkedIn stack? 3. Is LinkedIn worth it to Azure to sell to?
---
Often times, "at scale", you can support custom solutions outside of cloud providers that are purpose-built, and often times more resilient and efficient than the cloud providers.
AWS has taken a very interesting approach of building an incredibly wide set of solutions to support every customer under the sun, and their approach to being "customer obsessed" leads to them building super niche solutions if the deal is worth it.
I'm not sure how Google and Azure handle these engagements.
Microsoft will allow you access to the PM of the product if you are big enough, but his answer will be a not-exactly-committing "in 3 years" because their roadmap is full.
Microsoft owns both, so that question gets convoluted. It's not really about whether LinkedIn pays Azure enough to care.
The flip side is that it is freakishly expensive to keep doing that, and telco is not a business that is freakishly profitable.
I'm watching efforts like Azure Operator Nexus (https://learn.microsoft.com/en-us/azure/operator-nexus/overv...) which looks to be taking a stab at this, but even there, its more like OpenShift on specific hardware with some Azure dust, rather than a remaking of Azure primitives like redundancy for the telco universe.
I haven't followed what's happened since it was spun out, but I know that Network Cloud was heavily OpenStack based so I'd would be interesting to see how that may have changed but I no longer work in the Telco space either.
[0]: https://azure.microsoft.com/en-us/blog/improving-the-cloud-f...
> vs ""modern tech"" FOMO
Very tough decision, I'll need two MBA degrees to figure this out.
The folks who are too senior arent sufficiently technical to understand. The folks who understand benefit from the key-person risk and premiums paid, and thus are disincentivized to challenge the ROI comparison.
That's another great point; with the on-prem system where we run our own massive k8s/vm/data lake cloud, every team just throws things at it willy nilly and even though we know how to calculate total operating cost it's very difficult to set up backpressure to individual orgs and their budgets so there is quite a bit of waste I feel. With something like AWS the billing system is very detailed and is easy to attach to teams upfront. I'm not sure if it's good or bad that things are more efficient due to teams being incentivized to pay attention, but I kinda like it.
Costs more money but easier on the cash flow.
think very hard before you start building your cloud-native application on a specific cloud
They were simply saying that much as you can love one platform there are many valid reasons why your deployment on another makes it hard to shift that and recapture that configuration. Nowhere did they intimate it was a fault of the current platforms capabilities...
> At LinkedIn, we are focusing our efforts on advanced automation to enable a seamless, LinkedIn-wide migration from Rest.li to gRPC. gRPC will offer better performance, support for more programming languages, streaming, and a robust open source community. There is no active development at LinkedIn on new features for Rest.li. The repository will also be deprecated soon once we have migrated services to use gRPC. Refer to this blog for more details on why we are moving to gRPC.
Kind of confusing statement as they threw away tens of millions of dollars in a failed attempt to migrate.
It actually means the opposite. Instead of management identifying a boondoggle ahead of time they got the company into a mess.
Emphasis on knowing. I’m sure some people thought it was a bad idea ahead of time. But there are always people who think any new idea or big change is a bad idea.
The reality is that anything significant has to be tried out, for real, to find out. That’s the premise behind proven useful decision frameworks like agile or MVP. And as the GP says: good management shows up when they read the attempt clearly and resist the sunk cost fallacy.
Bad management sounds like “let’s never change anything” or “we’ve already come this far, we need to finish it no matter what.”
And to a large enterprise, tens of millions of dollars is not an existential amount of money to lose on trying something significant. Big sports teams lose that much on bad player moves all the time. “We can’t ever waste money on trying new things” is a great way to sink deeper and deeper into whatever rut you happen to be in at the time.
Literally your primary job as an executive. Strategic knowledge and wisdom
If wasting tens of million on a bullshit initiative isn't an example of bad management then literally nothing is.
The only place these types of terrible managers can exist are too big to fail companies like LinkedIn and the Government where theres zero repercussions for poor management decisions because money keeps pouring it.
In smaller leaner companies where money is actually important executives actually have to be good at their jobs.
> "We can’t ever waste money on trying new things"
There's a difference between a good strategic decision and starting a "facebook for pets".
Some ideas are terrible at the outset.
And executives whole reason for existence is to make this determination.
Blueshift was an objectively terrible decision and whoever made that decision is objectively bad at their jobs.
The hit to Azure's brand for this is probably significant. Who was looking at a migration to Azure before who's now discouraged?
LinkedIn's been extremely independent of Microsoft and still are just like GitHub and openAI.
This was a 100% internal LinkedIn management decision.
there are no competitors
I cant imagin being owned by MS they would be allowed to use Google cloud.
I've seen massive projects that have been train wrecks since day one pushed through until the bitter end because management can't admit failure.
Recognizing you fucked up and only cost the company tens of millions instead of hundreds of millions is "Not Terrible" management.
A major hospital has been running it for a year with lots of issues, some which has certainly affected patients gravely. It was rolled out despite heavy protests from doctors and nurses who had seen or experienced the new system at the smaller pilot hospitals.
Yet management is pushing on, and just decided the next big hospital should get the new system this spring, despite the many severe issues that are still unsolved.
I think it’s a pretty successful operation when you can make the exploration and come to the conclusion that it’s not a going to work, then pivot.
Sticking to the decision no matter what would be much worse.
Totally worth it. Your entire org also learned in the process.
Companies, teams and people do research and test projects all the time to determine direction. Making a decision like moving the entire LinkedIn data center is just a larger version of that.
you're saying they're good managers because they didn't lose billions.
going by your "logic" its literally impossible for management to make a mistake because there's always some slippery slope they didn't slide down.
"""we put the company out of business but at least we didnt cause the collapse of matter into a black hole! we deserve some bonuses!!"""
this is how people who work at monopoly tier companies actually think lol
> Any amount of employee time spent exploring options that aren’t pursued could be framed that way.
Employee time costs money. If 20 people are in a meeting for 1 hour, that 1 hour meeting costs the equivalent of 1 hour of each of attendee's salary. 20 people attend a meeting making $100 / hour, you just spent $2,000 for that meeting.
All time spent planning, also costs money. Every Agile "spike" to do research for a story costs money. Every architecture document. Every time you have somebody do a couple of prototype projects to test your options before you make a decision. Sometimes you can't easily find out what's involved until you really dive into the work. It's the nature of the beast.
For LinkedIn, purchased by Microsoft who also owns Azure I'm quite certain that during the process they assumed consolidation within Azure would be on the table. If anything it was probably expected to be a huge marketing push for Azure to be able to say that LI ran on Azure so it's likely a management decision that involved a lot of pressure to make it happen.
Being willing to abandon that decision after exploring to see what was involved is a big deal. In many companies you'd have just heard platitudes like "well we have to do it, find a way" perpetually extending the project while taking away from more valuable initiatives.
I used to work for a business telco that was acquired by a residential telco. They were so convinced they could roll this newly acquired company into their existing systems that they ran off all of the staff who built the systems. 2 years later, they finally realized that business telco was a lot more involved but by then it was too late.
I understand the thinking that got them there, but despite a lot of people trying to stop them they proceeded forward until their hand was forced. This LinkedIn decision could have gone much, much worse.
Yeah but spending several years and many millions of dollars isnt 'exploring'.
I'd say it's a solid fail for the management.
But we can agree to disagree on this. I see you strongly believe that losing tens of millions is good management.
For what it's worth there's many posts nowadays on blind about how the culture of LinkedIn has tanked and I'm assuming management probably believes they're doing a great job in that area too.
> I see you strongly believe that losing tens of millions is good management.
Costs are relative to size. There are a lot of people involved, it's going to cost a lot of money. Microsoft's operating expenses are estimated to be just shy of $350 million per day. At that scale, 10s of millions is not as impactful.
Could it have been done better? Of course.
Could it have been much worse? Yep. Significantly.
Also as part of Blueshift the plan was to do batch processing first but LinkedIn had a culture belief in colocation of batch compute & storage, which is against the disaggregated storage paradigm we see now. IMO this led to some dragging of feet.
Source: Worked at LinkedIn 12 years, am a director at Databricks now.
[1] https://engineering.linkedin.com/blog/2021/the-exabyte-club-...
Azure is developing its own inhouse message broker that they call EventHubs. And while it is able to speak the Kafka protocol it has some weird limitations and quirks¹. I'm wondering why they are so hellbent on developing their own inferior version of something when they could just use the real deal and host it as SaaS.
¹ No support for KIP-71 (Enable log compaction and deletion to co-exist), limited to 10 topics (Azure lingo: EventHubs) per broker (Azure lingo: EventHubs namespace), mandatory client config parameters that you must not forget to set, even though the documentation just calls it "recommended" (https://learn.microsoft.com/en-us/azure/event-hubs/apache-ka...), some weird protocol level incompatibilities that are nowhere documented, for example a consumer group state transitions to "Dead" after the last consumer left instead of "Empty", and auto scaling that only ever scales up, but never down.
This seems to imply Event hubs is some sort of upcoming beta or something. Event Hubs are one of the oldest parts of Azure IaaS - I think it was launched in 2014?
https://en.m.wikipedia.org/wiki/Embrace,_extend,_and_extingu...
We were then given the directive that everyone at my level would need to get some certifications so we could properly use Azure, assist the architects and more jr devs. It’s a good idea but my god the training is so poorly executed. I want to like Azure but it also seems like an uncoordinated mess.
Maybe I’m just a grumpy dev. Anyone else have a better and more positive perspective? Who has good training for certs such as the Data Engineer or AI Engineer?
For a long time this is why Azure has been beating GCP. GCP has been adement (not sure if still are) that prices are “what you see is what you get) and hope that small projects will suck you in. Microsoft however would wine and dine execs and then offer 50% discount on a xx million commitment.
I agree on training and the like. Azure feels like such a mess.
I assume if you're bigger, you probably get an even larger discount.
Yes and no. GCP is going after the non-cloud products instead of the big cloud providers now. So going after Snowflake customers with BigQuery.
It's literally a "me too" cloud.
Azure is notorious for not being "finessed" out and god speed if you have to use AKS.
If your apps are simple enough that you can get away with Azure Container Instances/Apps, use them.
If you need proper managed k8, use OpenShift. Microsoft has a partnership with RedHat and RH provides full support.
I don't think the age of the article matters.
https://customers.microsoft.com/en-us/story/1536483517282553...
Like there's a litany of literature of how AKS completely dropped the ball, you don't have to take anyone's word on it from this thread. The difference between AKS and other offerings is _striking_.
I know MSFT likes to sell that as "complete and flexible configuration" but more often than not it's like: if your prescription to use this one service is so rigid as to use these four other services in such a way then when not just package them completely as such?
1) Be comfortable routing traffic between on-prem and cloud over the internet or at least over a VPN, and
2) avoid the temptation to build your own platform (Terraform templates are a liability, not an asset!) and
3) Move tiny stuff first and bigger stuff later.
It’s amazing how many companies fail at that.
It’s like politics; the best person for the job is a sufficiently experienced person who does not want it.
I remember when S3 and EC2 were coming out. They tried to make us working on the retail website move year after year. Our excuses became their roadmap. We (developers working on not AWS) really didn’t want to move to AWS. It was 100% worse than what we had before.
“Network is slow”, “it’s not PCI compliant”, “the drive is unreliable”, “the load balancer is missing a feature”.
It took years and years. But they did it. Google and Microsoft don’t have the willpower to force this, and its why they will always be behind AWS.
When Google tries to sell you Google Cloud, remember they don’t use it for anything internally. They don’t think it’s good enough. So why should I?
Where do you think Teams, Bing, OpenAI, the billion internal systems they use and Xbox are hosted
It seems while some of the features implemented pre-Azure were supported, but not fully working as intended because they were sets of features that weren't necessarily in use by Microsoft themselves. But they became an issue as Microsoft was now interested in it working properly after they started using that capability, or parts of the Azure infrastructure depended on it, or some big money customers "'how about'ed" these features. I'm talking minor but important aspects of larger products, or Interoperability.
Windows Server had a lot of life breathed into it post Azure. I'm too lazy to find specific incidents, or recall the past annoyances as a sysadmin of years past (PTSD?), but they absolutely dogfood and we are all better off about it. What they can never dogfood however is the pricing.
I'm k-12 education sector (almost 200,000 staff and students to our district) and we get what I would imagine one of the sweetest of discounts, and if we were to move all of our operations to Azure, our TCO actually increases to maintain the same level of service and availability.
That being said, the performance hits aren't that terrible, but it's the little things that add up quickly at scale.
Of course, as public funded k-12, it's sometimes a decision that gets made when having to decide on buying new things for the data centre (capital cost) vs. subscribing it to Azure (operational cost). The money for both piles comes from the same vault but there destination pile is loaded with implicit reasoning excuses. For example if we spend $200,000 a year on a service as a subscription, it's easier to get money for that versus requesting $400,000 on something that would last us 3 or 4 years.
It takes great leadership at the IT level to liaison this impact to the business movers and shakers, and sometimes that's not there.
It doesn’t mean they will never use Azure, just that they’re being rational about what to use.
It would be very different if they designed GitHub today and didn’t use Azure.
Most groups in Microsoft use the Microsoft stack as it's best supported and already compliant and works pretty well - so most groups use teams, office, Azure etc.
Groups are free to use other tech and may for example use zoom+slack (some groups do) or GSuite. This is also true for technology (e.g. a team can use yarn and flow instead of npm and TypeScript). It's just very rare they do.
Source: msft employee
Our stack today has us using AWS for domain registration & S3. We use Azure for everything else. We actually log into AWS by going to the Microsoft MyApps portal and authenticating via our AAD/Entra credentials. Microsoft's docs regarding how to set up SCIM/SAML auth to AWS are excellent [0].
In Azure, we use ~5 products: AAD/Entra ID, DNS, Azure Functions, SQL Server, Azure Blob Storage. That's it. There isn't really any VM/container presence in our go-forward infra. Everything is managed/'serverless' now. There are some VMs but they are supporting legacy systems that will eventually be turned off. We have ZERO custom networking. I couldn't explain how to manage an azure vnet to save my life. We don't need VPN tech anymore either.
Github Actions->Azure Functions is pretty much the end game CI/CD experience for me. I am not rocking this boat. I never want to try anything else. I've spend a decade of my life keeping some flavor of Jenkins alive.
Could we do all this "cheaper"? Sure. But at what cost? My mental state is something that is a non-zero factor in all of this. Keeping track of hundreds of moving pieces in production is simply overwhelming. It's unnecessary suffering. I need something I can explain in 20 minutes to a new hire if we want to remain competitive.
[0]: https://learn.microsoft.com/en-us/azure/architecture/referen...
Google does use Google Cloud internally for some things. Source: I work for Google.
I feel like you might not have read the article.
They spent years and god knows how many millions TRYING to move to Azure with the Blueshift project..before pulling the plug. They hired armies of contractors.
They didn't stop by choice.
They stopped because their tech stack is a giant over engineered unmovable turd.
The amount of bugs I troubleshot in that tech stack was staggering just for basic day-to-day stuff.
promotion based engineering means an engineer shits something out that looks cool in a demo then bails to the next cool demo project while the rest of us are stuck with the reality of the turd.
[0] https://news.softpedia.com/news/Windows-Live-Hotmail-Was-Pow...
The whole migration of LinkedIn from their own data centers to the public cloud (Microsofts) isn't going well.
It appears they are still going to operate on-premise for many things. Some things moving or have moved to the public cloud.
Isn't this more a shot at the public cloud for all the things than to any specific one?
It’s possible given what they have, I t’s simply best to keep it on premise - at least to some degree. That would likely not be true with a successful re-architecture, but not everyone is up for that.
There are so many unknowns about how things are setup that it's hard to know.
The last 15 years possibly broke this rule by virtue of low interest rates, enabling the justification of large internal teams focused on modernization efforts which sometimes went as far as moving the state of computing forward.
I wouldn't be surprised to see legacy enterprises return to form now that interest rates are 7%
Having personally tried to be cheaper than the cloud in 2013 through large hardware buys, negotiated contracts etc. I found that the roi relative to a progressively discounting cloud weren’t there.
https://www.cnbc.com/2019/07/23/linkedin-is-moving-to-micros...
Strategically, I don't think you try to help someone with cloud until they've burned out their ego trying some grand delusion. Even one bad actor can throw the whole thing off. Cloud is about doing more with less, so committee bullshit is cursed.
Once you've got your customer completely humbled by reality, they be willing to listen to reason and you will save so much frustration.
We've got a huge one in the pipeline right now. They've been trying to "go to the cloud" for about 5 years now and executive management is ready to reset the entire tech org chart over the lack of progress.
Cloud native is the solution but many technologists perceive it as the end of their careers. Anyone pitching a "cloud native" solution that still has container spam managed by the tenant owner is either incompetent or trying to protect their career at this point.
Without losing data or disrupting the customer?
Or do the databases just stay in the data center and not migrate.
jk of course you never do any of the other stuff after turn off writes to the old system you just leave the FF turned on to 100% and never think about it again :-p
If you can afford downtime, or data loss that does make it easier. But that's not an engineering question, more a product one.
We used one of our ETL clusters to ship data to MSFT for various LinkedIn integrations, like seeing LinkedIn profile information in Outlook or Office products.
Also 26B is just initial investment, MS surely invested more money in the division in last 8 or so years, Linkedin was not exactly highly profitable entity in 2016, while it was not burning a lot of money, growth experienced in last few years would have needed additional investments in the business.
I don't have specific opinion on whether MS overpaid are not, just want to point out even 100B valuation today does not necessarily mean its high RoI for MS yet.
As far as M&As go, it is very successful outcome, vast majority of them fail spectacularly, RoI is not perhaps the right metric to judge a strategic acquisition.
Cloud platforms charge you much more than what the hardware is worth, the only advantages are that they provide a simplified resource management system (which is mostly an illusion, as any non-trivial system will require you to build in-house tech regardless) and the ability to scale more easily (which is done at a prohibitive cost, so a good reason to switch away if you need to scale up).
And Github also under Microsoft seems to be doing fine with on-prem as well. Why force LinkedIN to use Azure?
Blows my mind every time I see these kind of numbers.
[0] https://www.datacenterdynamics.com/en/news/stack-overflow-st...
Also Linkedin is 100x the size in users : 1B vs just around ~15M for StackOverflow.
All of a sudden the scale makes much more sense although I will have to fact check this because the number seems absurdly high for a professional platform.
https://news.yahoo.com/linkedin-now-1b-users-turns-125845570...
Facebook itself has been having significant loss of engagement in recent years. Instagram and WhatsApp are far more likely to have higher DAU than FB main app.
Not considering that 90% of all users are likely sleepy accounts.
Of course, I could also be entirely wrong, but I also am not going to pretend that IT resume padding then jumping ship and leaving a shart of an architecture behind doesn’t happen all the time in this industry.
Not like they are paying anyone to host something besides themselves.
You mean cloud is the holy grail where everything works and everything else is a failure waiting to happen? I claim BS.
What I mean is that in planning anything in tech you can’t always wait until problems arise before you take action to deal with them, particularly if the consequences of failure are extreme. The specifics of what is the right setup are going to depend heavily on the circumstances of the use case. Definitely being on cloud isn’t right for everyone, nor is trying to maintain on-prem hardware the right choice for everyone.
Because they’re owned by Microsoft and it looks bad if Microsoft doesn’t dogfood its own cloud infrastructure. Plus it’d be more economical long term if they didn’t invest in a parallel cloud infrastructure. They will migrate eventually, just apparently it’s not ready for them yet.
Their cloud infra is already mostly Linux. Bought company which already works fine not gonna add insult to the injury.
>"Plus it’d be more economical long term if they didn’t invest in a parallel cloud infrastructure"
This is big if. It may or it may not. Unless you are deep into their internal kitchen and have all the info your suggestion amounts no nothing more than a wild guess.
I think I need this translated back into tech-speak.
I’m not sure what else needs to be translated? Nothing, I think?
And then people also pick it up for non-persuasion, because it also sounds like a catchy name for an engineering approach we already had.
Of course it can still be used for persuasion for awhile, but will grow baggage over time, as efforts linked to the term don't play out that way.
The term didn't sound familiar to me (though the concept was), and the term might not have been familiar to some others.
People might not want to contradict an assertion because of language like "The term’s in wide use, talk to anyone involved with cloud-anything and they’ll be familiar with it. [...] I’m not sure what else needs to be translated? Nothing, I think?"
> > LinkedIn was having a hard time taking advantage of the cloud provider's software. Sources told CNBC that issues arose when LinkedIn attempted to lift and shift its existing software tools to Azure rather than refactor them to run on the cloud provider's ready made tools.
The only other terms I can see that are jargon are "cloud provider" and "refactor", and those are already technical (more or less) so don't need to be translated into technical language.
As for the other bit, I just meant that it's a widely-used term so one may continue to encounter it in these contexts. It truly is ubiquitous in discussion of and around "enterprise transformations" to the cloud, and among cloud practitioners more generally, so anyone connected to that space will know what it means. It's also kinda already a technical term, in that developer/devops and SRE sorts throw it around and do mean a specific thing by it, which doesn't need to be translated for other technical folks in that area.
The original person might've instead asked for an explanation in a way that didn't come across as criticizing the article.
But probably best not to insist that everyone should already know the term; just explain it.
It's usually done to avoid the engineering cost of making the services more cloud native. What tends to happen a lot is that after a considerable portion of the migration is completed, the cost of the lift-and-shift effort start to overtake the savings, and the projected costs, dwarf the future savings.
I suspect this is what happened with Linkedin.
In the real world, for baseline load, the big advantage for many large companies isn't price, but the massive lack of alacrity of many inhouse ops teams. If it takes me 3+ months to provision compute for the simplest, lowest demand services (as is the custom in many large companies full of red tape and arguments about who bears costs), letting teams just spin up anything they want and get billed directly is often a winner, even if it's more expensive. Having entire teams waste months before they can put something in prod is a very different kind of expense in itself.
The cloud native approach would be to modify your app so that it can be scaled up and down so you keep a few machines always running, and scale up and down with your traffic so you only run at capacity when you need it.
Sure, sure, you’re about to say something about discounts? Granted, that’s available, but only for commitments starting at one year or longer!
Okay, fine, I actually agree that there are savings available by reducing head count. The entire network and storage teams can be made redundant, for starters. Even considering that DevOps and cloud infra engineers need to be hired at great expense, this can be a net win…
…but isn’t in my experience. Managers are unwilling or unable to make many people redundant at once, so they stick around and find things to do…
… things like reproducing the mess that kept them employed, but in the cloud.
I’m watching this unfold at about a dozen of my large enterprise customers right now.
Got to get back to work and send the fifty seventh email about spinning up a single VM. Got to run that past every team! It’s no rush, it’s only been about fourteen months now…
Anecdotally, when my previous company was looking at costs, cloud unequivocally came out significantly more expensive, and that wasn’t even a large company (only 2,000 or so employees).
I will grant that we did not have globalization problems to solve (but I’d also wager that lots of businesses prematurely “what if” this scenario anyway).
If you neeed 4 CPUs for your peak load for 4 hours per day, and only 1 of them for the other 20 hours a day, you can save by scaling down to 1 cpu for 85% of the day.
It's also extremely uncommon to have loads that spiky.
And when you do, hybrid is often a solution (use a provider that can provide colo or managed servers for your base load and cloud instances for your peaks, or scale across providers).
Even at that, you said yourself that you can use "cloud" to scale into your spikes.
It takes very unusual load patterns for cloud to win on cost. It does occasionally happen, but far less often than developers tend to think.
There many reasons to choose cloud services, but cost is almost never one of them.
Guy asked what is a cloud workload, I responded. Nitpicking every tiny detail doesn't help.
> There many reasons to choose cloud services, but cost is almost never one of them.
It's cheaper to pay me to manage IAM roles for lambdas and ECS instances for 5% of my time than it is to pay someone full-time to manage some sort of VMware or other system. It's easier and cheaper to find someone with experience with AWS who can provide value to the team and product than it is to find someone who can manage and maintain a cobbled together set of scripts to update apps. There are click and go options for deploying major self hosted services like grafana, k8s with secure details that I can use without spending any time (and time == $$$) learning about the developers preferred deployment scheme.
> It's cheaper to pay me to manage IAM roles for lambdas and ECS instances for 5% of my time than it is to pay someone full-time to manage some sort of VMware or other system.
True, but it's a false equivalence, and one I often see used from people unaware of the ease of contracting this out on a fractional basis.
I used to make a living cleaning up after people who thought cloud was easier, who ended up often spending a fortune untangling years of accumulated cruft that just never happened for my non-cloud customers.
Because you are paying someone else for them.
This is considered rational because those operators are presumably more productive in a pool of people using similar skills to support many customers rather than just one. It is similar to hiring a cleaning service rather than employing individual cleaners in a department of cleaning because cleaning things is not a core competency of business.
It might be less irrational if some amount of compute is part of the core competency of the business. Since "software is eating the world," compute is a core competency of all businesses except for the ones that don't realize it yet.
I've not really seen this work out well. I think it might be true for simple set-ups, letting a tiny developer team also handle infra and support without going nuts doing it, if they set it up that way from the beginning, but more-complex setups always seem to have so damn many sharp edges and moving pieces that support ends up looking similar to what a far more DIY approach (short of building one's own datacenter outright) would, in terms of time lost to it.
... and so does downtime, for that matter.
I'm used to organizations moving out of the cloud when they realize that it's more expensive if you don't have very peaky load demands.
And it's difficult to make that as expensive as a cloud deployment.
Which… isn’t to say anything about which way we should expect that to swing things. But it seems quite unusual, as most companies have not been bought by a cloud provider. Yet…
Nope, LinkedIn executes completely independently.
The pressure to move to a cloud and to Azure specifically both comes entirely from MS. Linkedin was perfectly happy then and now running its own setup, this 4 years of trying to move is because of MS ownership.
If you treat us-west-1 as a single data center, you may find you are spending a lot on traffic between AZs.
A lift and shift might treat us-west-1 like a single data center. A more sophisticated strategy might treat it as three.
When we were acquired by MSFT we had the same project. We had to move from AWS to Azure. I made them all stop saying "lift and shift" because in reality it is "throw away all of your provisioning code and rewrite it using Azure primitives which don't work the same way as AWS ones".
It is more akin to writing an iOS app to work on Android.
Similar terminology is "forklift"... been hearing that one for well over a decade.
Migrations are oftentimes an opportunity to revisit scaling, configuration, build and deployment pipelines, platform primitives, etc. Every migration I've been involved in has a (probably necessary) tension between getting the job done efficiently, while not repeating all the mistakes of the past.
And it’s not just the service fees. I blanche to think of the opportunity costs we accrued by focusing for that long on infrastructure to the exclusion of new product and features. It’s truly breathtaking.
And then there’s the burnout, and the ruffled feathers.
Even if done skillfully with valid rationale, they don't show any value until you come out the other side successfully.
They were worried the old vendor might go under. My own track record with predicting company failures is pretty bad, so I suspect they'll still be around ten years from now.
I mean, if you're already with AWS using their services (besides EC2 for hosting) such as RDS or S3; moving to Azure SQL (or DB for MySQL or whatever) and Blob Storage is not just lift-and-shift anymore, since you are actually changing from a cloud provider to a different one.
AFAIK an actual migration to the cloud would involve rewriting some parts of the application to be cloud-native, such as using Service Bus for queues instead of a local Redis/RabbitMQ instance, using GCS instead of local disks, and using RDS instead of hosting your own single MySQL server.
I've always read it as being roughly analogous to "like for like," and dependent on the specific circumstances and status quo.
I've been on enough Cloud Archeology expeditions into the land of VMs where nobody knows what they do, it might as well be my job title now.
If refactoring is too hard for a Microsoft owned company what am I to think about my tech stack?
Reality has a surprising amount of detail, and any non-trivial, customer-facing system will have accumulated weird code paths to account for obscure but nonetheless expensive edge cases. A codebase built across >20 years, scaled to support millions of concurrent users is going to be absolutely filled to the brim with weird things.
When you add the need for live migrations with zero downtime, done every few years to account for next order of magnitude loads, you end up with a proper Frankenstein's monster. It's not called "rebuilding an airplane while flying" for a lark.
Every round includes a long, complex engineering effort of incremental live migration. With parallel read/write patterns between old and new systems, and all their annoying semantic differences. And then, to add insult to injury, while your core team was going through the months-long process of migrating one essential service, half a dozen upstream teams have independently realised they can depend on some weird side effects of the intermediate state and embedded its assumptions to a Critical Business Process[tm] responsible for a decent fraction of your company's monthly revenue. Breaking their implicit workflow will make your entire company go from black to red, so your core team is now saddled with supporting the known-broken assumptions.
Then you get to add wildly differing latency profiles to the mix. While you were running on your own hardware, the worst-case latency was rack-to-rack. Implicit assumptions on massive but essential workloads may depend, unknowingly, on call-to-call latencies that only rarely exceed 100 microseconds. In a cross-AZ cloud setting you may suddenly have a p90 floor of 0.2ms. A lot of software can break in unexpected ways when things are consistently just a little bit too slow.
Welcome to the wonderful world of distributed systems and cloud migrations. At some point the scars will heal. Allegedly.
https://arstechnica.com/civis/threads/i-thought-hotmail-was-...