Paypal To Drop VMware From 80,000 Servers and Replace It With OpenStack
forbes.com
forbes.com
My personal experience with OpenStack (admittedly late Diablo release timeframe) was that it was borderline unusable outside of the developer setup of a standalone node on Ubuntu 11.4 backed with KVM (edit: and that was only Nova/Compute, not even including Swift/Storage).
Anyway, very interested to see if this is KVM, Xen, or something else backed. Also very interested to see when they reverse their decision (probably after VMWare comes down on pricing by 10-20 percent).
OpenStack supports multiple hypervisors. They are implemented as drivers - and are in this part of the OpenStack codebase:
https://github.com/openstack/nova/tree/master/nova/virt/
See my slides from a talk I did yesterday.
https://github.com/sc68cal/openstack-theory-and-practice-pre...
> OpenStack is a cloud operating system that controls large pools of compute, storage, and networking resources throughout a datacenter, all managed through a dashboard that gives administrators control while empowering their users to provision resources through a web interface.
Should it (??) really say :
> Openstack is a ton of scripts and tools and things which pull lots of various bits of software together and allow you to have redeployable virtual machines all over the place
Also, <= Diablo and to a large extent Essex releases were known to be (politely speaking) less-than-usable OOTB. Folsom was really the first release that could properly be called usable for a production IaaS, and Grizzly is even better.
Source: knowing these things is my job.
We've come a long way in 2 years :) Redhat (EPEL/Fedora) and Ubuntu now have built-in packages that are much more solid.
That said, the way to think about OpenStack is like you think about the linux kernel. Most people want a distro, not raw source to setup their environments. Over the last couple years we've seen real advances in distros.
That said I know more people would would be content to manage ESX with ovftool or some homebrew solution w/ the soap api than I do who would be willing to use the linux kernel without the usual userspace tools.
I know more people would would be content to manage ESX with ovftool or some homebrew solution w/ the soap api
without usual userspace tools
OpenStack is a bunch of code that runs in an OS, there's nothing nearly so magical or different about an OpenStack node as you imply. Do whatever you feel is best for your host nodes.
I install Debian packages on the host and manage it with many of the same tools used to manage it's images. It looks very much like the same usual "just a boring usual Linux node" node I log in to. http://wiki.debian.org/OpenStack
Since he was comparing the relationship between a hypervisor and Openstack to that between the Linux kernel and the userspace tools, and I was just commenting on that comparison. That is, which I know people that use hypervisors without Openstack (or other virtualization management applications like Cloudstack or vCenter), I don't know anyone that uses the kernel on its own. Sorry, I thought that was clear but I guess it was not.
Has more details. Looks like it's KVM.
Agree with you about the headline. However, VMWare is not contributing Nicira for free. There just happens to be a Quantum plugin for Nicira (as there is for pretty much everything else).
I.E. - it likely didn't just work pretty well for them out of the box.
There is a difference between using a base technology which requires you have FTEs on staff to maintain it and grabbing an off-the-shelf app to help run your systems.
1. According to Business Insider (BI), PayPal is using Fuel from Mirantis to manage its OpenStack deployment.
2. BI also announced that Fuel was released on March 25 under the Apache 2.0 License. However, the fuel website points to a form that appears to give free access to fuel under a creative commons attribution, noncommercial, share alike license, with additional licensing options available from Mirantis.
3. To top it all off, the CEO of Mirantis contacted BI and said the information about PayPal migrating over was incorrect and based off of second-hand knowledge (as seen on the update in the bottom).
4. BI also spoke to paypal who stated they are diversifying their VM infrastructure to "enable choice and agility", not replacing VMware.
So, based on all this, it sounds like what really happened is PayPal decided to implement OpenStack to allow for a more diverse internal set of tools to develop infrastructure. A miscommunication occurred somewhere, which resulted in the BI and Forbes story.
http://www.businessinsider.com/a-dangerous-sign-for-vmware-p...
If you're interested in discovering OpenStack, you should check out devstack[1]. It's a bash script that (git) pulls different projects of openstack into a machine and install them.
The script itself is very readable, thoroughly commented, on purpose. You're encouraged to read it while the whole thing installs (which can take several minutes).
Feel free to launch it inside of a VM (preferably running Ubuntu or Fedora), in order to avoid polluting your current system.
[1]: http://devstack.org/
You really shouldn't, unless you're interested in doing development work on the bleeding edge code in Git. Devstack is a development tool, not a deployment tool.
If you're interested in discovering OpenStack, you would be much better off installing released, tested packages provided by your Linux distribution. It's true that there is still some setup work required, but there are many, many people working on tools to help with this:
I'd tend to avoid mixing the streams & go for a single node management system rather than attempt two at once- were it me. But I also am not invested in Vagrant, which people are free to enjoy, so if they can happily run both VM managers at once, so be it. Cool, I guess.
I would like some clarity on what exactly this hybrid achieves though- what is it's goal?
The only twist here is the software you are testing happens to be virtualization management software. Make sense?
>So, what, now you can virtualize while your virtualize?
... I am not expecting to deploy or do much on virtual hardware, but, the servers are pretty beefy and I just want to muck around with the management and learn how it works so I can deploy it at a later date.
It sets up openstack from source, tracking master.
It is what developers of openstack use to setup dev environments, run automated testing, ... There are also puppet and chef recipes that use packages (which is recommended for production deploys)
From the first line of the script: "stack.sh is an opinionated OpenStack developer installation". It doesn't provide nearly the flexibility of Puppet (or Chef).
Devstack does one thing, and one thing (almost) only: get a simple installation of OpenStack up and running as fast as possible (predominantly for developers to test their modifications against).
Puppet can handle much more complex scenarios, and is really meant to deploy production.
I'm stunned by the extent to which the forbes.com rehashing of articles crest the front page compared to the original articles.
It looks like they are adding OpenStack to the suite of tools they use at PayPal and not replacing anything(at least, not yet). Sounds like Renski is just a little over-excited when commenting about a platform he believes in.
That just boggles the mind....
edit: just noticed they sneak "and eBay" into the 80,000 total server count. Makes a little more sense now :)
I'm not really a fan of virtualization myself. We canned it at our organisation a couple of years ago. We had 30 ESX/vSphere managed hosts across 3 data centers which hosted app servers, web front end servers, virtual load balancers and some other minor infrastructure. The final straw was the upgrade costs. Not only that we had problems with volume size limits on our SAN which would require more expense and hackery to work around. Also, we couldn't host our database servers on top of it due to performance problems so we had to have dedicated machines there anyway.
The whole thing at the end of the day just added complexity, expense and didn't improve reliability, security or load distribution as our architecture was sound on that front already.
We've gone back to the original virtualization system: processes and sensible distribution of them across multiple machines. Performance, cost and sanity have improved.
I can understand that virtualization is useful when your resource requirement is less than one machine but above that, I doubt there are any real benefits. It's snake oil.
As opposed to a miracle elixir that cures all ills?
It's a tool.
Like most tools, it's best used when you thoroughly understand the challenge and how/why to apply that tool to it.
The problems you've noted here seem more indicative of poor planning and understanding of the solution than any intrinsic deficiency of virtualization.
We understand it. The numbers sold are not the numbers gained.
The entire infrastructure deployment was planned around virtualization, DC failover and resilience using VMware's guidelines and solutions (VMware were even paid to consult on this). It never delivered and doubled our administrative and licensing overhead.
Honestly I have to say, it sounds a bit like you guys got taken for a ride. I'd suggest sitting back, relaxing and taking a look at what other VM options there are out there, some of which don't have license fees attached to them.
When I first deployed virtualization there were really only two 'enterprise' ready solutions out there which were essentially VMWare or XenServer. Knowing about VMWare and their 'history' I chose XenServer. I haven't regretted the choice but they wouldn't be a fit for everyone.
These days there are multiple numbers of solutions out there to choose from, which Openstack may or may not be one for you... and that aren't going to whisper into your ear about how great their product is and how much money it will save you.
By now, I'd actually be more surprised not to find a VMAX or Shark in an "Enterprise" datacenter.
Calm down please.
>The entire infrastructure deployment was planned around virtualization, DC failover and resilience using VMware's guidelines and solutions (VMware were even paid to consult on this). It never delivered and doubled our administrative and licensing overhead.
That might say something about VMWare's specific solution and you being taken for a ride, but it's nothing intrinsic to virtualization as a tool.
The outcome is pretty grim.
Even if there was 0% performance penalty from virtualization, you'd still see suboptimal allocation of hardware resources just from trying to take an abstracted view of the hardware. Different applications have different performance profiles. You either end up with overbuilt hardware to support the virtualization environment and the different performance profiles of the different applications, or with multiple VM's for the same application on the same hardware which is totally unnecessary overhead. Virtualization is just not meant for large scale.
Jump to the results section. The more relevant bullets here are 'single guest' (either virtio or using direct mapping).
Highlights (or lowlights, depending on your perspective): kernbench - 9.5% overhead SPECjbb - 7.6% overhead
I don't agree with your point about suboptimal allocation of hardware resources. Virtualization does not require you to divide a machine in a different way than processes do (you could easily have one VM consume nearly all CPU cycles, one consuming nearly all I/O capacity, etc.) IMO, the key difference is that virtualization lets you easier establish hard, enforceable limits and concrete policies around resource usage (not to mention the ability to account for usage across all kinds of different applications and users). And, it lets you do that for arbitrary applications on arbitrary operating systems. So users don't have to write to one particular framework/language/runtime/OS whatever. That's all pretty important for large scale.
[1] http://static.usenix.org/event/osdi10/tech/slides/ben-yehuda...
Is there a system that would allow for mapping part of an IO device (such as a block range or a LUN) to a guest when multiple guests are running with the same level of overhead?
The distinction being made here isn't for a single guest or multiple guests, it's for a single guest OS or nested guests (i.e. a VM running another VM). To expose the hardware virtualization extensions to the guest VMM, then they must be emulated by the privileged domain (host). There are software tricks that allow this emulation to happen pretty efficiently (and map an arbitrary level of guests onto the single level provided by the actual hardware). It's not a common use-case, but for a few very specific things it's very useful.
There are a few different ways to map I/O devices directly into domains. Some definitely allow for part of an I/O device. For example, many new network devices support SR-IOV -- which effectively allows you to poke it and create new virtual devices (which may be constrainted in some way) which can be mapped directly into guests.
Parties that care muchly about fine performance margins apparently need to be using Xen or KVM or Illuminos then.
For instance it's easy to make a benchmark showing huge throughput to any given storage solution (and many NAS providers sell on this basis), but your I/O might be terrible because to get that throughput you're maxing the CPU (etc.). Likewise, you can change your benchmark and show high I/O, but the throughput is 'terrible'.
If you're seeing that, something is configured wrong. VMWare was 95+% of native disk and gigabit ethernet 5 years ago.
Either the hypervisor scheduler is shit or the abstraction is costly. I reckon its down to the reduction in CPU cache availability and the IO mux.
This was HP kit end to end, VMware certified, debian stable on and off ESX 4.
Second: Why did you implement a large-scale VMWare install without either having a testing period before sinking contract dollars and license costs into it, or at least having contract terms to opt-out if their claims didn't match reality?
We did have a testing period which was unfortunately run by a Muppet who decided the loss was acceptable. Muppet now no longer works for organisation.
We were running heavy loads and there was nothing like a 20-30% hit. I'm not saying you didn't see one but this isn't magic or a black box: we had a few spots where we needed to tune (e.g. configuring our virtual networking to distribute traffic across all of the host NICs) but it performed very similarly to the bare metal benchmarks in equivalent configs.
What precisely was slower, anyway - network, local disk, SAN? Each of those have significant potential confounds for benchmarking.
Tell that to Netflix - or any of AWS' customers, really.
The only use case I can see for it really is rapid scaling but that seems only viable for content delivery as your data back end architecture is way harder to scale than click the magic cloud button. Back in the old days we used to do content via CDN (Akamai etc) which actually works out cheaper per GiB.
Then we approach things like S3 which is terribly unreliable (some of our clients use it for storage as it's cheaper than a SAN but they suffer for that). Drop outs, unreliable network requests, latency, rubbish throughput and buggy as fuck SDK.
Its like consuming value line products from a supermarket. Sure there's quantity but its lacking in quality.
But the key advantage over traditional hosted solution is that you can run reproducible test and dev stacks for 8/12 hours a day without having to pay for hardware that sits there doing nothing while you sleep. Sure cloud computing has it's faults but the ability to run you a HA test stack in 3/4 datacenters in minutes, and pay for just the time you are testing it, enables people to move forward and develop more and more impressive technology.
OTOH I'm not sure why you would run cloud on your own hardware as you're already paying for the HW, I suppose it simplifies the management significantly for PayPal or they wouldn't be doing it
This whole thing has to play nice with a windows DFS cluster as well.
That's enough abstraction.
Besides the quotes suck ("OpenStack is not really free"; well, if you don't know how things work you'll have to pay who knows), so I'm not sure if I can trust that information.
Perhaps it's mean to poke fun at reporters, but this is pretty brazen.
The text straight from the results page tells an interested party that it's an "enterprise messaging system", which should be good enough for any blog on forbes.com. I really have no idea where they got the gobbledygook that they used.
Something like "middleware that connects applications" would perhaps be clearer.
Oddly quite a few build systems are now turning to MQ's for build systems - beats make and jenkins it seems
So yeah, it's owned by VMware.
References? I'm curious as to who's doing this and why. I'm trying to think of a reason and not coming up with much.
Currently we mostly use this to send internal release notifications around, e.g. "foobar-1.0.0 released, find it here <path to RPM>". Kind of neat with a n * 1000 developer organization spread over the entire world.
Currently the plugin is not open source but hopefully it will be later on.
THink replacing Jenkins not make
I'm always skeptical of abstractions but it'd be great to hear about any real world experience.
It runs.
Its a VM. I can add and delete and image. Remotely.
Its admin system is all REST(ish) and I did have a fully working auto-build system till they changed it and I have to rebuild (known API breaker, well telegraphed, unlikely to be more, I was just lazy)
I'm a big fan.
It's pretty trivial to get a basic system up and running and your basic functions for creating, controlling and destroying instances work flawlessly. In general, core services are quite reliably and reasonably documented.
I've managed to create some sticky problems fooling with Quantum [1] when it was still in incubation, but nothing a little digging couldn't resolve.
1: https://wiki.openstack.org/wiki/Quantum#What_is_Quantum.3F
If you look at their other offerings some of it is reasonable (VMWare Fusion) and others are set at whatever the market will bear (VMWare Workstation, vsphere Enterprise).
For the customers who do not need much more than basics (e.g. just virtualisation, storage server etc.), it is just so hard to justify the price now.
Just the other day I had a client who wanted basic fault tolerance and by changing versions, the quote goes from 6k to 47k! It is crazy pricing... :(
Because very often I tend to get confused between the two..because of the vague similarity in their names..
OpenShift could theoretically run in an OpenStack cloud, as just about anything could, but the two products are in no way related to my knowledge.
OpenShift Origin is a PaaS, as you describe. You generally run it on top of an IaaS such as OpenStack, although the hosted OpenShift actually runs on AWS (which is also an Iaas).
At the same time, I could be that OpenStack's API is superior to VMWare's. With a fleet of servers, you want to automate as much as possible - meaning that at the end of the day, API trumps GUI.
It reflects the bizarre decision VMware made when they wanted only sell to the top end of the market: the biggest customers are also the most likely to have the resources and incentive to build tools around that API. The small to mid scale shops depend on the GUI tools far more.
they would be talking about vmware ESX\vsphere not workstation.
I stopped supporting it in 2008 but at the time it was already apparent that they were trying to milk large companies rather than stay competitive – and that was before the price hikes.
At one point VMWare was actually pricing things by memory used, not processor core counts.
Licenses for the top-end ESXi run $1000 - $3500 list per socket [1]. If you have a large deployment you're going to be inclined to go for the top-end license in order to support distributed switching and granular control over storage/network traffic.
It's pretty interesting, I hadn't heard of OpenStack before, and the company I work for is migrating TO VmWare.