AWS EC2 Virtualization 2017: Including Nitro
brendangregg.com
brendangregg.com
And we are finally here, the true cloud, elastic computing with little to no performance overhead.
[1]: https://www.theregister.co.uk/2017/11/29/aws_reveals_nitro_a...
Zones however are superior to VM’s because they don’t require hard pre-allocation of resources like virtual machines do, so they are a superior choice in environments where Windows isn’t desired or required.
As for my Linux hate, if Linux were so great and well made, I wouldn’t hate it. I like well made things.
In public cloud that's not a problem, that's a requirement, so one VM wouldn't trash all its neighbors.
Where we both agree is that it’s tiresome, but for very different reasons.
Some may appreciate being made aware of another technology; unfortunately this specific example reinforces the importance of marketing and pursuing mindshare.
Sounds to me that your understanding is the real problem here. There are a lot of resources you can use to read up on the value proposition, and while I do think AWS is pretty expensive and perhaps priced a bit too high, it's a great platform that goes way beyond just the speed of the VM tech.
With Linux running on ~10^11 devices, everything from watches to supercomputers to the Falcon 9 rocket, the market disagrees with you to a pretty comical extent.
As for Latin, this is the same reasoning of "learning" Linux and not understanding UNIX and mis-configuring the OS or breaking Linux backwards compatibility. Or re-inventing the wheel all the time while basic functionality like booting or the fiberchannel stack still do not work correctly after 20+ years.
Thank you masses for all the sleepless nights I was forced to endure because Linux has the same problems Solaris solved decades ago. Because masses know what's good and what's bad.
Yes!
That AWS should go home and re-write EC2 to just be zones instead of VMs?
There is no reason for AWS to exist.
Kernel modules are only meant to be installed in the global zone, on the hardware. This is by design. If you need access to that particular device, it can be delegated to that zone with the add subcommand to zonecfg(1M).
Zones do not increase attack surface, I don't know what would make you think that. They reduce the attack surface because one can set up read-only zones, good luck changing anything in there. In a zone, attacker lands in a jail, assuming they actually manage to break in!
What strange times we live in when there is so much misconception. So much knowledge lost! No wonder AWS and Linux rule the day!
Zones are a revolutionary technology in terms of network security. What makes you think zones increase attack surface?
In fact, if you look into Joyent, you'll see they don't use regular zones for multi-tenancy, but rather a KVM-inspired hypervisor implementation of zones.
Apropos "KVM-inspired hypervisor": the zones still function in exactly the same way as they did when I first used them in 2006; in fact when running lx and KVM Joyent is exploiting an obscure capability called a brand (branded zones), but that capability has always been there, since day one back in 2006.
Finally, there has only been one known case of managing to break out of a zone and this was through a kernel driver exposed to it, exactly the kind of scenario you'd have on VMware. In the decade since, there have been no known cases, so not only is your argumentation cookamamie, it's in the realm of pure hypothesis.
Hypervisors have their own security issues. They're just programs. That's why there've been all the patching over the years for Xen and for KVM.
Last year, an independent researcher did find an issue related to Dtrace that Joyent patched. However, the security and engineering that backs Zones is way more than with other kernels.
https://www.joyent.com/blog/dtrace-conf-16-wrap-up
DTrace exploitation: http://slides.com/benmurphy/deck#/
http://benmmurphy.github.io/blog/2017/01/06/arbitrary-kernel...
What do I mean by that? Brendan Gregg did mention in his article about Solaris migration certain things like amount of developers out of box performance etc.
Here are a few issues that the Solaris crowd failed to address and if they don't address it soon imho they will fortify their own demise:
Hardware availability, if you have a blade center with 10G intel hardware everything will work fine, but want to run it on a ryzen? tough luck. This already disqualifies a bunch of curious developers.
KVM support? Sure, but it's not well documented the tooling if you don't use the enterprise management tools by Joyent is horrible. The standard tooling `vmadm` in SmartOS will leave logadm residues that will spam your notifications it's been an unfixed bug for multiple years.
UEFI? Nope. The tooling and documentation is the worst. Want to migrate from libvirt? You have to shut it down, dd the image to the zfs zone and hope that it works. If it didn't well you can't access vnc before the host is already booted into a failed state.
The actual killer:
Documentation. The SmartOS confluence wiki starts off by giving you a taste and the typical enterprise way they offer you training.
That works just fine so long as you're one of select few choices to pick from(Jira loves that model).
But when you fail to attract the crowd of developers, you're bound to fall behind. There's exactly 3 books on OpenSolaris, 0 on SmartOS and a bunch of technical old documents on Solaris from Oracle.
There's probably been at least 15 Docker books this year alone. Whole bunch to manage kubernetes. For creating network topologies in linux you pick up a bunch of books that aren't outdated.
I love the idea of SmartOS and I'll run my machines as long as I feel it makes sense and you might argue that the complexity is a lot lower than what linux provides, but that's besides the point.
What the point in crossbow, HVM, etc. when the only place of information is:
a) an old oracle tech paper
b) an old oracle/sun/joyent consultant
c) one of 5 blog posts that explain how to run it in your home NAS
d) a wiki with so basic information that it's guaranteed not to work on your system
At least the complexity of Linux is visible, and you can find a solution somewhere. Rather than having to find a consultant that guards that solution with his life.
So let's get this straight: when Linux zealots 15-20 years ago answered "you have the source! Fix it yourself!" that was okay, but when it comes to SmartOS, it's out of the question?
Sure, but it's not well documented the tooling
Read. The. Manual. Pages. It's a real UNIX and that means high quality documentation! No "Stack Overflow" vomit!
The actual killer: Documentation.
Repeat after me: I will read the manual pages. There was just a discussion on this subject recently here.
b) an old oracle/sun/joyent consultant
so old Solaris guys are no good?
At least the complexity of Linux is visible, and you can find a solution somewhere.
Linux has incredible kludges and brokenness under the hood, it's just that one needs to be an old UNIX guy to understand just how much of garbage "solutions" places like "Stack Overflow" offer.
Rather than having to find a consultant that guards that solution with his life.
I've always been happy to teach all that I know about UNIX and never had a complaint or heard an unkind word about my teaching efforts, on the contrary. Find yourself a mentor. "Stack Overflow" isn't one and could never substitute one.
Repeat after me: I read the manual pages and it's still crap.
Your best source of documentation is the oracle documents they don't map 1:1 to SmartOS. Wanna learn about network topologies in SmartOS. Here's the 425 pages Oracle document that provides a tiny intro to how you "could" "maybe" get started [1]
When I asked Joyent in 2013 for information needed to improve the wifi stack there was exactly zero interest.
SmartOS supports docker images in a really interesting manner. But again it's half baked. It may work fine in the Joyent UI but that's meaningless. This is what a normal person has to do to get it running [2]
Judging by the way you talk I really seriously doubt you have any meaningful architecture running on SmartOS that's self managed. If you want to set up a small private cloud that's SmartOS based you're better off trying Danube cloud, but who knows how long that's going to last [3]
Compare that to AWS, you have a proper fully documented VPC tool that's Cloudformation, you have a whole bunch of recipes, you have tooling around working around the nasties of CloudFormation. It's just not attractive.
You're the zealot here. If I may ask, have you ever contributed to illumos codebase in any way yet?
[1]: https://docs.oracle.com/cd/E23824_01/pdf/821-1458.pdf
[2]: https://blog.jasper.la/docker-on-smartos-the-harder-way/
[3]: https://docs.danubecloud.org/user-guide/esdc/index.html
I am not farm help and there is no need to insult me. It won't help you.
Repeat after me: I read the manual pages and it's still crap.
Then you need to go back to the basics and learn UNIX. Start with "The art of UNIX programming" (it's freely available on-line). If you don't grok UNIX (not Linux) manual pages, it's time for back to school. It'll help you being better on Linux too, because you'll grok the overarching concepts.
When I asked Joyent in 2013 for information needed to improve the wifi stack there was exactly zero interest.
That is completely understandable to me; Joyent doesn't want to finance the development of a driver for your laptop because their resources are constrained and their focus is on serving data across datacenters. In the UNIX mantra: "do one thing and do it well".
They will however help you to learn how to write device drivers but you obviously have no interest in that, otherwise you'd have done so like for example Murayama-san did.
Here's the 425 pages Oracle document that provides a tiny intro to how you "could" "maybe" get started
You only needed to read the manual pages for dladm(1M) and ifconfig(1M) and to have looked at the EXAMPLES sections. Crossbow, the underlying technology in both Solaris and illumos remains unchanged or at the very least backward compatible.
Judging by the way you talk I really seriously doubt you have any meaningful architecture running on SmartOS that's self managed.
If you equate "self managed" and "architecture" with clicking around on little VM buttons in a (web) GUI or editing Docker YAML files, then you are correct. I do my (configuration) management with operating system packages. When configuration overlay packages are installed, systems talk to each other automatically. If a human has to click on anything anywhere, it's a bug and gets logged and corrected as such. I on the other hand suspect this is a concept so alien to you that you probably think it's absolutely crazy. Fact is, it worked for 70,000+ systems for the past 12 years and continues to do so.
You're the zealot here. If I may ask, have you ever contributed to illumos codebase in any way yet?
Yes I am because it's excellent technology; I don't hide it and I don't pretend otherwise. And yes I have.
What does Linux have to do with this? This is a new hypervisor backed by hardware virtualization, it isn't remotely related to Linux. Sure, you can run Linux on EC2, but you can also run literally any other OS as well.
How does AWS Nitro help solve the hard resource preallocation problem?
What about CPU bursting capability of the mighty illumos kernel, how does AWS Nitro compare?
Can you beat zones on pricing (in terms of resources required) with AWS Nitro?
Local NVMe storage is also handled through our Nitro hardware, and the underlying physical flash devices are protected from firmware modifications.
https://www.youtube.com/watch?v=o9_4uGvbvnk provides an overview.
If your use case is "I don't want to understand Xen", then fine, just use AWS and pick the newest instances, they're usually the best.
AWS does have some words about PV vs HVM here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/virtualiz...
But really, they don't say much and that's fine. If you just want to run stuff, you probably don't need to understand the deep details of Xen or PV drivers or whatever; the cpu speed, memory, and "disk io"/"cpu credits" they document quite well will be what matters significantly more.
If you are at the scale of netflix, then it does make sense to figure out an extra several percent disk performance.
> I would totally care more about replacing stuff if I knew ... instead of "I have no clue what if anything is different
They almost always have blog posts accompanying each new instance launch which roughly outline the performance improvements and reasons that instance type is cool vs previous ones, even if they don't go into the depth of virtualization differences so often (which again, usually doesn't matter! for nitro, the C5 blog did talk about the visualization stuff in depth https://aws.amazon.com/blogs/aws/now-available-compute-inten...)
ie the ability to run VMs within an EC2 instance?
Both i3.metal and c5 use the same underlying Nitro technology.
https://www.youtube.com/watch?v=o9_4uGvbvnk might be of interest - Matt Wilson talks a lot about the underlying magic of the bare metal instances.