However, I have found that a lot of the benefits of virtualization are often lost because of the flexibility of having a full blown Linux OS. People stick too many services on the host and/or don't know how to properly allocate resources.
I think there is something to a simplified model where you no longer have access to any host environment and can only "fill slots". In practical terms, it makes it easier to implement features, like firmware-style upgrade and rollback of the host, for the masses.
Yes, you can do this with a full blown Linux OS on the host but it requires an operational discipline that most environments don't seem to possess.
I'm not sure I buy the whole "run multiple containers on top of an EC2 instance" thing though. I understand why they're using LXC vs traditional virtualization but it seems to me like it's a case of solving all problems by adding another layer of abstraction.
There are too many different kinds of virtualization so we'll just add another layer of virtualization.