While you could schedule containers manually, or just run your application on VMs or hardware manually, something like kubernetes will let you define rules which it will dynamically evaluate against your infrastructure. You can instruct kubernetes to run your map reduce jobs on different nodes than your user-facing site... and you can give kubernetes an arbitrary number of nodes to work with, and it can scale your workloads for you automatically while also following your rules.
Using VMs with Kubernetes only makes sense when you need the strict isolation. If you are running own code, then containers are faster. Containers also perform better because they can share resources on host. In Kubernetes, containers can have minimum and maximum limits, which means they can dynamically use space not used by other containers. VMs need to be allocated memory when they start.