KVM is a userspace API, and kvm-intel/kvm-amd are the drivers for the hardware.
You will be using hardware features. That's also why it is in the kernel: nothing but the kernel should have full unlimited access to the CPU to set this up.
So you could say it must be in kernel to keep the kernel secure. And the performance benefit is "just" exposed hardware features.
The kernel does not provide additional things. As far as I understand: you set up a dedicated memory space and handle traps that halt the execution e.g. when the VM talks to the PCI bus. (It's been a while since I looked this up)
But you need the pieces, especially virtual PCI devices. That's where qemu or VirtualBox enter the scene (or minimalist systems like firecracker). They provide a repository of virtual hardware and all the auxiliary methods to boot a virtual machines. You also need to emulate something like a BIOS or UEFI.
You can think of it as your CPU removing the need to emulate the very same CPU (and a memory controller). You still need to emulate the rest though! But running on the same CPU removes most performance penalties. You run at native speed.
Newer generations can even nest this. Having virtual machines in virtual machines. That's mostly useful for cloud environments so that the cloud provider can run kvm based VMs and you are still able to run VMs inside that VM.