There's a surprising amount of userspace code which has to check the kernel version number because some behaviour has no "feature check" that can be used. On rare occasions that gets tricky when a distro has backported some kernel change to an old kernel version.
Because it's not supposed to be done, most breaking changes are unexpected so it wouldn't be possible to implement semver anyway, or are bug fixes which happen far too often for semver.
If it helps, I remember the transition from Linux 0.99 to 1.0, the switch to ELF and Glibc, the first non-x86 architecture support, the introduction of kernel modules, threads being a new feature, and the dropping of 386 support - all but the last happened while I was using Linux as my desktop OS at my day job :-)
I would nearly bet that there are still vendors shipping 2.6.x kernels in their embedded systems today. Probably rare...