Sometimes this hinders you and your customers from getting things done. Mainly because vendor kernels are, generally, really bad. Second to none other than their userspace drivers, and I want my users to be generally running a well-supported, solid software instead of whatever ${VENDOR} ships.
> USB, ext4, SD/MMC, SPI, I2C, PWM, GPIO, backlight, framebuffer video...all pretty mature now.
That's pretty basic stuff, but SPI/I2C drivers for example have issues /all the time/. It's just nice to get the fixes from mainline and integrate easier.
Other than that, custom SoCs are much more complex than the bread and butter you mentioned: NPUs, VPUs, signal processing chips etc are all a gigantic pain in vendor kernels, simply because the code is garbage.