Yocto can be incredibly simple, this is my favorite example: https://github.com/bootlin/simplest-yocto-setup/

Only the kernel and bootloader usually need to be specialized for most modern arm boards: the userland can be generic. Most of the problems people have with yocto are due to layers from hardware vendors which contain a lot of unnecessary cruft.