I don't know if that's the case anymore. The kernel ships with a lot of drivers now, and Root FS for embedded devices aren't really space constrained anymore. The needs of an IoT device have also grown to include things like containers and telemetry. In my experience, forking off of an existing distribution like Debian is way easier than navigating Buildroot or Yocto's documentation and cruft, and results in faster builds.
I got so fed up with those tools I eventually built a custom tool (Etcha) to build my own meta distribution (EtchaOS) of immutable variants for Debian, Fedora, etc. It owns the entire provisioning pipeline and builds multi-arch images in less than 10 minutes: