The thing is, that these SoCs have no PCIe or other enumerable bus. You (=your kernel) must know what hardware it is running on and which drivers to load without being able to ask the hardware, what is really present. One wrong POKE and the entire system can hang.
And that is on top of the problem, how to boot in the first place. There is not such thing as UEFI on these SoCs. Every single one is a special snowflake with its own special way to boot.
Hence, kernels built for specific systems, even if it is just Device Tree.