Arm, who has refused to bundle or assure system IP compliance to the level of their CPUs (aka interconnects, interrupt controllers, iommu's, pcie bridges, etc). Meaning that even very basic things like how one gets an interrupt isn't reliable, and somewhat worse are the buggy implementations of arm system IP that are allowed to reach the market.
Google, for failing to provide platform standard interfaces, similar to what the Arm server vendors enforce via SBSA/SBBR. Meaning that every single phone is doing low level platform mgmt in proprietary ways. That includes its own power mgmt, led blinking, etc, a good part of the time actually in the linux kernel, or via proprietary hooks in the linux kernel. Further despite making more money on the android ecosystem than RH+suse+canonical put together their changes are generally dwarfed by other much smaller players (although they have hired a bunch of maintainers over the past couple years).
Linux itself, for failing to provide a stable/backwards compatible driver API. Meaning a small simple driver either needs to spend months upstreaming (if that is even possible, see GPUs) or man years of maintenance keeping up with the kernel churn over the lifetime of the device. Further the arm/kernel community encourages a "everything in the kernel" (everything from firmware functionality, to the actual machine descriptions in the form of all the DTs) attitude which completely fails to grasp the huge number of bugs and device varieties on these arm devices. Its rumored that QC by itself had a million plus lines of out of tree code a few years ago. Its likely that this attitude could more than double the number of lines of driver code in the linux kernel if just the past few years of Arm Soc's were fully supported.
So, the combination of the three are the perfect storm of massive overhead for supporting the couple hundred phone models in existence. Imagine for a moment if every PC model made in the past 10 years required a few (tens) of thousand lines of kernel changes, how unworkable that would be.
There aren't enough engineers to solve this as a brute force problem, the solution is to look at the PC market and consider that maybe it would be better if there were actually some standardization in the phone space.