Leaving phones behind on old incompatible OS versions 2 times in 3 years and switching app frameworks 3 times in 4 years does not a good app developer experience make.
Piled on top of that, Google became actively hostile to 3rd party developers building support for YouTube (and Gmail and Gmaps, but those had workarounds / alternatives).
The advantage being, we can manage packages using a regular Linux distro
Drivers doesn't just mean the kernel. It's the user space binary blobs and services that need to talk to the kernel to enable the hardware.
Other than that there's waydroid, alien dalvik etc.. that run another Android instance in a container.
The thing is a lot of Android applications use safety net/other methods to make sure they only run on. "Approved"/stock hardware.
Google thought about this, don't worry. They learned their lesson after CyanogenMod tried to compete by offering an alternative. Non-Google Android are now dead except in China.
> The advantage being, we can manage packages using a regular Linux distro
It's much easier, more plausible, and more logical to sandbox a regular Linux package-managed distro inside of Android, which numerous solutions exist for.