A lot of the recent improvement in graphics on Linux has come from manufacturers deciding to become cooperative rather than there being more people reverse-engineering graphics drivers. (Broadcom is contributing support for the Pi 4; AMD added its drivers to upstream; nVidia releases binary drivers to support enterprise customers.)
What isn't always available is a way to dynamically enumerate a device's, uh, device tree. On PCs, this is usually accomplished through ACPI tables, where the bootloader (BIOS or UEFI) populates a table of data and callbacks to AML bytecode functions which the OS kernel can read and eval. [0]
So on x86, the OS finds the PCIe memory mapped addressed in the MCFG table.
On most ARM devices, this data has to be passed to the kernel at runtime, in what Linux calls a "device tree reference", compiled to a "DTB" file.
So, on a UEFI-based ARM system, it should be possible to just insert, say, an AMD graphics card onto the PCIe bus and get a working GPU.
[0] https://www.kernel.org/doc/html/latest/arm64/acpi_object_usa...
IP blocks inside the SOC itself usually just expose their registers on an AXI bus with no enumeration support at all, this is most likely the case with the M1 gpu.
They're functional for very basic desktop stuff but 3D performance is abysmal
Kepler is anywhere from 2x to 10x faster on the proprietary driver here: https://www.phoronix.com/scan.php?page=article&item=nvidia-n...