Broadcom SoC is probably protected by a bunch of patents and/or trade secrets, which makes it impractical to open source. For example the whole boot process is a black box, which is why you have to use Raspbian and not regular Debian.
Device Tree and other things help with this, although you do still require certain BLOB's to access the GPU and other features.
The 4’s hardware support is still being merged upstream last I heard though, so the easy way for that is still to use the Foundation’s kernel (which is what Ubuntu does, probably to also make overlays work as expected).
https://github.com/librerpi/rpi-open-firmware https://github.com/itszor/vc4-toolchain/issues/7