This issue with needing more flexibility on SoC's makes me wonder if its part of why the Linux Foundation has put a lot of work and resources into Zephyr RTOS [1]. Its gaining a lot of popularity and incorporates some of nice elements of linux like device tree syntax (DTS) files [2]. I could readily imagine using Linux & Zephyr on a SoC, and using shared DTS to describe the SoC architecture.
Actually, come to think of it extending DTS syntax to describe the pointer swizzling the OP video mentions. Then both OO'es could derive correct physical address handling.
1: https://www.linuxfoundation.org/blog/the-power-of-zephyr-rto... 2: https://docs.zephyrproject.org/latest/guides/dts/index.html