> xtensa/tensilica is a good architecture, cheap and lean, unfortunately its biggest problem seems to be its harvard architecture
Pedantry: everyone uses a Harvard architecture, that just means having separate buses internally for instructions and data -- essentially if you have a separate icache/dcache, you're Harvard. Xtensa is weird because it's L1 caches are incoherent.
As far as the architecture, it's sort of too clever for its own good. The register windowing scheme seems amazing, just a few comparators at runtime and some special exception handlers is all it takes to get 64 hardware registers to work. But... no one ever taught compilers how to deal with the complexity (the generated code needs to decide how many of its local registers to preserve across a call, which is a global optimization that depends on the behavior of the code it's invoking). So the end result is that for the same code, Xtensa code actually uses significantly more stack space than conventional architectures like ARM or RISC-V. (Their own toolchain is better at this than gcc, but still bad).
And FWIW: Zephyr doesn't do dynamic loading yet. We have crazy linker behavior, but it's all static.