I think the difference is that misbehavior isn't as expected of libraries as it is of tests. But yes if your libraries are prone to misbehavior, it's common to compartmentalize them in separate processes.
that's roughly what the wasm component model is aiming for!
Honestly, I often feel that’s the world we should be heading to. Why should my zlib have any access to my programs memory, other than an input and output buffer?