The biggest issue is that it's a bit harder to maintain ABI compatibility. You have to carve out reserved vtable space, avoid exposing STL in your interface, etc, but it's doable.
As for Swift, it has heavy userspace dependencies that make it non-viable for kernel work. Rust does much better there.