I've been using unikraft (https://unikraft.org/) unikernels for a while and the startup times are quite impressive (easily sub-second for our Rust application).
Still difficult to see how the unikernel could be slower, but I doubt the difference would be huge? Don't have anything to back that up though.