For what it's worth: references in Rust (both `&` and `&mut`) are just pointers with stronger static (compile time only) analysis. They have zero runtime impact vs raw pointers.
Even heap allocations like `Box` are still just pointers, that have destructors that run when they fall out of scope, but access is just a pointer dereference, the same as you could do with unsafe.