So this is basically a *safe* optimization, since the index will always be valid and there's no need for the compiler to do a check and unwrap, panic.

This is how unsafe {} should be used. Sometimes some things are true but the compiler can't know that. And here the unsafe {} means that it dereferences a raw pointer (the index that we know is valid). If the unsafe {} safety condition is valid, unsafe {} is, well, safe.