This arguement has a long history.
It is a widely used pattern in rust.
It is true that panics are memory safe, and there is nothing unsafe about having your own ref ids.
However, I believe thats its both fair and widely acknowledged that in general this approach is prone to bugs that cause panics for exactly this reason, and thats bad.
Just use Arc or Rc.
Or, an existing crate that implements a wrapper around it.
Its enormously unlikely that most applications need the performance of avoiding them, and very likely that if you are rolling your own, youll get caught up by edge cases.
This is a prime example of a rust antipattern.
You shouldnt be implementing it in your application code.