You could do a lot worse than modern GC. For instance, in Rust, lack of GC may cause you to reallocate memory or use reference counting, both of which are likely slower than GC. For instance, building a tree in Rust will do a lot of reference counting, which is a lot of heap accesses.