Use defer to make code readable. Go code should be readable, so you can figure out if it is correct, and only then should it be fast.
When benchmarking/profiling leads you to a hot spot in the code, rewrite it to remove defers.
This is typical: https://github.com/cornelk/go-benchmark/blob/master/defer_te...
Yeah, it's in the class of "things you should know about when optimizing tight loops", but not "things you should always be worrying about"; remember 5ns is on the order of a branch mispredict or an L2 cache miss [1]. If you haven't already squeezed out all the main memory accesses you can from your algorithm at 100ns a pop, or if you're in code dealing with networks (routinely micro seconds, even within datacenters; milli seconds if you have to leave), or files, or anything else like that, micro-optimizing defers is not going to have any visible results to your speed but can very badly damage your code's correctness and ease of writing and modifying.
The other thing to know about defers is that they are not declarations, they are instructions; every time the program counter moves past one, another defer call is added to the stack of calls to make while exiting the function. So you do have to be careful about deferring in a loop, not so much because it's "dangerous" but just because it's easy to be unaware that the defers are not declarations. If you intend to do it, it's fine, it can just bite you if you do it accidentally.