When my own voice arrives at my head 20ms late, my pitch performance suffers, as well as comfort. It feels very weird.

When playing an instrument, I can live with a 20ms roundtrip.

I don't care about playback performance. I usually have my playback buffers at 512 samples, but I could easily deal with 1024 samples.