100ms is too much. That's almost 1/16th at 120bpm. Anyone multitracking would try to use at most a 256 sample buffer, which gives a 6ms (extra) latency.
Sound travels about 2m in 6ms. Iirc, it's difficult for humans to perceive time differences smaller than 20ms, so humans perceive sounds from within a radius of 6m simultaneously.
When playing an instrument, I can live with a 20ms roundtrip.
I don't care about playback performance. I usually have my playback buffers at 512 samples, but I could easily deal with 1024 samples.