A cell phone tbh, is about the same latency as a landline in most end to end call circumstances. Latency only really is noticeable when better than 600ms. (And only a real problem over 1000)
One other big factor is consistency: if the variability is due to compressor overhead which is constant, the effect will be noticeable but less distracting than if it's varying due to something like wireless conditions.
Musicians have that internal metronome to compare things with.
For many people, end-to-end audio latency in a 1:1 conversation becomes noticeable/annoying at 200ms. And in a multi-participant conversation, talking over each other becomes noticeably more common even at 100ms compared to 50ms.
I would've thought it would be closer to the 200ms range, but I don't have any data to support that.