This is exactly my confusion, why people say there is one. I guess things would be different if I was trying to do it over low bandwidth high latency sat connection or something. At least with connections as slow as 50mbps I have not noticed any issue, I don’t have anything slower to readily test to see what people are talking about. I’ve no doubt it’s theoretically slower due to e.g. compression but that doesn’t seem to be a big practical concern at least for most connections and use cases.
I’ve found it highly concerning because the other solutions will not work for me, but adherents seem to insist what’s working for me doesn’t.