I think the latency becomes most problematic when it's inconsistent, and if you want to fix input errors, or move the cursor and then input something.

I've experimented with video games with various amounts of display lag or dropped frames; nothing blinded or anything (although, setting up a blind test sounds fun), and there's clearly an increase in difficulty the farther you get between input and response. Writing code is clearly not Mario Brothers, but small delays can add up.