I don't notice any latency whatsoever. If you're not exaggerating and are genuinely experiencing "brutal" latency, then something is definitely wrong with your setup.
Not necessarily. Some people are just less sensitive to latency, so even if a 20ms added delay might not be noticeable to you, it doesn't mean that other people won't feel it.