4ms is 1.4 meters (4'6") of sound in air. That's less than half the length of a concert grand, so at most it would double the latency as experienced by the pianist's ears.