Drift velocity is unrelated to signal propagation. Otherwise it'd take entire days for packets to cross the country to get to you, which is clearly not the case. If we want to take an analogy, signal propagation is like a sound wave in the electromagnetic field inside the conductor; in real air, the speed of sound is clearly faster than the wind.