>I could see larger addresses increasing latency
???
???
On the internet, most links are faster than 1 Gbps and most paths are shorter than 100 hops, so that's a conservative estimate.
If you're sending lots of 10-byte payloads, then IPv6 requires (40+10)/(20+10)=166% as much network capacity, but are you really filling up an expensive link with VoIP traffic?