I have GigaBit internet from AT&T. Speeds have been terrible and pings were like 130ms. I disabled IPv6 on my laptop and things improved. I disabled IPv6 on my router and it improved even more.
You’re making a good case for something other than IPv6. And since we’re running out of IPv4 addresses, we will need to switch to something else, eventually.
IPv6 enabled in your tcp/ip stack doesn't create latency. You weren't even using IPv6 - ping is IPv4 by default.