I maintain many Debian VPS for my product and I had to ditch all IPv6-enabled servers because they were so slow. Like, they couldn't pass the apt-get step.
Which researching before posting this answer, I just noticed that it is due to DNS timing out for IPv6 requests [1]. IPv6 is hard to set up.