My server (https://serveur.clauzel.eu/) have been connected using spdy on IPv6 for more than a year, and I didn't notice any problems.

Google crawls it daily : « 2001:4860:4801:2:aa00:6006:1300:b075 - - [20/May/2013:11:54:32 +0200] "GET /robots.txt HTTP/1.1" 200 4109 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) »