An opinion from the other end: In 1st grade (7-8 years old), I tested out of my grade and was able to go into the gifted program. My parents gave me the choice instead of choosing for me, and I chose not to because I had just made some of my first school friends and didn't want to leave them.
It's been well over two decades, and for almost all of that I've regretted the choice.
In addition to not getting the academic boost the gifted program might've given me, socialization with general-age peers didn't really help; I remained the odd one out for pretty much all of my school years, it was only come college that that really changed.