Downsides are that kids develop at different times, have different educational needs, have home life issues that can temporarily derail progress, etc and if those happen around the time kids are getting tracked, they may not reach their full potential.
A good education system would offer students a way to rise up whenever they're ready to rise up, let them learn at their pace, focus on mastery, build upon knowledge gained rather than schedule followed, etc. There's a lot of edtech out there that incorporate these concepts but school models struggle to integrate it into the (literally) old school way they operate. It's quite difficult to reorient school around these new concepts at scale, it has to be done school-by-school, leader-by-leader, school board by school board.
Agree its complex, as it may be the 'best of the worst' option for certain contexts. Anything involving balancing equity/access/etc is like that.