Here in the U.S., very few public high schools have single-track education, at least not for "core" subjects such as English, math, and science. There are typically remedial, standard, honors, and AP classes for these subjects.

And I would think that if you're OK with multi-track education, it would seem like having a separate school for gifted students is just an extension of that.

That said, in my own experience, although there were down sides to having some single-track subjects (e.g. physical education), one benefit of attending a school with children of all ability levels was that you were exposed to children from all walks of life.

I would imagine that a school for gifted students would have skewed socioeconomic demographics compared to a school for students of all ability levels, and in many places, that might also mean less racial diversity.

At my public high school, we had a program where students at standard-level or above would assist special-education students. That type of opportunity would probably be missing from a school for only gifted students. You would be less likely to come into contact with (and befriend) people with intellectual disabilities, or even people with average intelligence.

Edit: One other point. Academic giftedness is often thought of as one monolithic thing, but I know several people whose giftedness varies wildly by subject. If you attend a school for gifted students, would there be an expectation that you would be gifted in all subject areas? Would it be more difficult for you to get help in the subjects where you struggle?