Are you claiming that there weren't enough interested kids to put together a full class to teach algebra in middle school? Specifically in large cities like San Francisco or Seattle where this kind of thing started? As far as I know, that justification is not being given, and instead the argument is about how some kids are learning more advanced material than others, and the "fair" thing is to hold back the more advanced students.

The thing about tracking is that it doesn't require more resources. If you have 6 classes in a year, you can put e.g. the top 17% together, and they can move at a faster pace than the bottom 17%. Or you can let kids take classes with mixed ages. Gifted kids don't need more resources. Literally you could have them go sit in the library and do what they want and they'd probably learn more than sitting in a class where they're forced to pay attention to something they've already mastered. When I was in calculus I was bored enough that I would also read my uncle's discrete math book that he let me borrow, but most teachers don't like when you ignore them, and when you're in a class with the lower end of the aptitude curve, there's negative peer pressure around being a nerd.