What you noticed is what is well known in education--school ratings are merely a proxy for the socioeconomic status of its students/families [0].
What can be teased out from that regarding cause/effect/deterministic is still mostly unknown.
I don't have a canonical source for this, but this nature paper gets the point across. This link [1] also has a useful chart on p. 6 to visualize the point.
[0] https://www.nature.com/articles/s41539-018-0022-0
[1] https://cepa.stanford.edu/sites/default/files/reardon%20dist...