Some people might be fine with that, it's just the combination of an aptitude test with another type of metric they object to. Chicken soup and chocolate milk are both good, but maybe not mixed together.
The problem with an "adversity score", as stated in another comment, is that it doesn't actually measure the adversity that an individual faces. It's not chocolate milk, it's diarrhea milk. It doesn't accurately measure the unique stresses or difficulties that a student faces in life. Plenty of students do fine being raised by a single father or mother, and some have a terrible time being raised by two terrible parents. Some students have mental health issues or are victims of bullying. Some students have disabilities that affect their academic performance.