i'd actually say that's an argument against having an 'adversity score' - a single value simply isn't enough to capture the nuances of your situation, and having a standardized score might pressure colleges to rely more heavily on those values.
specifically for your example, there doesn't seem to be any factor in the score calculation that takes into account 'distance from school,' while 'quality of high school' does play a role. so it's quite possible that since your high school did better overall (i.e. serviced 'wealthier' kids, skewing the score), your score would look better than things actually were.