I recently listened to an NPR piece on this and it seems like you're right here. Even with affirmative action you're really only affecting those that are going to elite colleges, and a large amount of people who benefited from it actually didn't come from low-income households. Essentially affirmative action benefited a tiny (like <1%) portion of all people going into universities in the way it was intended. What would be much more useful is exactly what you say, taking much more of the student's overall socioeconomic status into account rather than race.
We would probably find that this does a better job of helping those who actually need help. And that would likely mean that minorities are still prioritized.
Finally, what we should really be doing if we care about helping those at the bottom of the socioeconomic ladder is make all community college free. It's already very cheap as-is , we just need to take the leap to make it totally free. This would also help the student loan crisis since those who are going for bachelor degrees would only need a couple more years of education vs. 4