Most poor people, even those with a single job must work _at least_ 8 hours a day, 7 days a week to come close, but nowhere near enough, to supporting themselves—let along improving their station in life. And this is simply their job. Add onto this time that must be spent taking care of children and parents.
This class of individual is not going to be able to find time, even on a weekend, to vote because our society is explicitly engineered to put them at a disadvantage in time, capital, and thus political influence.