Honestly, I assumed he was white because it was entirely unremarkable for a black person your grandfather's age to be illiterate or close to it. (I know your grandpa isn't black).
An illiterate black person had almost no chance to have a successful life.
> If you have confidence, charm, and enterprising spirit, the world is open despite any particular individual limitations.
THAT is white privilege. You must understand that the world is simply NOT open to black people this way. Not at all.