If you look at this through the lens of Maslow's hierarchy, a kid growing up in a poor environment is going to lack the fundamentals needed for cognitive development. He may be undernourished, for example. His guardians are probably too busy trying to make ends meet to read to him. They may even actively ignore him. He's handicapped right out of the starting gate and will likely never have a chance to reach his genetic potential.
In contrast, a child growing up in a rich family will have access to plenty of material goods. He's not going to worry about having to move 4 times in a year because his parents keep getting evicted. He's going to get sent to the best preschool money can buy. However, money can't buy him the drive to succeed on his own - to self-actualize, if you will. He's grown up in an environment where everything has been provided and nothing intrinsic to that environment is going to push him. He'll get closer to his genetic potential than the poor kid, but unless someone actively encourages him to develop his own potential, he's not going to get all the way, and there's your gap.