I think it's definitely a starting handicap, no doubt about it. But I'd be very wary in telling those people - if any are reading this - that they're just not going to be successful.
Without getting too touchy feely - I definitely had un-supportive, unstable, conditionally-loving parents. Was it a setback? Yes. Would I be financially ahead of where I am now had they been better? Probably. But you know what, I'm doing alright. Maybe not by HN silicon valley standards well ahead of the curve in my little first world country. Never lost ambition even if I stumbled a lot along the way.
So yeah if you're younger, and reading this, and a bad childhood is still messing with your mind - please don't think all hope is lost. It will be harder but not impossible.