Maybe your parents left you a few millions in a trust or even a billions worth family business... or your parents left you outside, in the streets, without school, making you beg others for food to survive.
You had literally zero choice of what family you're getting born into.
Every person bears responsibility for every decision they partake in, just as we also need to acknowledge that some have significantly more agency in such engagements than others.
This is straightforward Rawls-liberalism.
You say that like the same baby might conceivably have been born into more than one potential family. But the opposite is true: examine any baby and you'll find that there's only one possible set of parents they can have, barring identical twins.