I had a psych professor with a similar story that she liked to share. Her prevailing hypothesis on why we don't remember our infancy was that there's typically little to no reason to recall it, so the memories either fade or morph into general feelings and emotions that often help form the bases of our early personalities.
I may be getting some of that wrong, too. I was not particularly engaged with that class since I was still a rather immature student, back then. I'm not saying I subscribe to her idea, but it's certainly interesting to think about.