If you can remember all the way back to the day of your birth, those memories are surely not real memories. Newborn babies are essentially not fully conscious - not in a way that would create any long-term memory of lived events anyway.

It sounds to me that this memory-recall game likely started when you were a toddler or preschooler rather than an infant, and that while this may give you additional reach back into your earliest memories, it was likely more a game of creativity than genuine memory recall. Into adulthood the real memories and the invented ones can end up feeling roughly the same.

Unfortunately there's really no way to test the accuracy of your memory aside from comparing your memories to a previously unobserved but concrete record. For most of us, our childhood was documented by our parents, and the stories shared with us over the course of our lives, so there's no way to prove that even the accurate recollections weren't simply our memories of the stories rather than the events themselves.

Humans have poor memory anyway. We misremember things that happened recently, so it is probably unwise to trust your memory of things that distant as being accurate, especially when they reach back to an age where most established science suggests that you wouldn't have even possessed self-awareness.