I feel similarly to you, and distinctly remember feeling empathetic even when I was young. Now, I see life as being deterministic, which makes it easy to think "would I not be behaving in an identical manner?"
Do you think that people who struggle with empathy have that issue partially due to a non-deterministic view of life?