Therein lies the problem. What makes you think the past can be changed? If the past can’t be changed, there’s no problem with time travel (at least on these grounds).
Moreover, if you adopt the block universe view, as general relativity seems to demand, there’s no such thing as change at all. There’s no becoming, only being. This was the conclusion reached by Parmenides in ancient times. Julian Barbour wrote a book on the subject called The End of Time, for those who are interested.