Exactly. The whole timeline already exists and is self-consistent. Whatever you do in the past already happened, and it did so in a way that was consistent with the future you travelled back from.
Time travel to the past is not inconsistent with known physics (see the various GR spacetimes with CTCs). There are reasons to think it’s probably impossible in our universe, however.