For one thing it can create paradoxes, like your future you deciding they didn’t like where they ended up and stopping yourself from setting out on a journey in the first place.
This relates to the different interpretation of quantum mechanics. The one that has paradoxes is the Copenhagen interpretation and the one with the branching is the many-worlds interpretation.
There is also the pilot wave interpretation, but I am unsure how to map the example you proposed to that one.
That said, decoherence doesn't branch time in the sort of absolute manner you might be thinking of. "How many timelines exist" is a question similar to "how long is a coastline"; it depends on how closely you're looking. At the smallest scales, 'timelines' interfere and mingle in a way that distinct universes really shouldn't. It's only once decoherence has gone well out of control that they can be said to be truly separate, and even then the interference never goes to zero, only approximately zero.
In a growing block universe, I think time travel would be impossible, but I also think that the growing block universe theory is cope.