Special relativity does not, in itself, forbid you going backwards in time. But it does tell you what would be involved in going backwards in time that way. They involve things like imaginary mass and imaginary energy -- things that pop out of factor sqrt(1-v^2/c^2) when v>c.
As far as we can tell, imaginary mass and imaginary energy are nonphysical. Relativity doesn't say it has to be that way, but nobody has ever seen them and they don't seem to exist.
There are hints of it in the Standard Model, which define mass and energy in terms of even powers of real numbers. Again, that doesn't prove that other things can't exist, only that we've never seen them, and would expect the universe to look very different if they did exist.
It might have something to do with the configuration of the universe at very early times, the low-entropy state that is the reason that the universe is interesting (and, in particular, capable of having us in it). But that is an open question.
Meanwhile, special relativity is sufficient to rule out time travel unless you want to make some assumptions that mass and energy are very different from anything we've ever seen them do.