It can't possibly add any restrictions to the use of your project, because people can always just use it under the terms of GPLv2 if they want.
I guess it could remove restrictions, if the hypothetical GPLv4 was a total 180 and looked more like the MIT or Apache licenses, so I guess if that's a concern then you have a point.
If a third party forks a GPLv2+ project, adding features that are GPLv3+ licensed, the result is GPLv3+ licensed.
That means those wanting the GPLv2 licensed project won’t be able to use those new features.
As drran mentioned, I'm sure RMS would say the problem here is that Linux is licensed under v2 only. Linus of course would disagree.
That's a bit disingenuous. Linus has said no to the GPLv3, yes, but his reasons for doing so have merit.