We could debate forever on the semantics of which is more free, but I don't think having the freedom to restrict the freedoms of others is going to encourage freedom.
My freedom to swing my fist stops at the tip of your nose; my freedom to modify legalese surrounding GPL software stops when it would deprive others of their "four essential freedoms" (in the words of the FSF).
0. The freedom to run the program, for any purpose.
1. The freedom to study how the program works, and change it so it does your computing as you wish. Access to the source code is a precondition for this.
2. The freedom to redistribute copies so you can help your neighbor.
3. The freedom to distribute copies of your modified versions to others. By doing this you can give the whole community a chance to benefit from your changes.
If you don't believe that these are important and should be respected, then you shouldn't be licensing things under the GPL. If you believe these are more important than the restriction imposed, then releasing under a permissive license is not itself "restricting freedom", but all it is doing (compared to the GPL) is enabling "restricting freedom". When the only thing a change does is make a bad thing possible, it's still better not to make that change.
I find it somewhat interesting we have no viral license that protects freedom 0 and 1 in derivative works but allows adding additional restrictions that conflict with 2 and 3 (or don't, which would make it GPL compatible...).
Interesting that preventing users from doing something with a bit of software (e.g. selling non-free derivatives) is considered restricting freedom.
Kind of self evident when stated from the users point of view really.
I use gcc, and I work as a treesurgeon.