Your freedom ends where it conflicts with the freedoms of others. At that point, clearly something needs curtailing and the question is what/how. The GPL is one resolution.
My freedom to swing my fist stops at the tip of your nose; my freedom to modify legalese surrounding GPL software stops when it would deprive others of their "four essential freedoms" (in the words of the FSF).
0. The freedom to run the program, for any purpose.
1. The freedom to study how the program works, and change it so it does your computing as you wish. Access to the source code is a precondition for this.
2. The freedom to redistribute copies so you can help your neighbor.
3. The freedom to distribute copies of your modified versions to others. By doing this you can give the whole community a chance to benefit from your changes.
If you don't believe that these are important and should be respected, then you shouldn't be licensing things under the GPL. If you believe these are more important than the restriction imposed, then releasing under a permissive license is not itself "restricting freedom", but all it is doing (compared to the GPL) is enabling "restricting freedom". When the only thing a change does is make a bad thing possible, it's still better not to make that change.
I find it somewhat interesting we have no viral license that protects freedom 0 and 1 in derivative works but allows adding additional restrictions that conflict with 2 and 3 (or don't, which would make it GPL compatible...).