Allowing you to do whatever you want is not the same as allowing everyone to do whatever they want. You can't understand free software from an egocentric view of the world. A non-free derivative will ultimately restrict someone's freedom, hence, considering it restricting freedom is perfectly applicable.

We could debate forever on the semantics of which is more free, but I don't think having the freedom to restrict the freedoms of others is going to encourage freedom.