According to many, including the FSF, GNU, Stallman and myself, increasing access to the source code for everyone is increasing freedom. In fact, it's freedom number 1. "The freedom to study how the program works, and change it to make it do what you wish."
https://en.wikipedia.org/wiki/Four_Freedoms_%28software%29#D...
> The slavery quip doesn't really make sense, because in that analogy the source code is the slave.
No, it's about freedom for people in both cases. The lack of source code would be analogous to the chains of slavery. Not equivalent, mind you, but analogous. A software user is less free when he or she cannot inspect or modify the software that runs on their machine.