I have a free game that you can download from my website. How do you propose to install it on your phone? Under iOS ecosystem rules, you can't (beyond some limits like 99 friends and family copy and a USB connection or something).
However, if you rely on other GPL'ed code then I can see that it would be impossible to do that, as then it break the licenses of that other software. In which case it seems that the strong copy-left is what's screwing things up.
Also, I must repeat: "whatever the end user have access to is not free". My point is, if we want to free the end user, then the copy hold by the end user should be free. But a regular user isn't free to modify and redistribute the copy he hold on his iPhone, even if the original licence was a BSD one. Heck, even if he could, his modifications still have to pass through the App Store review process, which means there's no guarantee other users could benefits from them.