Silence in public places is a coping strategy for crowded environments - it's the least intrusive thing you can do to the other people in your space. It's like the difference between jumping around and shuffling quietly when you're in the middle of a crowd - one may be more fun for the person jumping around, but it's selfish, because it makes everyone else's experience less fun, as their toes get mashed on.
That's really the crux of it. Talking on your cell phone when you're mashed up against other people is a purely selfish act. That's why it will always be considered rude.