We have to leave aside two unanswered statements in your comment: how many people in prison actually deserve to be there, and how many of their rights prisoners surrender by being convicted of a crime. I'd argue that one's human rights don't ever get revoked, but that's just me.
What is the purpose of prison? I'd argue that it's a dual purpose: punishment and rehabilitation.
Being away from everyone you've ever loved or known is a terrible punishment. Once you walk out of a prison years after you went in the world will have changed out from under you. Your babies are grown and won't know you. Your parents might have died. Your friends' lives have moved on without you. The rhythms of daily life are now foreign to you. The particulars of any specific penitentiary system might make this easier (visitation, parole, etc.), but this is pretty horrible.
Why does someone commit a crime? Unless they are mentally unbalanced (which, in the US, is served by separate processes... e.g. pleading insanity), they have behaviors that are unacceptable outside of a set of social norms. Giving someone the opportunity to learn what society expects of them should be part of being in prison. Isn't education a great way to figure that out? Why should we be against someone trying to turn their life around after making some terrible choice? (Again, leaving aside questions of over-eager judicial systems e.g. the US' war on drugs)