Off the top of my head: Jail is local and generally short term. Prison is regional and longer sentences. Penitentiary...I think federal (national rather than state level) and higher security.
I keep the difference straight be remembering the phrases "county jail" and "federal prison", which makes it clear that going to prison is much more serious, and a much worse place to be.
The idea of prisons (or even proper police forces) operated by individual counties in the US still strikes me as odd, but then, those counties are likely more comparable to German states.