PGP is a two-party system. The sender has a public/private keypair, and the recipient has a public/private keypair.
The sender encrypts a message with the sender's priv key and the recipient's pub key.
The recipient decrypts the message with the the sender's pub key and the recipient's priv key.
> Almost all of the transactional emails I have received (receipts, confirmation numbers, etc) are probably unencrypted, right?
Totally up to your email provider and a sender's email provider. Your provider may choose to send/accept email over TLS, which is also encrypted. Gmail, for example, does this.