This is different than storing a key in the database that can turn encrypted data into unencrypted data. This is what I meant by not storing secrets. Instead of storing keys, pgsodium use encourages storing key ids (of a given length in a context). These keys can be used to reveal plaintext, so they shouldn't be stored but only derived.
So the trick to avoiding storing keys is to store the key id and derive the key, being careful not to leak it in the logs. This can be done by passing it directly from pgsodium_derive() straight into a function that takes a key. I'm considering adding a "mirror" api that doesn't take bytea keys but instead only key ids and always derives. I'm still kicking that idea around.
But your point is good, I should be less cavalier with my definitions of secret and key. I'm working on improvement for the "encrypted column" section that will explain more of this, and show how pgsodium can work in hand with Row Level Security. Stay tuned.