I'm worried about that too, but you can deal with it by making the system that automates your backups also test your memory of the password (while still creating a non-interactive backup even if you ignore the prompt). Alternatively, reuse a password that you regularly use offline, like a disk encryption or SSH passphrase.