You don't. You can use gpg-agent with --use-agent.
duplicity inc \
--sign-key $PGP_PRIV_ID \
--encrypt_key $PGP_PUB_ID \
--use-agent \
${source} ${dest}