If you want to be really paranoid you can make it an online process: application server makes an HTTPS request to a key server on different infrastructure to get the decryption key, then decrypts the secret. Use an IP whitelist.

That way read access to your infra provider is not sufficient, it requires an active component.