That's why I said "except if your SLA requires it." If you've agreed--with other teams, with outside clients, or just as a project goal--that your service should work on 99.9% of calls, and you find in practice that you need to retry S3 calls to meet that target, then adding retries is reasonable.
If the problem is just that the 99 calls could overload downstream services on retries, the ideal solution is to add rate-limiting, though admittedly that is an imprecise science.