Nope, because you just get a 404, even on public buckets, because you have no access to the kms key.
To be clear, I'm no aws advocate.
Yes, it sucks if someone randomly decides to download files from you all day. You should probably set your budget to alert and attempt to blacklist them when it happens. That’s rare, though, and aside from a few cases of actual malice, the convenience is worth the cost.
“Always”