We also use server side encryption. But honestly encrypting the files before putting on s3 has given us 0 grief and gives us a feeling even if we somehow are “hacked” the files are useless to the hackers.
We have a one and done rest api that will only send the key once after being staged as part of our pipeline. So that’s the delivery mechanism. Key is on an internal server that isn’t on the public internet except the one and done. Super simple.