I didn't know that neon supports using S3 as cold storage. That's seriously amazing. How does that work with indexes and queries? It's there a single index which will always grow and stored on disk or is the index split?
The problem that S3 doesn't love small files. So we organize pages in LSM trees on the pageservers and offload layers to S3.
I hadn't heard it before and am not finding references googling.
I'm curious what problems you have with small files on S3; or if someone wants to feel free to point me to a link to discussion of this apparently known fact!
Lots of stuff on the web. Quick googling found: https://www.upsolver.com/blog/small-file-problem-s3