Experienced Spark / Data Engineering teams would not assume S3 is readily useable as a filesystem.
This [1] seems like a good guide on how to configure spark for working with Cloud object stores, while recognizing the limitations and pitfalls.
[1]: https://spark.apache.org/docs/latest/cloud-integration.html
---
Amazon EMR offers a managed way to run hadoop or spark clusters and it implements an "EMR FS" [2] system to interface with S3 as storage.
[2]: https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-fs.h...
AWS Glue is another option which is "serverless" ETL. Source and Destination can be S3 data lakes read through a data catalog (hive or glue data catalog). During processing AWs Glue can optionally use S3 [3,4,5] for shuffle partition.
[3]: https://aws.amazon.com/blogs/big-data/introducing-amazon-s3-...
[4]: https://docs.aws.amazon.com/glue/latest/dg/monitor-spark-shu...
[5]: https://aws.amazon.com/blogs/big-data/introducing-the-cloud-...