Spark is useful in that it’s pretty cheap and flexible. I can grind through 10 TB of data for like $50 and a few hours. If my experiments work, I can use it as a component in other projects. Plus, I can use (slightly non-standard) SQL right in pyspark, if I want to. And, it often performs faster because of the query optimizer.

If your db is scalable enough (like Snowflake) and you have the money, you can use SQL directly in the same way. But, I've seen data analysts writing bad SQL against Snowflake costing thousands of dollars per query.

It’s really tough to future proof anything. At least with pyspark you will have the code. And, there will always be experts out there. Worse case, if it’s important enough, someone will wrap it in an API and use it until something better comes along (see: 50 year old mainframes still in use).