First, like I mentioned, Spark uses separate contexts for all executors (by design). This means that running nested RDD mappings is a nightmare. Just Google "nested for each loop Spark" or something along those lines[1]. This is an extremely common paradigm, especially when dealing with more complex transforms (in my specific case, we needed to hit an API, and then hit another API based on results of the previous API).

Second, Spark is not re-entrant. This means that to really get usefulness out of Spark's massive parallelization, you need to use clever and non-obvious tricks to do something that "seems" simple[2]. In some extreme cases, the RDD needs to be fully serialized (danger zone if it's a big one).

Finally, and related to the two above, Spark is just simply strange. You really need to have a "parallel" mindset when writing code on a Spark cluster. I constantly had to look stuff up and documentation was spotty at best. Once you throw in the rest of the ensemble (Pandas, Numpy, etc.) you end up with a very domain-specific codebase. In reality, most companies will have datasets that could be ETL'd on my iPhone and a Spark cluster is just overkill.

[1] https://stackoverflow.com/questions/34339300/nesting-paralle...

[2] https://stackoverflow.com/questions/32619570/can-i-use-spark...