Spark is great, but at that point why not just use Scala? It offers Python-like conciseness/productivity, and by using the same language Spark is written in you avoid a big class of possible interop issues.
PySpark should only be used for prototyping. It add an enormous extra overhead on operations due to serializing data back and forth between the Java and Python processes.