It sounds like many of the improvements are not available on PySpark yet, which is disappointing. (Notes say feature parity for MLIB, so I'll look into that) However, the notes sound promising.
That said, as part of Project Tungsten, we have some ideas about a batch columnar format that can be shared by Python, R, Scala and Java, and that should be able to eliminate most of the inefficiency in serialization across process boundaries.
Btw, I was critical about the issue above, but I do love Spark, using it on a daily basis. :)
Thanks for the reminder!