Author of the post here. If you have additional key insights about the Parse infrastructure, please post them here and I will directly add them to the article.
I think one of the major instabilities not mentioned the request fanout. On startup the parse ios client code could fan out to upwards of 10 requests, generally all individual pieces of installation data. Your global rate limit was 30 by default. Even barely used apps would get constantly throttled.
It was even worse running ParseServer, since on Mongodb those 10 writes per user end up as a sitting in a collection write queue blocking reads until everything is processed. It was easier to kill the entire feature set it was based on.
I know there were a ton of other issues but I've forced myself to forget them.
How often (if ever) did you experience data loss due to the MongoDB write concern = 1?
After moving the APNs (v2) server to Go I was able to map the wonky networking to Resque much better. My show & tell that week was literally "this MacBook is now running higher throughout than our cluster of >100 push servers". For APNs, the rewrite was a boon of over 500x per server.
As a great fringe benefit, fewer Resque servers meant less Redis polling. CPU dropped from 97% to <10 IIRC, which helped save us the next Black Friday where Redis falling over from Push load often caused problems.
I don't recall how often data loss was a problem. One of the points that hadn't gone into a lot of detail was when certain ops tricks happened. For example, I recall that reading from secondaries was reserved for larger push customers only. These queries could be wildly inefficient, have results in the 10s of millions, and due to mongo bugs would lock up an entire database if they had a geo component in the query. In later versions of Mongo where the bugs were fixed and after another generation of the auto indexer we were able to send all traffic back to primaries.