Given this is failing due to HN hug of death, might I suggest that you do a periodic batch, save the results and serve static?