I had a similar issue with my Django project on Heroku. My site performance went down once I enabled memcache, almost entirely due to latency (going up to 100ms at times). This is one of the reasons why I ended up moving off of Heroku and onto an EC2 instance: now I have full control of my stack, I can run what I want locally on the server, and I'm getting much better performance (nginx+gunicorn+memcached+django+johnnycache+rds)