I wonder what your actual cache hit rate is? Sounds like it could use some tuning since 100-150ms sounds consistent with most requests hitting the origin?