Lucene has supported near real-time search since 2.9 or so. By calling IndexWriter.getReader(), your index reader has access to both committed and non-committed changes.

In my experience, this near real-time search picks up changes within the last 500 milliseconds or so. I think it's basically a workaround for the latency related to disk I/O on commits.

What is Twitter doing to make that faster? I'd appreciate any links you could post which explain their changes.