It is also caching requests on-disk. See for instance https://github.com/ianb/infinite-ai-array/blob/main/iaia/gpt...

The problem* is that if a prompt changes slightly then it won't be a cache hit.

* okay, one problem.