saw your tweet on X, nice work and congrats on launching!
i'm curious about the caching mechanisms you've implemented to reduce repeated evaluations - are you using a traditional cache store like redis or something more bespoke?
i'm curious about the caching mechanisms you've implemented to reduce repeated evaluations - are you using a traditional cache store like redis or something more bespoke?
So there's 2 levels of cache - the LLM request itself might be cached (simple and semantic) and the guardrail response can be cached as well.
We use a mix of a distributed kv store and a vector DB to actually store the data