An open fast model gateway for resilient GenAI applications
github.com
github.com
As a part of this initial scope, we had to set up a bunch of common things to make it roll. As for the core functionality, we have brought up:
- The routing functionality with four types of routing strategies (including a tricky one like the least latency routing) - The first-class adaptive resiliency & fallbacking across all routing strategies - Unified Chat API that supports popular model providers like OpenAI, Azure OpenAI (on-prem models), Cohere, OctoML, Anthropic - The ability to have model-specific prompts - Installation via Docker & Homebrew