Bitfrost – LLM gateway 90x faster than Litellm at p99

Bifrost is the fastest LLM gateway on the market. Built in Go with careful garbage collection, it adds just about 11 microseconds of overhead at 5,000 requests per second (with 4,100 RPS throughput) on a t3.xlarge instance.

The benchmarks are here: https://github.com/maximhq/bifrost/blob/main/docs/benchmarks...

Some features: • Built-in governance and routing rules • Supports over 1,000 models from different providers • MCP gateway included (HTTP, SSE, and console transport) • Out-of-the-box observability and OTel-compatible metrics