Show HN: OSS tool that let you create OpenAI DX with any LLMs

Hi HN, Spike and Donovan here. We have been building BricksLLM (https://github.com/bricks-cloud/bricksllm), an API gateway that allows you to add the "OpenAI DX" to any arbitrary LLMs. For example, you can create a self-serve dashboard to allow end-users to create API keys for accessing your LLMs, impose different rate/spend limits for each end-user, track their token usage, and detect PII. Everything is done via a high performant API gateway written in Go.

BricksLLM has native support for OpenAI, Anthropic, Azure OpenAI and vLLM. You can also integrate with other OSS LLMs using our custom provider feature.

Currently, people are using BricksLLM for the following use cases:

* Safely sharing access to different both commercial and self-hosted LLMs (e.g. teachers sharing LLM access with students, managers sharing LLM access with data scientists etc.)

* Building an AI dev platform with observability and monitoring

* Building token usage based billing

* Blocking or redacting requests that contains PII (https://github.com/bricks-cloud/BricksLLM/blob/main/cookbook/pii_detection.md)

* Improving reliability via retries, failovers and API key rotations

We believe that one underrated aspect of an LLM platform is the developer experience. The performance of a foundational model is important. However, LLMs will become less differentiated as the competition heats up among different providers. Therefore, a good developer experience is just as crucial in creating a competitive advantage. This is why I believe OpenAI has invested a lot into improving the DX of its platform. Now with BricksLLM, you can do the same to your own LLMs.

The latency of the gateway is about 15ms on my M1 Chip MacBook pro. We have load tested our applications to be able to handle more than 1000 requests/s (https://github.com/bricks-cloud/BricksLLM/blob/main/blog/how-we-built-a-highly-scalable-LLM-gateway-with-go.md).

Would love to hear your feedbacks and opinions!

Open Source Repo: https://github.com/bricks-cloud/bricksllm

Landing Page: https://trybricks.ai