numlocked

We are heavy Hex users at OpenRouter. I was skeptical that it would replace "core" reporting tools like Looker, but the efficacy of the Hex agent has made it our go-to for just about all of our analytics, and allowed us to get incredibly far with a single (very good) person on the data team.

We can make completely reproducible reporting when we need it (for e.g. our weekly business reviews), while the agent can answer all of the ad-hoc stuff that would normally be done by a small army of analysts scurrying around.

No stress! My first reaction to this was OMG THIS IS AN INCREDIBLE RESOURCE!! We are 100% grateful for this sort of feedback! Here is a direct quote of what I said at 8:17am this morning when someone sent me the article and I scanned it:

  this is amazing!!
  [8:19 AM]The first obvious win is routing around providers that arent handling image inputs correctly. That should be straightforward
  [8:19 AM]The effort param stuff...I thought we had addressed that, but will dig in. This is incredible feedback
  [8:20 AM]We should hire this guy.

Our goal is to get better, fast!

Pulled some data - a surprisingly large number of requests to models that generate images, where the user seems to want an image, do NOT return an image. In our chatroom it's ~11%! However, in almost all of those cases, the model is instead deciding to return text, and you are being billed (correctly) for that text. We are not billing you for an image that wasn't returned.

Concretely, we pulled data on the last few days of image gen requests in our chatroom (50,893 requests). 5,846 got a text response (which is frustrating, I'm sure) and were billed for text appropriately. The model did not generate an image.

There were 16 requests where a customer was billed, but neither an image or text was returned. Those should not have been charged, and we'll see if we can either fix that issue or ensure that customers aren't charged.

We are doing continuous benchmarking of each endpoint, for each provider, and it is very expensive :)

We started actually expiring the credits a ~month ago. If you make any kind of API request, it resets the clock. We try to make it a very generous policy, but we can't keep a monotonically increasing liability on the books. We end up owing (a lot) of taxes on it, but can't actually recognize revenue. We would much rather you spend the credits! Hence the reminder emails, and generous "clock reset" policy.

We have made MASSIVE improvements here, and network-wide caching rates have been improving relentlessly. We do publish the cache rates for each endpoint; see the "performance" area of our model pages. E.g. https://openrouter.ai/deepseek/deepseek-v4-flash-0731#perfor...

Open to feedback on how to make this better.

Will look into this.

Yeah we are very under-staffed. We are hiring aggressively. I readily admit we have inadequate support staffing right now, but we are going as fast as we can to ramp up.

Hmm, let me check. That certainly seems wrong. Can you send me an email w/ your email so I can look into it? im cc at openrouter.ai. Or DM me on X? x.com/cclark

We run benchmarks against all of our endpoints, in production. That first chart that the author shows is in fact our live benchmarking data. If providers underperform, we kick them out of the routing pool. That is why we run those benchmarks. Performance

And errrr...yeah...that auto-exacto performance chart is both 100% useless, and totally unclear. We will get that fixed. But under the covers it is doing a lot of valuable work! https://openrouter.ai/docs/guides/routing/auto-exacto

Co-founder and COO of OpenRouter here.

Thanks everyone for the feedback here. Some of this we are aware of, some of it we aren't. Some we can fix, some of it is inherent to inference (and we in fact improve the situation dramatically).

Philosophically, at OpenRouter we are trying to do two different things, that are sometimes at odds with one another:

1. Let you use a lot of capacity across a lot of providers, in a way that "just works" and you don't need to worry about it.

2. Have a huge variety of inference available so you can pick radically different price/performance tradeoffs, data policy decisions, geographic destinations, inventive hardware, etc.

These are inherently odd bedfellows, and we are still very much improving how we can make both of them true at the same time.

Some quick thoughts on the article itself:

1. Benchmarks: YES! Providers benchmark differently. We run benchmarks on the live endpoints continuously, monitor the median performance, and kick providers out of the default routing pool if they vary by more than a standard deviation. We work hard (and continue to invest) to make sure that providers serving sub-par inference can't game the system, and that our routing actively avoids them. So the chart is accurate (it's our chart) and it actively influences our routing decisions!

2. That is bad and we will fix it. Sorry.

3. When we on-board providers we run essentially the same test as the author did to verify that the param is working as expected. If it isn't, we don't launch the provider. However this is not one we are running constantly in production. We are working on making this more robust in general and I do believe is fundamentally solvable in a way where it will "just work".

4. We 100% agree that users should not filter by quantization. It's a bit of a legacy concept in general; there is a huge amount of code between "model weights" and "inference API" and in almost all cases quality degrades in that part of the stack, NOT in the model weights themselves.

5. Hmm...we will dig in here. We monitor tool calls in real time and route around providers that are regularly mis-parsing tool calls. So you should get a very low rate of these in general. Another area we have invested a lot in: https://openrouter.ai/docs/guides/routing/auto-exacto

6. We will dig in here as well. I'm surprised this is happening frequently enough to be noticeable. We eat the cost when the finish reason is an error, but not when it is "stop". Perhaps we can expand our "insurance" program: https://openrouter.ai/docs/guides/features/zero-completion-i...

7. Will investigate.

8. We attempt to heal these, but obviously missed some. Will fix.

9. We do not rate limit by IP. Would love some more information here, as that is very surprising.

10. Ugh. That sucks. I'm sorry. We are introducing QoS tiers for production apps, which will address a lot of this.

We do now indeed have IP restrictions for API keys

Can you share more about leaking keys?

Just sent you a DM on twitter!

Refund policies are clearly documented in our terms. We actually DO offer refunds within 24hrs of credit purchase, which is significantly more flexible than most companies that operate in a similar way. And we try to use good judgement when there are extenuating circumstances.

By default (and in most cases) investors and operators are aligned. When we diligence our investors, we call companies they worked with where things didn’t go well, and speak to those founders. Understanding how investors operate when it’s not all up-and-to-the-right is important when picking partners!

Yep!

Everyone wants a conspiracy, but what I originally posted is in fact the boring truth. Having a bunch of cash in the bank makes for a durable business!

We have two mechanisms whereby we retain data. Both are opt-in and off by default.

One mechanism where you get a discount and we can use the data (in theory this does mean sell it; but our intent is to use it to make efficient dynamic routing solutions. But absolutely we could one day sell it) and another where we retain it for you so you can see it in your logs. We have no rights to this data in any way. This is similar to how any tracing/logging solution works.

Both and opt-in. If you don’t opt in, we don’t retain anything and are a pass through with regards to your prompt data.

All of this is carefully documented and I encourage you to explore and chat with the docs.

We have never sold any prompt data to anyone, in any form, and have no plans to do so. Full stop.

Great investors are helpful, not harmful :) You want accountability from smart, experienced partners!

Our general theory of the case is that, in the not so distant future, inference will be the second largest opex line item for most companies (behind headcount) and that sourcing, measuring, and governing those tokens is a massive horizontal opportunity.

We will inevitably expand into adjacencies because we like building things and experimenting and we have a lot of people with great taste who are likely to ship cool things that customers want to use!

Edit: also - THANK YOU!

Interesting. Will look into it! We are releasing pass through API params soon which might hit the bid, but is a bit different than what you are describing.

What differentiation are looking for? We have good documentation of every provider and what their data retention stance is, and you can figure allow/blocklists for all providers.

Check out the Guardrails section under settings and tell me what’s missing!

Hi HN! OpenRouter co-founder and COO here. Lots of questions about why we raised!

First off: We remain founder-led and founder-controlled, and intend on being here for a long time, creating awesome products for builders all over the world. We are basically a bunch of tinkerers who like building things, and try to make stuff that we would like, when building with AI.

Since this is about the raise though, happy to share perspective on it.

We believe that strong companies should have a strong balance sheets. We touch large volumes of spend, and have large spend commits across the ecosystem; having the cash to withstand what may come is a responsible buy-down of risk, and makes the company extremely durable.

It also tells our larger customers and provider partners that we will be able to continue to serve them (and pay our bills) for a long time to come. We don't need venture dollars to continue scaling (indeed the business is healthy) but you know when you don't want to raise $100m? When you really need it!

This is also good validation to employees (current and future) that the value we are creating together is real. We also take seriously our obligation to make a return for anyone who invests; we aren't valuationmaxxing and have the privilege of getting to pick who we work with. I don't think that gets a lot of airtime in the overall start-up world, but I think it's important!

Happy to answer questions and THANK YOU to everyone here who uses OpenRouter, and to everyone who has feedback for how we can improve!

Thanks! We work really hard to make sure we are ready at launch :)

(openrouter co-founder here)

Yeah we should do something to indicate cardinality. I can share that there can often (I'm talking generally; not related to this model in particular) be e.g. a very large app that can be pushing a lot of volume. But in almost all cases that app has a large number of end users. Hypothetically, for instance, would Cursor be consider one user, or millions?

Will think about it! Thanks for the feedback.

Can you share more? I'm with OpenRouter and we would love to address this! We don't see this in our own testing, I don't believe -- but will share this feedback and dig in.

Doesn’t it seem more plausible that the marketing shots are AI (where the “generated by AI” note appears) rather than the cover designs themselves?

You are absolutely allowed to expose access to end users, as long as you continue to abide by terms of service. We have hundreds, if not thousands, of apps built on openrouter that in turn have end users of their own. We showcase many of them on our /apps ranking page!

COO of OpenRouter here. Thats right — we haven’t done it to date but we can’t have unlimited liabilities stacking up forever. At some point we will start expiring credits from accounts that have seen zero activity in over a year.