Hacker News Reader

Comment by cfn | Hacker News Reader

I tried Cerebras this week for my project and their 27b model at 2000tks would be perfect for it. Unfortunately it doesn't do the 2k tokens per second which is what I am looking for.

View on HN

We can push 1k tps on our speed lane with the right request parameters with deep seek v4.1 flash. Is 2k a hard requirement?

I will check it out! I am using that exact model and it is very verbose when compared with the latest Muse, for example, increasing the completion time a bit too much.

are you evaluating using OpenRouter.ai or other routers/gateways?

they have never been able to keep up with Groq/Cerebras-level inference speeds for some reason.

I am using opencode and Inception.ai (the fastest so far)

And the fact that you will never be benefited from the batching they’ve explained either, you’ll be paying for every token sequential or parallel unlike nvidia or amd gpus

What's your use case?

I bet it is large-scale data crunching for labeling or categorization.

Probably not because this is usually not a time sensitive task.

High frequency trading? :)

Fast LLMs seem to be mostly useless, otherwise it wouldn't be so hard to come up with realistic use cases.

You think too negatively about its use. My company definitely needs it desperately.

If it did, you would be able to specify for what, you can't specify for what, so it doesn't.

I need to have the answers faster it is that simple, I don't want the users to wait for more than a few seconds.

I literally said it in my first comment, which is large-scale data crunching for labeling and categorization.

Why is it time sensitive?

Sorry, can't say just yet but nothing as dramatic as what was guessed below :)

what is a project that requires 2k tokens per second?

It's not launched yet and 2k isn't a hard requirement but faster inference of relatively simple prompts make a big difference for us. Cerebras also failed because it has very small context window.

Well, a weekend project that requires 345'600'000 tokens — you only get so many seconds in two days ;)

Reply on news.ycombinator.com