Hacker News Reader

Comment by segmondy | Hacker News Reader

"We deployed GLM 5.3 with a 1,048,576-token context window, but the KV cache pool on our prefill GPUs only had room for 681,280 tokens. We were aware of this mismatch before launch. "

Did they advertise it as 1,048,576 token when they only had room for 681,280 tokens?

View on HN

Next sentence: "We expected the engine to return an error for any prompt it could not process, and our long-context tests returned successful responses, so we did not believe this to be an issue."

Which is pointing the finger at Sglang, who also weren't responsible for this.

At a minimum someone should have realized during testing that their utilization figures are off by 40%.

A much better title would be "Cloudflare configuration error leaked data between user sessions"

I mean they are responsible for this. It's definitely a bug on their end.

Doesn't matter, they could have marketed it as 1 million token window, you would then infer, get to 600k and get an error. You won't be able to finish your work, and you won't get credit for 600k tokens.

They thought it worked up to 1m token window

why would they think that? I'm not a pro, but I know you can't serve more than kv cache pool that you have. If you want to run concurrent requests, you need to have more kv cache pool than your context window. The entire thing is weird, my bet is they were thinking most people will be using about 100k cache, compacting, etc and not nearing 600k ever.

See my other comment, this is likely only for the prefill GPU pool, a normal 1M token session would work fine. I think Cloudflare should probably link an explainer on how model serving works at the top of the post because a lot of people are not familiar at that level of detail.

Bro, I had to read this sentence in the context of the previous one, 3 fcking times.. and it sill does not make any sense to me.

Sounds like they are saying they tested with a long prompt and just tested for a failure, but the test likely would have seen that same KV cache leakage and just interpreted it as success.

thank you...

So they knowingly deployed a configuration with an overflow risk, much like writing code with a buffer overflow exploit.

They assumed that it wasn't like an overflow exploit, despite the fact that's exactly what they'd set up to happen, on the grounds of thinking the equivalent of, "oh this probably uses smart pointers, we'll be fine".

That's how I read it anyway.

I read that as you can only do a single prefill up to 681k tokens, separate from cache hit or generation. I think people try pretty hard to avoid a cache miss on sessions that long and rarely do standalone requests that big so the situation triggering this condition is rare. If you run a normal coding session with many turns it would get to 1M ok.

Shit happens, this is normal. However, this:

> We were aware of this mismatch before launch

Isn't normal. Am I misunderstanding or are they openly confessing they've stopped caring about quality at all? Why would you launch when you know it's wrong?

Maybe they forgot to run this one by the lawyers before posting. I'm surprised they'd publicly admit to that

Reply on news.ycombinator.com