samwho

How would information leak, though? There’s no difference in the probability distribution the model outputs when caching vs not caching.

The only thing that comes to mind is some kind of timing attack. Send loads of requests specific to a company you’re trying to spy on and if it comes back cached you know someone has sent that prompt recently. Expensive attack, though, with a large search space.

I was wondering about this when I was reading around the topic. I can’t personally think of a reason you would need to segregate, though it wouldn’t surprise me if they do for some sort of compliance reasons. I’m not sure though, would love to hear something first-party.

With KV caching as it’s described there it has to be a prefix match. OpenAI state in their docs they don’t cache anything below 1024 tokens long, and I’m sure I read somewhere that they only cache in 1024 token blocks (so 1024, 2048, 3072, etc) but I can’t find it now.

There’s been some research into how to cache chunks in the middle, but I don’t think any of the providers are doing it yet because it needs the prompt to be structured in a very specific way.

Could you tell me what browser/OS/device you’re using? A few people have said this and I haven’t been able to reproduce it.

Another person had this problem as well and we couldn’t figure out what causes it. We suspect something to do with WebGL support. What browser/device are you using? Does it still break if you disable all extensions? I’d love to fix this.

It’s funny, I didn’t set out for that to be the case. When I pitched the idea internally, I wanted to scratch my own itch (what on earth is a cached token?) and produce a good post. But then I realised I had to go deeper and deeper to get to my answer and accidentally made a very long explainer.

Yay, glad I could help! The sampling process is so interesting on its own that I really want to do a piece on it as well.

Yeah! It’s planned for sure. It won’t be the direct next one, though. I’m taking a detour into another aspect of LLMs first.

I’m really glad you liked it, and seriously the resources I link at the end are fantastic.

Huh, when I was writing the article it was GPT-5.1 and I remember it got it no problem.

Thank you so much <3

It is the same ngrok!

The product has grown a lot since the mid 2010s. Still got free localhost tunnelling, but we also have a whole bunch of production-grade API gateway tooling and, as of recently, AI gateway stuff too.

Thank you so much <3

Yes, I recently wrote https://github.com/samwho/llmwalk and had a similar experience with cache vs no cache. It’s so impactful.

The one where he does a landscape is one of my favourite videos of all time.

Someone gave me the advice to use animals, ideally animals of very different sizes or colours. People instantly picture them and remember them.

This is what I use and I’ve been very happy with it for many years. It hasn’t caused me any trouble and as far as I can tell it hasn’t changed in the whole time I’ve used it.

Hehe, yes, Ned sent me a lovely email the other day about it. Happy to be doing my part.

Can’t imagine what it might be.

I’m not sure that’s quite my position. Happy to cede that I lack fluency, and I appreciate your time and the time others have given to help me understand.

I did push up a rewording that hopefully lands better. Fingers crossed.

Thank you, I appreciate that a lot.

All the code is at https://github.com/samwho/visualisations. I didn’t use much more than basic JS, CSS, and HTML for this one. The only third party dependency, iirc, was one to parse JavaScript into an AST.

I’m glad you like them! <3

Ah, I conflated worst case with upper bound. My mistake.

That’s a good idea, I should try that.

Ahh I see, thank you!

They do! Part of me wanted to go back and change the sum function to something else but in the end I figured it’s not important.

I should have probably been a lot more clear about who the target audience of my post was.

I replied to this effect to someone else in this thread, but I think it's reasonable for people to want to have an idea of what big O is for (in software engineering!) without having to have a grounding in calculus. The notation is useful, and used, without it regularly.

I made some wording changes based on what you wrote, I'd appreciate your feedback.

I'm struggling to understand how the part about big theta is false. Is the best case for bubble sort not O(n)? I've deleted the section for the time being.