Hobbyist harness builder here. Thank you for the article.
What would be helpful would be a cheap API call that would give me the energy cost estimates:
> Given some prefix-cache uuid, what is the cost (in mW of GPU energy) of adding X tokens that will branch off from what's been cached at offset Y tokens.
Suppose at some turn n, I read a file and pull it into the context. Now at turn n+m, I discover that the file has mutated on disk. The most energy efficient option might be any of these:
1. Replace the content of the file in the cache
2. Append the new file to the end of the cache
3. Add a thing to the context saying the file has been updated.
4. Compact memory then do option #3
As a harness builder, it's pretty hard for me to pick the most energy efficient strategy - I assume every model has a different cost for prefill vs decode and I also expect that a provider (an inference cloud) might have slightly different implementations that would impact how expensive a prefix-cache miss is.
Ofcourse, I also need to target aggregators like openRouter because as a solo hobbyist harness builder, it's really hard for me to track every cloud's unique API. Then there's the fact that I'm optimizing for lowest billed cost, not lowest energy consumption.
I'm waiting for the industry to standardize on a good way to express these in an API so that I can start doing these tradeoffs at runtime in my harness.