2,500 karma · joined August 22, 2018
On the other hand, the newer variants also tend to benchmark higher so it's not quite a clean argument of "hey the new version eats more tokens"
For things where I'm not familiar with the code base, it can take days or weeks to become familiar with structure and flows.
If I can get an agent to help visualize and analyze the architecture and zero in on a subset of code, that can be a huge win.
For instance, we had some archaic "cache" that just dumped things into a static/module-level map. I tried a few different things to try to find it over the course of a few days and eventually gave Claude a Python REPL into a running process with pyrasite after some memory leaked and it traced through heap allocations and references to find the referent. It would have taken me probably 2-4+ weeks of just learning about Python heap to figure that out of my own.
Remember credentials to query database are in secret store x
Remember company documentation is in platform y
Remember bug fixes always need tests added
With the reliability of current LLMs, if you're outsourcing that much of your thinking, you're producing mainly slop and were never a good engineer to begin with.
If you have a quality threshold beyond "it appears to work" then agents still require a lot of hand holding and guidance
The commit should be a short summary <1 paragraph for quick context
I have an example here: https://gist.github.com/nijave/2873b8b10d8c732e46264237b0755...
Tldr; all the Claude models had identical tools and some used them efficiently and verified data while others did a crap job and hallucinated responses. Additionally, Exa MCP tools generally worked better even on older/smaller model (Llama)
You can also invert and have k8s cronjobs provision the generated certs into other infra
With this setup, you don't have to worry about the RHEL certbot snap updating to a broken version which gets blocked by SELinux...
With this setup, I don't have to grant 3rd parties DNS access.
I actually made a webhook that allows per hostname API keys to wrap dnsimple because they only had per zone keys and I didn't want each VM to have access to the entire zone. These challenges would have solved that by allowing the DNS record automation to pull record values from the VM instead of having the VMs push values.
I think someone told me dnsimple might have more granular keys now but I haven't checked. Iirc we have the same concern with external-dns at work (some things need subdomains on the TLD but we don't want to give external-dns access to the whole zone so we usually cname the TLD subdomain to a per environment zone external-dns is allowed to update). With this, we could have the same pull based setup that applies arbitrary rules to decide if a requested record should be created.
I think the main takeaway is allowing pull instead of push model
Could be achieved with cnames but it's an extra layer of indirection to deal with and doesn't fully solve the "semi trusted 3rd party" case
Most browsers support trust on first use for leaf certs
Removing attack surface is better than trying to hide it.
If the central question was "what is the bus schedule on `day`" and the model screws that up, it gets a fail in my book.
Also curious if Google Maps gets the timetables correct (assuming it has them).
Semi-related, I also discovered that the default web search/fetch tools are pretty primitive and Exa MCP annihilates them. I ended up doing some comparisons with Claude Code comparing built-in server-side to Exa and to a Python MCP that used SearXNG for search and Exa was a clear winner and Python+SearXNG ended up coming out roughly the same after a few cycles of letting Claude optimize the Python code and adjust SearXNG settings. Ultimately it landed on this (making some changes to optimize returning relevant context directly in the search results so the model didn't need an additional web fetch call) https://gist.github.com/nijave/604c43e3e0fdcd60f5280d3a6b109...
Opus 4.8 will burn 10k tokens trying to answer something 100% whereas GPT-5.5 will burn 2k getting it 90% which is good enough for many things.
Some personal testing on a "help me find that restaurant" prompt https://gist.github.com/nijave/2873b8b10d8c732e46264237b0755...
I did a smaller version on Azure and disk latency had a massive impact much more so than max IOPs (although their crappy storage offering needed like 64-128 iodepth to get advertised iops).
Results seem mostly in line with expectations. Iirc vcpu is threads so on arm64 you get 4 smt1 cores vs Intel/AMD you get 2 smt2 cores.
That's showing 3 layers of nesting: LPAR, hypervisor, nested hypervisor
Even the most basic business app has an app server and a database server. If they have 6 business apps, they'd have at least 7 dedicated servers (assuming we're allowing a database server to have multiple app databases sharing it)
By your own numbers that's 200x+ as expensive.
Really 20 containers is a pretty small app considering 5 app server containers, a DB, a cache, a load balancer, some monitoring/alerting crap 2x for redundancy.
Found this https://sixe.eu/news/kvm-nested-vms-ibm-power-linux-lpar