I built an eval framework to look just at tool calls given a static prompt, with the idea that LLMs should be able to deduce the best tool calls and arguments needed to get requested data. Not as great as full observability, but helpful for complex tool interactions. Anyone have any good tools for this problem?
In the same way we mentally walk through deterministic logic, SWEs need to learn to anticipate LLM context and tool awareness, which is much trickier to reason through, especially given the various LLM IDEs and how they manage context as a black box.
Respectfully, the reason you think “AIs suffer” is because of a shortcoming in your understanding of what an LLM actually is.
This scenario is no different than considering if a shovel gets tired after using it all day to dig holes in the ground.