OP's description very much mirrors my experience. I was able to do many things much more quickly with the help of the LLM, but there is no substitute for actual users interacting with the tool, saying, "I like this," or "no, this is wrong or needs work," or even, "here's something none of us thought of before, but now this tool makes me think XYZ would help us and might be achievable." That whole interaction takes real time and I don't know how you replace it with coding agents.
Secondly, on the matter of code structure, just on a qualitative level, I can see that claude will do things very efficiently on the way to a goal I give it, but it can't read my mind and know that I may want to repeat a specific pattern across two client apps. Or that its shortest-distance solution makes extensibility or broad applicability difficult. That I might want to share code and structure things in a certain way. Not without me saying so or, in many cases after it has built something workable, duplicating refactors I make with an eye for reusability or maintenance. Making those changes in time is important if you don't want to burn tokens later as the LLM tries to unravel its own spaghetti. And again, whether I am coding those intentions directly, or writing out detailed instructions in english, all of that takes time.