What's really being asked is, what if you stop performing any better than an unmodified frontier model with no system prompt?

It seems you are betting on frontier models improvement being slower than the pace of new features in your tool

If this bet doesn't pan out, the only thing I can think of is pivoting into something else

I think all of those are very naive approaches. What I currently do is script out screenshot + application state pipelines and then run those screenshots and states through clef with questionnaires. The scripts are deterministic, clef costs pennies or less and is also reliable, the questionnaires are derived from the script with escalation to stronger lmm's if necessary (historically very infrequent from my usage). It isn't a "drop in" solution, but it is fairly trivial to set up and you can attach it to CI. It isn't perfect, but I've found it to be more successful and way more cost effective than throwing Claude or GPT models at it regardless of harness.