1,805 karma · joined July 7, 2012
But I guess you mean the discovery part? Yea, agents are not going to totally replace that anytime soon. Though, Claude voice mode is pretty good these days, so I’m sure it’s on their mind.
Not sure what you mean by “stay the course.” I think the winners will be those with the most effective marketing strategies. Surely that will require automation.
You’re not running tests in CI? The agent should only run the minimum tests necessary and then idle and wait for feedback from a CI run.
Yesterday, I ran an identical bug identification dataset from two weeks ago, saw a 50% drop from a few weeks ago, putting Sol on the same level as Luna. Sol had been finding 40-50 bugs per set, then dropped to 25, matching Luna’s performance. Not enough to establish a pattern, but enough to raise eyebrows.
Our review workflow is public if you want to peruse it, dataset isn’t. The process isn’t really stabilized yet either as I have to balance running this against limited budgets.
https://github.com/BiggerPockets/.github/blob/main/.github/w...
I use these loops to monitor the CI build and PR approvals rather than having the agent poll, and even Opus gets the commands wrong enough to make it worth it.
Last week I wired up a skill for the agent to share screenshots in PRs via specific S3 buckets and AWS CLIs. Again, the agents guess at the right commands often enough to make it worth being explicit.
Sure these could have gone in CLAUDE.md, but not every agent needs the context.
And at the company level, I can push skills to everyone’s Claude via the Teams function, they don’t need to edit configs or even know what a skill is.
https://github.com/BiggerPockets/.github/blob/main/.github/w...
Still a bit nicer to just do these yourself rather than reusing someone else’s project.
Same for being stuck in local maxima: getting out of that requires big changes, not small-step optimizations.