Over the last few days I’ve been cobbling together a little service which asks GPT to identify issues with code, then if it can generate solutions, the service attempts to apply the solution and run the code.
I’ve been learning to dial in prompts and finding this truly is a major factor of the job that needs to be just right. I went from unreliable response formats, hallucinated solutions, naive solutions, and so on all the way to fairly conservative, pragmatic, consistent responses.
My success rate isn’t as high as I’d like, but I find it absolutely remarkable.
My key takeaway so far has been that a lot of perceived limitations are in fact a lack of exposure to what effective prompting makes possible. Another feature that’s powerful yet overlooked is what recursive prompting can accomplish. This is what makes me think complexity might be overcome in the not-so-distant future.
For example, if I apply a solution and it breaks something else, I can then reform a prompt based on the previous prompt along with a new one informing it of what went wrong. These little recursive exchanges have actually yielded useful results in multiple tests. That’s a huge deal in my opinion; this is totally automated by a smooth brain like me.
As token limits increase and the models become more capable, I suspect this tool will actually improve without me changing much. I suspect it could still work quite a bit better and more efficiently if I could more succinctly provide context about the code in my prompts, too. That would overcome token limitations for the time being and likely improve accuracy quite a bit.
So, if things like this are possible already… I have a feeling we’re going to overcome the complexity issue sooner than it currently seems.