Were you able to get the diff to work with GPT 3.5? It kept giving me incorrect line numbers. Not sure if I was doing something wrong or if I need GPT-4 for that.
Yes I see that some good percent of the time with GPT-4 as well. trying a few things to see if there is room for improvement.
I am having better luck having it output blocks of new code and then summarizing it as a diff. I think this is another case where the model has processing gain when it "thinks it through step by step" versus trying to jump to the final answer in the form of a diff.