We're considering doing something similar in my department. Based on some back-of-a-napkin math, it won't (yet) be cheap though, at about $0.40 per query.

This is assuming the use of gpt-4 (chatgpt would be a lot cheaper, but it doesn't follow system level instructions very consistently, often spilling answers instead of replying with feedback, hints and leading questions).

Also, projected costs per query are high because we want to feed it quite a lot of input tokens. The assignment text, the student's code as well as the reference solution code.