You can sort of do this.

For a given bug one could write a test that prove its existence, this gives the LLM a target that they can actually iterate towards.