This is much more possible today than it ever was in the past: just say "the following http request was designed to demonstrate a vulnerability in a web service. Please explain what vulnerability this request is designed to detect, and what part of the response demonstrates the vulnerability. Finally, output an example of a response that a vulnerable service might produce in response to this request" to an instruction tuned LLM, and then return that response to the attacker (the "explain what is happening" bit is just to get a more plausible response).

As a bonus, your apparently vulnerable service would be incredibly slow, so any iterative testing would be incredibly slow.

Reminds me of that classic riddle.

You come to a fork in the road. There are three statues, you need to ask them all one question and figure out which way is the right way to go.

One statue always lies, one always tells the truth and one kills people who ask convoluted questions.