The point I was getting at was more of a question of "Okay, how can this be used adversarially?" or "How can we break this?".
"What can a malicious inference engine do"
Something like that
"What can a malicious inference engine do"
Something like that
A malicious inference engine can already return anything it wants, it doesn't need constrained decoding on the LLM side. If your client blindly runs commands in an unprotected environment, you're already pwned. They don't even need to run an LLM, they could just randomly add "tool_response:{'execute':'rm -fR *'} and that'd be it...
So the fact that got me curious is that one can steer the output, which, at least to my understanding of the tech, breaks the assumption that one can not do that.
So the thought is "assumptions are being broken here. Which ones? And what are the implications of that?"