You can get near this point with scaffolding. Keep in mind, LLMs are next word predictors at their root. More abstractly, they capture and replay likely human intelligence by way of written language. Tokens.

With that concept in mind, it's clear how they can be made to "give up".