First, almost anything can be prevented by the torturing, killing, or detaining of other people, as long as you involve enough people:
Kill all terrorists -> no more terrorist attacks
Torture all terrorists -> learn enough to prevent attacks.
Nuke entire country -> nobody left in that country to attack you
Detain everyone you ever find on a battlefield -> unable to get back to battlefield
Trying to apply some cost benefit to these mechanisms is certainly complex, as you say, but that actually makes it worse here:
None of these complex cost-benefit tradeoffs are at all changed by Gitmo being there. There is no material difference in these tradeoffs between a prisoner in Gitmo and a prisoner in the US other than the legal rights being afforded the prisoner.
Instead, they try to say "well, they are in gitmo for a reason". This is actually completely irrelevant to preventing the next terrorist attack. Either you need to torture them/deprive them/render them/whatever or you don't. The fact that they are in Gitmo does not change this.
The only thing closing down Gitmo or not closing down Gitmo does is allow certain folks to try to play fast and loose with the legal and political realities around us having a constitution and being a nation of laws.
As an example:
We find a guy. He may or may not have been complicit in terrorism (he was a driver for a bad person). He may know the location/time of the next terrorist attack.
Torturing him (or not), detaining him forever (or not) certainly presents complex tradeoffs. None of these tradeoffs change whether I put him in Gitmo or not.
The only thing putting him in Gitmo does is:
1. Often make us feel better
2. Allow you to try to deny him what would otherwise be various rights he would have if kept in the US
3. Keep him out of sight and out of mind, and enable you to more strongly control the public narrative about the guy.
As you can see, none of these are related to the complex tradeoffs involved in deciding what to do with them, only the legalities of those tradeoffs, and ability to convince the public of those tradeoffs. We should not play fast and loose with these legalities, we should be straightforward and clear, and accept the consequences.
So, if the argument is "well, if we close Gitmo, to get the same effect we'd have to make torture/deprivation/indefinite detention legal in some cases in the US" (which is what you seem to be saying), we've already got a problem, and it's not related to closing down Gitmo or not.
If we want to declare torture and indefinite detention is okay in certain cases, that should be a national discussion we have, because, as you said, it involves complex tradeoffs.
Right now, with Gitmo open, it enables a lot of folks to ignore having this national discussion, and thinking hard about which tradeoffs they are willing to accept.
This, of course, is why Gitmo is still open.
It's the only available compromise position between the two currently extreme sides around what it is okay to do to in the situations you posit that doesn't involve either of them really losing.