Trying to know everything does not work. Targeted, traditional police investigation works. Money diverted from the latter to the former is wasted and reduces safety.
These programs have been in place since at least the Clinton years, yet that didn't stop, say, 9/11. Nor did it stop the recent Boston bombers. Cases we do see stopped are often by lone nutters "buying" terror materials from law enforcement agents posing as terrorits. Kudos for catching these guys, but they are completely unsophisticated enemies. The sophisticated ones don't go on AOL chat asking for plastic explosives.
There are probably practical limits with SIGINT domestically. I think the argument that stopping crime is just like detecting a hacker on your network is fairly weak sauce. Our prevention outcomes will probably be the same if we started cutting these programs.
The thing is, security agencies (not just in the USA) have failed, repeatedly, to act on information they did have. Part of if may have been due to inter-agency rivalries, part of it may have been the agencies drowning in data and being unable to draw any useful intelligence from it. But with this in mind, the demand to let them gather more and more data seems unlikely to solve any of these problems, even if one does not care much about privacy.
And the reality is, effectiveness is complete irrelevant: the search is unconstitutional, even if it demonstrably saved lives.
Americans are just too willing to throw their freedoms at the boogie man.