Yes, but it changes the dynamics. If the defendant can point to solid prior art, the patent troll will likely beat a hasty retreat. The last thing they want is to have their weapon taken away from them by the adults in the room.

Prior art is usually resolved by the patent office (or the patent trial and appeal board). It's only after they rule against you that you can take it to the federal circuit.