Yes, that's my understanding too. But if the party asserting the patent decides not to defend it, and drops their claim against you specifically, can you go ahead and invalidate the patent in court? If you can, and there's no defence, then presumably the patent is invalidated by default.
But it seems rather odd; in normal civil proceedings, you have an adversary against whom you allege some kind of wrongdoing. Who is your adversary if you are attacking a patent that nobody cares to defend? Can you bring a case against the patent itself?
I understand that in parts of the USA, the police can sue a car, a house, or a sackful of cash, so it's not beyond reason.