If he's independently wealthy he probably has a lot more to lose than you do. You can use this as leverage. If you can get him in a position where he may lose a lot then you can probably get him to back off and drop the lawsuit.
So be willing and prepared to fight dirty (but still get a lawyer). He's bullying you, and against bullies it's often a good strategy to escalate beyond the comfort zone of the bully. He can afford to sue you because he doesn't feel vulnerable; losing the lawsuit isn't the end of the world to him. This is what you must change. Find a way towards mutually assured destruction.
If he's just a bully he's very likely to back off if you put up a fight.
However there is nothing wrong with counter-sue(ing) for the cost of your lawyer and the cost of the trial. Find a good lawyer and defend yourself.