Once you're able to do that, it doesn't matter if you add him or not, go with whatever costs the least amount of energy.
If you take the LION approach, then yes.
If you are already selective then it's fine not to, but keep in mind that Linkedin is about business networking not friending and burning bridges without good reason is bad business.
I'll point out that bad workplaces make bad managers, too. All of the faults you list can be embedded in a corporation's culture.
Finally, it is an opportunity to place professionalism above some past personal friction in a workplace you have moved beyond.
"The most dangerous type of asshole is the one that thinks he's not one."
The only benefit in accepting this guy's "connection request" is that you would be introduced to his social graph. These are people who either think he's an a-hole and arent interested in meeting you (as his connection) or who are accepting of the way he behaves. Either way I dont think it'll benefit you.