If this is the situation, I would not be so hard on yourself. It seems like there's a lot of history and emotional trauma that would need to be resolved before any progress would be feasible. If this is indeed the situation, getting your wife involved in the process could make it worse, not better. Imagine all the fights they'd have with so much on the line, when they'd have so many fights before when there wasn't much on the line. You just walked into an impossible situation and did your best. You probably would have done better if you were a sympathetic complete stranger with whom your brother-in-law had zero emotional history (and even if he had zero emotional history with you, he certainly had a lot with your wife and probably subconsciously viewed you as her proxy).
edit: Thinking about it, I think the only thing I'd be able to say in that situation is something like this, just be supporting, not judgmental, not trying to convince him about anything: "Hey man. I'm really sorry this is happening. I know that things are hard right now. If there's anything I can do to help you or the situation, let me know, and I'll do it. How are you yourself? Do you need to talk with anyone about how you're feeling? Let me know if I can help, I'll be right here, OK?"