It's ok to let someone out if:
* they are waiting at a 1-way stop sign or uncontrolled junction
and
* traffic on the other side has stopped or slowed, so you are going to have to stop or slow down anyway. In that case, it's polite and efficient to let them out.
Otherwise, barring some unusual circumstance, you should proceed. When you have the right of way, and it's safe to go, then go - that's the rule. Forcing everyone else to slow down or stop so that you can let someone out is not courteous or polite. And, as you have seen, there may be people who are not required to stop, and who are not parties to your little arrangement with the other driver. You would fail a driving test (in any country that has a serious driving test) if you make a habit of this.
I want to reiterate that this doesn't make the accident your fault - and it may have been perfectly reasonable for you to slow down; I wasn't there.
I agree with your wife and don't think you caused the accident. It was their fault for not being more aware of their surroundings.
Lay low and pray no one escalates the matter into a frivolous lawsuit, although the incoming driver is clearly at fault. Continue waving people in when slow traffic conditions warrant, as this is a very normal and nice thing to do.
INdirectly, and not even that. If she had been a few seconds behind where she was now, you hadn’t stopped, he waited for a few seconds to let you pass and then hit her, would you feel guilty, too?
The problem is OP performed an unexpected action which created an anxious moment for the incoming car who drove in hastefully to minimize OP's bending over backwards. A cyclist parallel to OP could be excused for barrelling ahead assuming right-of-way traffic remains right-of-way.
I was always taught to never wave or otherwise indicate to another driver to proceed but slowing down is fine, and never ever ever EVER act upon the invitation or waiving from a pedestrian to proceed(Such as a person waiting at a crosswalk that you stop for that waives you to go) as the person in the vehicle is always in charge of that situation. I know this was not your situation but feel it is relevant enough to mention.