Holding him responsible, no. Demanding he recuse himself: yes. His vote after a failure to recuse on a key judgement now looks extremely suspect, and he should be recusing himself not just to avoid a conflict of interest, but to avoid the appearance of a conflict of interest.

Failing to recuse himself has really caused him a level of reputational damage that non-lifetime appointees would not survive.

Kind of a cosmic irony to me that after years of being the Supreme Court's most reliably banal, consistently aligned judge after a confirmation hearing where he was accused of sexual misdeeds, it's his long-suffering wife who might bring him down.