Have you seen the research about "ablation"?

https://www.lesswrong.com/posts/jGuXSZgv6qfdhMCuJ/refusal-in...