If you use SIP and use package managers (npm, cargo, pip, etc) outside of a VM you are substantially more vulnerable to attack than someone who doesn't use SIP and doesn't use package managers.
So if you want to fix your corners, you can do it guilt-free by adopting some better security practices around the malware delivery systems / package managers that you have installed on your computer.