I have read but not verified that washing with water is about as effective as with soap at removing bacteria from hands. Having soap on your hands forces you to get all the soap off, which requires touching more skin surface.
Hard to find anything on it now, and I understand soap was particularly effective against SARS-CoV-2 compared to water alone (can I just call it the COVID-19 virus?) so it looks like the idea is mostly gone now.