I'd guess the chances of infection go up if you have a vegetable garden in a suburban environment. Cats use freshly turned soil as a litter tray and salad vegetables such as radishes and spring onions are often eaten raw.
Being eaten raw doesn't mean being eaten without washing. That should get rid of most of the stuff. My understanding is that cat litter is a danger factor because it gets dry and fecal matter particles get thrown in the surrounding air. Which wouldn't be a problem in the garden.
Scrub those vegetables well, friend!
I wonder if rabbit repellent also deters cats.
You can buy bottled coyote urine. Pretty sure that would work.