Each cat can release toxoplasm parasites once, maybe for less than two weeks in 12 years, if healthy
If you have an old cat in your garden is removing toxoplasmosis (killing rodents that act also as hosts and crap also over your kale and under your carrots).
If you kill this cat or chase it off your garden, then new cats will apear and refill the territory. Think that you could be increasing the risk, in fact. Not to mention that people can pick up toxoplasm perfectly in a supermarket can of soup / rodent dance-floor. If you want cats out of your vegetables install sprinklers.
If you are concerned about the cat effect in birds, just add more natural areas and tall trees
Life is full of "why me?" and little dramas... If you want my silly advice, just try to enjoy it more.