In addition to the point skoob makes, there are also the considerable economic and societal costs of an epidemic. These costs are borne by the society as a whole, and therefore the society as a whole has a prerogative to minimize contagious epidemics.

By your same reasoning, the Holocaust should not matter to you as long as you’re not a Jew or a homosexual. The point is that mass sickness has an appreciable effect on you even if you don’t contract the illness yourself.