There may be some cost to each resistance gained, reducing the fitness of the bacteria
Ultimately resistance can evolve that kicks in only on exposure to the chemicals in question. Bacteria already do this with, say, the enzyme needed to metabolize lactose. The gene isn't expressed until lactose is present.