Bags collect condensation from cold products, which attracts mold. They collect dirt from raw vegetables and from other surfaces they come into contact with. (Imagine how the seat of your pants would smell if you rode a city bus for a week without washing it. American public surfaces are filthy!) Worst of all, some grocers just can't seem to keep the bottom of milk cartons and other dairy products clean. A hint of stale milk + condensation + fabric + time = a perfect recipe for food poisoning.
One problem is that most reusable bags aren't designed for frequent cleaning. Canvas bags get ragged after only a few cycles of typical washing & drying. Some bags are built much stronger and heavier, but they need to be hand washed and thoroughly air dried. I just can't see getting enough uses out of them to offset their carbon footprint.