The stores are getting wiser about this. My local Fred Meyer (a Kroger brand now) has a fuel rewards program -- for every $100 you spend, you get 10 cents off per gallon on your next fillup. Given how expensive groceries are, a lot of people are saving more like 50 cents per gallon, not 4.
They've also started doing instant discounts at the register, which was something that Safeway aggressively did from the beginning. FM isn't quite that aggressive yet, but when I scan the shopper card just before paying, it isn't unusual for it to knock $20-30 off a $150 purchase.
If it really were just 4 cents a gallon, I expect less people would bother. But it's not. The stores are steadily increasing the penalty for shopping without a loyalty card.