Objective: It seems that the world is largely organized into two classes of people: those who care about precision of speech beyond the simple need for making someone else understand your point, and those who believe that as long as everyone is on the same page, then for God's sake, it bears no further discussion. "I could care less" apologists are in the latter group, clearly, but it is important for both groups to exist and speak up. Of course, this means that while I disagree that he shouldn't have mentioned it, I agree that you should have criticised him for mentioning it.
Subjective: My disgust with "I could care less" goes beyond mere imprecision of speech. There are people in this HN thread claiming that when someone says "I could care less," they mean "I could care less, but it would require effort." I think that this is definitely not what most people think they are saying when they say "I could care less." I think most people are trying to express, "I couldn't care less even if I tried," but are failing to express that because they are simply regurgitating things they have already heard. This is a pernicious habit indicative of a deeper lack of thoughtfulness and creativity in our society, and it drives me completely crazy.