I think it's because the price is posted. That's all.
If the price of milk were posted on every street corner, the braindead megaphone of mainstream media would amplify the concern for milk prices.
The reason they complain is like you said... it has somehow become a really visible number. The morning news here cites the price every single day, as if the price of a barrel of oil was going to determine how my day would go. For once, we actually have something worse (more meaningless) than citing the Dow Jones every day.
Inflation comes from only one source- an increase in the money supply. When the government prints lots of money, inflation gets high (like the %10-%20 we're seeing now). When the government isn't running printing presses in overdrive, then inflation is lower.
Milk, Oil and Gas prices are affected by a lot of factors such as regulation of these industries, farm subsidies, discoveries of new resources, productivity improvements, etc.
But when you define "inflation" as a measure of prices, then you speak in error. Using the CPI as a measure of "inflation" is really untenable, and its a measure of just how economically ignorant americans are that every news report uses the term "inflation" when the government releases their current CPI numbers (which, by the way, are so manipulated that you can't compare them from year to year as they don't measure the same thing from one decade to the next...)
CPI is a fabrication designed to keep people from realizing how much of their money is being stolen by the government thru the mechanism of monetary inflation...
BTW, the constitution has written into it the requirement that it be run on a gold standard-- this trick is not new and the founding fathers were well aware of the actions of the bank of england. WE were on a gold standard up until the 1930s-- the period where we were the fastest growing economy on the planet. Since then, we've been slowing down, and this is not a coincidence.
This is really important for startups because if you are measuring your returns in USD you will get a very differnet number than if you are measuring them in Euros, and both of those will do poorly compared to an objective standard like gold.
We have the situation where the economy has not grown very much but the DJIA is at 14,000 or something.... people think this means the economy has grown, but the reality is that the dollar has simply declined.
The dollar is not an objective unit of measure, due to monetary inflation.
Measures of the CPI that remove these derivative effects are better. They demonstrate the true responsibility of the fed, much more than a half measure of the cost of petroleum.