The grains of sand example should make it clear: it is not true that this grain is one of few at the beach, but it is true that it's one of the only. This makes it a poor expression for scarcity, and while obviously we know what was actually intended (as with "could care less"), that doesn't really make it any better.