Though the cog for settings is, like the three-dots commonly used to represent a menu itself, is only recognised as such because it has often been used that way. The first time(s) it was used it would not have been obvious to people. So the rule isn't that a good icon should be obvious. A good icon is either immediately obvious or consistently used.

The common save icon is a good example of this in both cases: to those of us who have been around a while it is obvious because it looks like a floppy disk, to those who never used floppy disks it is obvious because that is what save icons (almost) always are.