Right, that is how we do it. But isn't all the phase information present in two channels of audio to know how to map all the signals to an arbitrary number of angles? You'd need to know the spacing of the mics, but that could be a standard.
About the pinna, a fun way to demonstrate this is to have someone close their eyes, then you snap or jangle keys at various locations and have them point. Then tape their ears to their head and repeat. They will be way off in the vertical. We can do left right-ish (with coning error) without the outer ear. But up down is impossible.