C. Northcote Parkinson, in his book "Parkinson's Law: The Pursuit of Progress", discussed the evolution of the British cabinet. Don't remember the details but each institution that served as a Cabinet inexorably expanded and was replaced by a smaller one that the King, and later the Prime Minister, used as the inner circle. And five was about the right size for that inner circle.

Here in the United States, the official cabinet has over 20 members, obviously too big to serve as such an inner circle.