As an American English speaker, I have never come across context where a handful means about five. Handful has meant a relatively small amount, with the actual quantity always depending on the context of the item being discussed.
When the same term is used in my (western European) language, I would probably put the assumed range in 4-6, potentially 3-7.
- The number of items being quantified is small
- You could make and understand a list that included them all
- I don't know the exact number offhand
But what counts as a "small number" varies depending on what you're talking about.