Partly because there are animals with larger brains which we now know are not very intelligent, with no assumptions needed, and partly because some of the most intelligent non-primate animals that we know of actually have very small brains - like crows and other birds in the corvid family.
The largest brains belong to elephants and cetaceans, and both are among the most intelligent animals. However comparing their brain size to that of primates would not provide useful information, because for both elephants and cetaceans large parts of their brains are dedicated to the control of various functions of their large bodies that we do not consider as relevant for "intelligence", including things like trunk control and echolocation.