This study basically says that Ghrelin makes you smarter. It may well be that this is to counter the effect of lower blood sugar due to hunger.

Being hungry could still make you "dumber".