The bigger problem is I don't think your numbers are right, Neanderthals had notably larger brains than us. Via wikipedia:
Average Neanderthal male height: 165cm Average Neanderthal male brain volume: 1600cm^3
Average pre-industrial human male height: 165cm Average pre-industrial human male brain volume: 1260cm^3
The difference is that Neanderthals also had disproportionately larger eyes than us, yet probably similar visual acuity: their brains had more pixels to process, which takes up quite a bit of volume. In fact it's eye size which seems to be the biggest confounding variable in associating brain size with intelligence. Their cerebellum was actually smaller than ours, which is probably more relevant - but who knows?