I recall the story too: if I recall correctly, it was actually a branch of chemistry we had missed, not a math problem per se. The aliens stormed off the ships with muskets and we slaughtered 'em.
It's also my working theory of why we won in Independence Day the movie :) Aliens were never warring among themselves and/or allowing any kind of a hacking culture, so didn't think to secure their systems!
They were essentially a hive mind working with a single purpose and were therefore incapable of conceiving the notion that they would need to secure their systems against an attack that essentially used their own technology against them.