I disagree with your implication; modular arithmetic is much more obviously useful, and would have been back then too (supposing hypothetically you explained the notion of a galactic algorithm to Fermat), because it's an abstract mathematical structure and thus likely to recur in new scenarios.