The idea that magic is a branch of maths (mixed with heavy doses of Lovecraftian horror and dour cold-war spy thrillers) is the basis for the splendid Laundry series of novels from Charlie Stross:
http://tvtropes.org/pmwiki/pmwiki.php/Literature/TheLaundryS...