If you think you'd like TAOCP, but with just the math, not the programming (hah), take a look at Concrete Mathematics by Graham, Knuth, and Patashnik. I haven't read anything else by Graham or Patashnik, so I don't know how much their voice comes through, but Knuth's playfulness and joy in the material comes through strongly. It's a blast just to read for fun.