You could do very much worse than Katz & Lindell's "Introduction to Modern Cryptography". It's fairly mathematical, which fits well with Colin's point.