I'd recommend using this book as a rough guide:
Don't let the title fool you. It's really a guide to advanced applied mathematics.
I'd also suggest the material be supplimented with a good bit of statistics. In my experience, statistical reasoning (particularly as it applies to noise, perturbation, and system reliability) tends to be weakness with hackers.