I've been looking into this myself and I have a similar background as you (elementary calculus and linear algebra; I also took number theory as part of a math minor). So far it looks like I should pick up a bit of abstract algebra/topology (rings, fields, groups, etc) then I should be able to start off on the shallow end of category theory and work from there. At the same time I need to improve my understanding of how monads, monad transformers, monoids, functors, etc work in Haskell from an operational standpoint; I have a basic understanding now but I'd like to dig deeper.
The thread I linked above has a lot of links to books and stuff like Reddit's r/haskell and other places, that you can sort through to see what works for you. I have yet to make that determination for myself, but there's plenty to choose from. I actually picked up a copy of Benjamin Pierce's "Basic Category Theory for Computer Scientists" a few years ago based on someone's recommendation, way before I got into Haskell, and didn't make the connection until I picked up his book on type theory recently (while researching a paper on FP) and thought his name looked familiar.
Ultimately I'm hoping to be able to get deep into type theory and go through that Homotopy Type Theory book that came out recently.
As for category theory, after reading through that thread [1] and the Reddit thread [2] that it links to, I think Conceptual Mathematics: A First Introduction to Categories [3] will be my first read on category theory (but probably followed by the Rosetta Stone paper). That book receives some strong recommendations as a first book in the Reddit thread [2].
[1] https://news.ycombinator.com/item?id=6041726
[2] http://www.reddit.com/r/haskell/comments/1ht4mf/books_on_cat...
[3] http://www.amazon.com/Conceptual-Mathematics-First-Introduct...
[1] http://www.reddit.com/r/math/comments/1eiyid/category_theory...
[2] http://www.amazon.com/Algebra-Chapter-Graduate-Studies-Mathe...
[3] http://www.mimuw.edu.pl/~jarekw/pdf/Algebra0TextboookAluffi....
If you enjoy it anyway: I like (http://www.haskell.org/haskellwiki/Typeclassopedia) for how it discusses how the different type classes in Haskell relate. I found this a lot more practical coming from this direction but your taste may vary.
I'd also highly recommend going through Types and Progrmaming Languages by Pierce (http://www.cis.upenn.edu/~bcpierce/tapl/) if you are interested in the area. This gives you a good background on type system stuff.
There are tons more links to category theory stuff here (http://mathoverflow.net/questions/903/resources-for-learning...)
Some people like thinking about category theory when they program but in the end its still just a bunch of (cleverly organized) abstract interfaces.
As for background, I think some familiarity with set theory would be most useful, groups and rings and such a bonus. Calculus won't be of much help. Linear algebra might, depending.
of course, that doesn't have much to do with haskell, but if you have an interest in abstract algebra beyond its applicability in haskell, you're in luck, because it doesn't seem to require much background. all it needs is for your head asplode.
caveat: i've tried to learn category theory, and found that i could barely follow a proof, let alone write one. so i picked up a book: http://www.pearsonhighered.com/educator/product/Mathematical...
and i'm working through it now. i could come back to category theory and find that i'm still lost, but i don't think so.
i haven't really had much trouble writing haskell without a decent math background.
Your implication is that "jump in the deep end" means learn category theory. That is entirely unnecessary. If you are interested in category theory, by all means go for it. But don't think it is in any way a haskell requirement.