> I suspect that many people who are good at number theory can probably wing it in a literature course.
You may be right, but the worst grade I got in school (as a Mathematics + CS major) was in some absolute throwaway, freshman-level liberal arts class (Intro to Japanese Popular Culture) I took in my third year!
Don't underestimate the value of staying within one's wheelhouse.