This indicates that he views a certain type of worthlessness in math. Then he can justify not being good at math, and distance himself from the pursuit of math on its own grounds.
Of course his perception of him not being good at math and actually not being good at math are two different things. Physics uses math, but not for maths sake. And in reality, I'm sure he was more than competent in math, so the level of math required for undergrad physics was not a problem.
It was only, through his journalism, did he let down his guard enough to enjoy math for maths sake -- and realize that a test you took in elementary school needn't define you.