"It was my understanding that you looking for someone who will help you with Django tooling. I can do that. Although number theory sounds interesting, I never felt the need of diving into it in order to solve any Django-related issue. I would be happy to learn more about how you think number theory relates to Django tooling should I start working here."
When people try to be important like that, it is paramount to take them at face value. In a lot of geek environments this happens all the time. People trying to push that one niche topic they really know about, because this is their comfort zone. The problem is: that one niche topic will rarely be a natural fit for any given conversation. More often than not just asking how they think this relates to the conversation at hand is enough to tame them a little bit.
Many times the stuff people say will contradict with the stated (or implicit) goals of a conversation. in this case the chosen topic was one with which the interviewer felt at home.
So another approach would have been to make it even more about them by e.g. instead of answering their question showing that you did your research: "Ah number theory! I saw you majored in number theory, I admittedly never had the need to dive too much into it — what role does number theory play in $Companyname?"