The cliché is that humans are fundamentally social creatures, and I think one of the primary ways we maintain the high engagement in a discussion thread is by becoming interested in the other participants, not just the topic. That's usually a good thing, because it motivates us to respond and care about the discussion, but it also means that people will go off-topic, get into heated arguments, or bring up memes and social topics.
If it were possible to strictly enforce rules against off-topic or non-technical discussions, I suspect engagement would drop very quickly. Learning is a strong motivator, but I doubt it provides the same semi-addictive feedback loop that the social aspects of a site do.
(I don't have any evidence beyond my own gut feelings based on web forums over the years --- happy to be proved wrong! A high-engagement non-social discussion forum would be an interesting place.)