I don't know about overqualified, it's more that the job is hard in a different way. I'm not a CS grad, but it's not as if I'm actually using my numerical analysis or computational fluid dynamics coursework either.

However, I do jump between discussing if a trie is the right way to implement something, discussing whether the "simple" model should be a SVM or boosted tree, trying to figure out if we've covered the major abuse angles in our product, coordinating with all the other teams we're collaborating with, analyzing the experiment results, etc, etc.

Having at least some actual CS certainly helps fill in the gaps, even if it's not what I'm doing most of the time.