I can see talking in a monotone and fidgeting as an annoyance, but was it show-stopping?
The refusal to answer why your response was (in his opinion) wrong is definitely a problem.
I'm worried about this because I've been unofficially diagnosed with Asperger's (I believe I'm not, but I'm definitely borderline / close) and I intend to run a company. I do fidget and miss some clues, but I'd have answered if someone asked me why I believed an answer is wrong.
Edit: I'm sorry if I came across in such a way to merit a downvote. I was genuinely interested in the answer. I'd also appreciate an explanation of what sounded "wrong" in my post.