Interesting. I would have thought that the training set (basically the whole internet AIUI) would have included various "teacher's version" exams with enough word problems with intentionally-distracting extra information, that the models would be able to ignore that sort of thing.
This sounds like they're inspecting existing models. Maybe a model trained specifically on "word problem" question-answer pairs (as in, the sort of things that show up on tests and always pretend that the sort of complications a domain expert would know about just don't exist) would do better?