Yep that is something I've been dumbfounded by as well - the official Microsoft phi4 upload also suffers on MATH so at least we can rule out it's because I did something wrong.

I thought of two possibilities:

1. 509 does better on MATH but absolutely terribly on IFEVAL because it does not use a chat template - whilst others so use the chat template.

2. I think HF uses exact matching I think so maybe that's the culprit.

I can test 1. by resubmitting without using the chat template!