Then you should have a policy that says "English only during office hours", not one that says "no foreign grad students admitted".

These students had already received an acceptance letter, so possibly they already turned down other positions based on that letter.