If you allow any term which has ever been used in a racist manner to be considered offensive you give enormous power to racists. They get to DoS attack your language by using whatever they want as slurs. I don't like empowering racists.
That is, if your issue with this wording is only that it's subjectively, personally unpleasant to you for inner reasons, and not that you see it as actively harmful.
Note that is just a parallel with colored functions, the "colored" word does not mean the same thing here (the meaning of "colored" in graph theory has little to do with the one used in programming).
Nonetheless, if I think that using this word is upsetting to some group of persons, whether I understand their reaction or not, I try to avoid using it. On the Internet it's difficult, because we're addressing ourselves to a very large potential audience.