It's not about "looking racist"; or at least, it's not about public opinion. A racially targeted measure would violate specific laws and would be challenged in court, likely successfully.
And as a bonus upreme court practically ruled president can be lawless as he pleases.
SCOTUS: Nothing Trump does is illegal.
Trump: "does illegal things"
Courts: You can't do this, it is illegal.
Trump: "ignores courts"
Courts: "shocked pikachu face"