Any statement of the form "[Racial identifier] men/women/people are [criticizing adjective]" is per se racist.
If one wants to argue that wide-ranging systemic racism is worse than incidental one-off racist comments, I totally get that and I'm inclined to agree. But trying to redefine "racism" to exclude racism against a certain group is not acceptable.
1984 was a cautionary tale, not a how-to guide. If a proposed method of political argument requires redefining the English language, that should give one pause.