'colo[u]r' is merely a spelling difference. 'irregardless' is a double-negative to some people, and not to others. It's an ambiguous word.
Or should I say 'irreinterpret it'? After all, you would 'know what I meant' if I wrote that.
Ambiguity is fine if you're writing for poetic effect, but if you're trying to get a point across, it should be minimised.
And it does fit the pattern of negating the root morpheme.
[1] why not irregard instead of regardless?
Also, most English dialects don't strictly follow double-negation rules. I know this drives non-native speakers crazy, but you'll get over it with enough practice.