People make those broad claims because they never been colonized themselves, and it may not be worth your time educating them on the difference since they haven't experienced anything else.
In my humble experience, they are disproportionately likely to be the descendants of colonizers and are sometimes uncomfortable with their own privilege.