Because a lot of groups that go under that name are violent and don't try to hide it. And just because they say they're "anti-fascism" doesn't mean you need to believe them.