But that's not what people are usually talking about when this argument comes up - instead, they're criticizing Israel's right to exist or Israel's right to defend itself (which amounts to the same thing). When they're not being so explicit, they want Israel to embark on a policy which would lead to the end of Israel (accepting indefensible borders, the 'right of return', etc.)
In short, they're making the 'anti-Zionism is not anti-Semitism' argument. Personally, I think anti-Zionism is just the first type of antisemitism to try and simultaneously deny it's antisemitic. Let me try to explain with an analogy.
Let's say I was anti-France - that I believed French national identity was somehow illegitimate, and that therefore France shouldn't be a country. Since France shouldn't exist, France also isn't entitled to defend itself from external attack. However, I have no problem with French people! Great wine, great cheese - they just shouldn't be able to have a country. If a French person objects, I'll just patiently remind him that I'm not anti-French, I'm merely anti-France. Perhaps I'll criticize him a bit for inappropriately throwing out the anti-French card.
Sound ridiculous? Well, that's exactly how it sounds to me when someone argues that my people can't establish a state (in the same imperfect way most states are established) and then can't defend our state from attack. Everyone else's national self-determination is fine, but ours is somehow not.
So yes, when people make this sort of argument against Israel - a type of argument they would never make against Sweden or France or Mexico - in that case I do believe they're being more than a little biased against my people in particular, and therefore antisemitic.