There are definitely people in situations where the burden of a vegan or vegetarian diet would cause much more harm to them personally than the harm an omnivore diet would cause. I don't blame people for the uncontrollable, and I don't know that anyone who holds a well-thought out position on animal suffering would either. Suffering is the superset of all suffering. My ethical heuristic is to make choices with an eye toward reducing that set.