I think of it like gluten, or lactose, for most people its fine, but some people definately don't tolerate it and only they need to avoid it. If you have the enzymes required to process it then you're fine, if your genetics are missing something required for processing it then you'll have problems.