I agree with that point with respect to the host's fitness.
But the conflict here stems from the fact that those other constraints (such as the host's long term survival or reproduction) has little or no value in itself to the parasite, so there's a chance to have a mismatch of fitness objectives between the host and the parasite.