Yeah it'd be very slight, but things like stress and nutrition can affect your memory in the long term which is a part of intelligence.
We should perhaps recognize that and try to compensate for it, and it's not a value judgement on the person so afflicted, but pretending it doesn't exist just confuses matters.