Or maybe it was "You shouldn't have accepted gifts from contractors", my point is only that you are guessing with no basis in underlying fact.
He did preface with "My guess is". A guess is just that - a guess, without any info to back up your reasoning.