Yes, there's a big difference between not fitting, vaguely saying "it's not a good fit" out of politenss and being unfit.

Yeah, there might be legal/monetary implications to it too, as I have taken this role as a consultant through my own company.