An unsubstantiated accusation is not enough to tank a career.

A cross-correlated one, from multiple independent reports is another story. As is one that stands up to cross-examination and scrutiny.

Reporting harassment by your manager is just as likely, if not moreso to shit on your career, as it is on theirs. They have all the power in this relationship, and every opportunity to retaliate if you aren't believed.