In most cases, at least when considering the higher-profile ones of the last few weeks, stories have been about multiple occurrences. The reporting typically identifies a pattern of behavior from multiple sources who were confirmed to have told friends, family, or colleagues at the time of the incident.
If a single, unverifiable account was thought to be sufficient, I'd be with you. But when you have multiple occurrences and patterns, you can argue that it's clear that the person is creating a toxic work environment and can be dealt with more swiftly than the legal process.