In Trump's case, specifically, echoing the “Unite the Right” marchers call to respect history while making the “many sides” statement was pointed to as dog-whistle support for the alt-right groups. (The weak statement itself was also called out, but it wasn't the weakness or vagueness of the statement that was the basis of the “dogwhistle” characterization.)
Not sure what the specific alleged dog-whistles were in Damore's memo.