Are these the traitors you're talking about?
Even if people couldn't dig up anything against the person, the monument could simply be linked to the "wrong part of our history" and be removed, like the Thomas Fallon's statue in San Jose [4]
[1] https://en.wikipedia.org/wiki/Bust_of_Ulysses_S._Grant_(San_...
[2] https://en.wikipedia.org/wiki/Statue_of_Francis_Scott_Key_(S...
[3] https://en.wikipedia.org/wiki/Statue_of_Junípero_Serra_(San_...