This offhand reason does not happen regularly. There are cases of places without historical slavery who perceived as poor workers and places with historical slavery who are known for their work ethics.

Maybe it's not only "the legacy of slavery".