Remove the material with mold from the space. Duct work is a terrible idea in humid climates. It's better to get those individual wall AC units per room. Mold is nasty stuff that will cause long term health problems.
Regardless of what you have periodic maintenance is what makes a difference.
Ductless mini split units.