Only white asbestos (also known as Chrysotile) --- and the great number of people who have been exposed to large amounts of it and lived long healthy lives certainly makes one wonder; AFAIK it's only the other types, particularly blue asbestos, which are unanimously carcinogenic and the industry stopped using them a long time ago, but the effects from that (and contamination of white asbestos with the other types) have continued and led the majority of fears.

This article is worth reading: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3581056/