The WHO says "all forms of asbestos are carcinogenic to humans" [1]. The CDC specifically addresses "naturally occurring asbestos," which it finds "is only a health problem if it is disturbed," e.g. via "surface soil disturbance" [2].
[1] http://www.who.int/mediacentre/factsheets/fs343/en/
[2] https://www.atsdr.cdc.gov/noa/where_is_asbestos_found.html