Unlike radiation, there's no safe asbestos exposure, if you're really unlucky a single strand can screw you up. On the other hand, of all people having worked at asbestos facilities with early 20th century approach to PPE only 20% developed mesothelioma and 10% died from it.

Thinking before swinging your drill will get most people safe enough not to worry about it.