My understanding is that it's only bad if disturbed and in the air. That's why popcorn ceilings, roof tiles, and insulation with asbestos is all "fine" and not mandated to be removed.

But yes, the long-term effects are the real problem.