Although that sounds like fun (for certain values of fun), the issue would exposure during fabrication and possibly during destruction of nano tube containing equipment.
We have to rely on them handling hazardous substances properly anyway since semiconductor manufacturing already involves many of them. They use ClF₃ and sulfuric acid as cleaning agents, phosphine for doping, silane for CVD, etc.
Industrial scale manufacturing of just about anything probably involves quite a few carcinogens.
I'm afraid that there are already a great number of unfriendly chemicals involved in the manufacture of regular old silicon ICs.