There's also things which are toxic at the same levels as olfaction, and of course there toxins which are dangerous by the time you have smelled them, or you lack the gene to smell. Most really smelly things aren't acutely toxic like that though. Take DMS (dimethyl sulfide, byproduct of Swern oxidation. Smells like boiling cabbage at low concentration, and liquid ass at high concentration. If you are running a Swern in a fume hood, you are likely quite well-contained even if you've made the whole lab smell like a cesspit.
When we ran cyanation at the kilo scale, you better believe we had HCN detectors all over the place, wore full-face respirators, etc.