This is absolutely correct. Neither chanterelles nor puffballs are "foolproof" mushrooms, even slightly. Mistaken puffballs are probably the number one cause of serious mushroom poisonings and even fatalities, since small puffballs look so much like amanita mushrooms that haven't fully opened yet.

And chanterelles have a number of at least moderately toxic lookalikes depending on the region you're in and how much an eager amateur is willing to stretch the definition. I don't think a lot of beginners can reliably tell the difference between "gills" and "folds."

Plus what you referred to about toxins leached from the tree that the mushroom was growing on. I'd not eat anything growing on a locust tree or a fallen rhododendron. Just seems like inviting trouble.