Most milkweed is poisonous to humans, dogs, and livestock. Not saying don't plant it, but just be aware if you have small kids around.
The non-native daffodils that everyone and their dog plants are extremely toxic to dogs and cats and also humans. Same goes for many plant species, especially those that are commonly planted. The, again non-native, yew bushes that are planted everywhere are also extremely toxic.
It isn’t an excuse but people will hide behind it (not saying you do). If we wanted to eliminate all toxic plants, we’d literally have nothing and certainly no agriculture.
I have children now. I’ve never seen them taste arbitrary, random plants.