There are a lot of compounds that are poisonous to most species, but where some species have adapted defences against them, rendering them harmless. (IIRC capsaicin is like this in humans.)

There are also compounds that are toxic to basically every animal species for low-level common-animal-ancestor reasons, but which exist in the plants/fungi they exist in, in concentrations dictated by what’s required to kill/ward off the particular predators that plant/fungus had in its evolutionary environment. Aspirin, caffeine, nicotine, cocaine — these are all effective pesticides. They kill or severely harm insects. But we’re orders of magnitude larger than insects, so they have orders-of-magnitude less effect on us (unless we make orders-of-magnitude more concentrated doses of them, of course). That change in quantity, is reflected by a change in quality — i.e. having “a little bit” of caffeine does very different things to a body than having a lethal dose of caffeine.