PlantNet often works better for trees.
The training imagery is also really inconsistent in inaturalist and again for plants it's hard to establish context. These are mostly smartphone pictures that non experts have captured in the app. While someone might have verified that a plant is a particular species, because there isn't any foreground/background segmentation, there are often confounding objects in the frame. On top of that you only get a 300px input to classify from. With animals I'd say it's much more common for photographers to isolate the subject. There's also massive class imbalance in inat, a large number of the observations are things like mallards (ie common species in city parks).
I guess the best solution would be to heavily incorporate geographic priors and expected species in the location (which I think is partly done already).