Seems counter-intuitive and definitely warrants research to understand the mechanism. The most unintuitive part being local plants not returning.
Likewise, I don't know if Japan has to deal with wildly aggressive invasive species like I am right now with buckthorn, but there's lots of plants that will simply choke out most others and aren't very beneficial. If nobody's around anymore and they were actively grooming the pastures and such, that will also put a ton of pressure on previously flourishing species.