In a lot of other rural places on Earth, even if they are in poverty, there is usually a decent amount of activity even if some buildings are abandoned. There are families (often very large ones with many kids), drunk hooligans, outgoing salt of the earth types.
Rural Japan in contrast is quiet, mostly elderly, and more culturally “refined” at least on its face. And if you go wandering around, even as a very conspicuous foreigner, because of the language barrier and general cultural introversion, the few people you do see will mostly politely ignore you or not initiate conversation. Similarly while buildings may be abandoned and overgrown with plants there is usually very little trash (not counting what was simply left there, think more like plastic bags and empty beer bottles from teens hanging out) and no graffiti.
Other places that may be obviously in decline have evidence of a future and continued habitation, even if bleak, from young people vandalizing abandoned places, families going about their day, and confused middle aged folks trying to ask why you are even visiting their place and if you’re lost. In rural Japan, especially if you don’t speak Japanese, it’s more like you are a ghost wandering around a place resigned to evaporate, whose remaining inhabitants stoically continue on their peaceful day to day.