Why would he become so startled(and obsessed) by a house he never visited for 8 years? To the point that a marriage had to collapse, nevertheless.
My guess is that Neville probably had schizophrenia, and thats unique thing about this story; I've heard a lot of stories of closed rural communities treating newcomer as pests.
As a example from my life, as a teenager I went to help a family-friend that had newly switched to agricultural lifestyle in Japan, over the summer holiday. I remember arriving at the rice field for the first time, noticing that the field was dried up and there was cracks in the soil. I asked the friend if this is how it is supposed to be, and he replied "no, probably some of the old farmers clogged the water channel". I asked why they would do that, and he said that the old farmers here didn't like newcomers, and would in the night go out and sabotage his farm.
I took a look around the farmland, and noticed a rotting hut, rusting tools, and how most of the rice field was now filled with bushes and weed and in no condition of agriculture. I again asked why, and he told me that because the old people can't take care of the land, and because they bully away any newcomer, most of the farmland have been taken back by the nature, pointing to a forest telling me that it used to be farmland. It baffled me how these old people would rather see their entire village, way of life die out than letting any new person in.
Someone that just bought a house and left it unattended 8 years in such community, is to me not surprising at all that someone there took an ire. Maybe Neville just didn't catch that in his documentary about the island, and couldn't process the deeds done to him