> Where the hell were you guys the past three years?

Maybe they were there when you weren't there and they're asking the same question about you? Maybe your aunt never really thought to bring up that so-and-so had come to visit while you were there, so then as far as you know, nobody came to visit? Maybe they called, sure it's not the same as visiting but it's something.

It's true though, as humans we tend to not think about things and other people until they are gone. Don't be angry and upset, if anything, pity them for not having the time that you had.