Yeah, I don't really consider virginia part of the south, culturally. Maybe it was different in the past but proximity to DC has rotted any of that away.
There is another, semi-derisive, use in which it means any non-southerner. But that is less common and context-dependent.