I've been a driver for twenty years and I have never felt like I cared about 'handling'. When car reviewers talk about it I always picture a 45 year old man-child white-knuckling the wheel at the local intersection while revving his 325i, eyes darting between soccer moms and pensioners in the cars around him and imagining himself to be Ayrton Senna. Though that is probably uncharitable :)
I have owned many different cars between a top-end BMW and a beater Hyundai. The BMW is certainly a more enjoyable ride, but 'handling' was never a concern, except perhaps in a very occasional circumstance that might have gotten me arrested.