First, are you on Virgin ADSL? Or "Virgin National Broadband" as they brand it. If so you're going to get the same speed variability as any other ADSL provider. I'm talking about their cable service.

Anecdotally, any time I've measured I've got almost bang on what I'm paying for (at several speed rates and in three different houses over several years, though all in one general area).

More scientifically the Ofcom report about the poor performance of ADSL compared to advertised speed noted that it didn't really apply to cable based systems.

http://www.ofcom.org.uk/media/features/broadbandspeedsjy

They got a table there that shows on ADSL you're getting 50% of the 8MB they advertise on average, on Virgin Media you're getting 85% of the 10MB advertised speed.

Obviously if some folk are getting 100% then other's can be on 70% and it'll still average out but I'd say it's worth getting your line checked if you're only getting 20% of what you're paying for.