Each year the chance of someone dying in a vehicular accident is about 1 in 10,000 in the US. Over 80 years that means a citizen has roughly a 1 / 125 chance of dying in a vehicular accident. The total reduction in life expectancy from vehicular accidents is thus well under 1.0 years. Assume, for example that such an accident cuts an average of 60 years off a person's life -- they would have lived to 80, say, but their life was cut short at just 20 years old. That works out as a 0.5 year reduction in life expectancy due to accidents.
This is without taking into account the vehicular fatality rates in other countries. When you do that for, e.g., Japan (1 in 20,000 per year), the difference in life expectancy is likely around 0.25 years.
The homicide rate in the US is about 1 in 20,000 people per year, or about half the vehicular fatality rate, so perhaps a further 0.25 year reduction, without factoring in homicides in other countries.
For infant mortality, the chance of dying at birth is a little over 1 in 200 in the US. That translates to a reduction in life expectancy of about 0.4 years, without factoring in infant mortality in other countries.
A similar analysis shows suicide rates are probably not a factor -- in fact, in some countries, such as Japan, the rate is higher than in the US, while in countries such as the UK it is comparable to the US.
So the contributions from all four of these sources is very unlikely to be more than a 1 year difference in life expectancy. It's probably quite a bit less, since I've ignored (e.g.) infant mortality in other countries. Certainly, the contribution from all four of these is nowhere near the dominant effect to explain differences in life expectancy.
A caveat: some people pointing to infant mortality are claiming that what's going on is that a large number of stillbirths simply aren't being reported in some countries outside the US, and this is skewing the results. That possibility isn't accounted for in my analysis above. I doubt it's true - in my many years living in Canada and Australia I've known quite a few people who died in car accidents, and none of whom had a stillbirth (to my knowledge). But while this suggests to me that this effect isn't large, it's of course only anecdotal evidence.
Sources for rates:
http://en.wikipedia.org/wiki/List_of_countries_by_intentiona...
http://en.wikipedia.org/wiki/List_of_countries_by_traffic-re...
http://en.wikipedia.org/wiki/List_of_countries_by_infant_mor...
And if you want to look at the details of the suicide statistics:
http://en.wikipedia.org/wiki/List_of_countries_by_suicide_ra...