Well, I would argue that in most so called extreme sports the mitigatable (is that a word?) risks play relatively much larger role than in traffic. You can't mitigate a drunken driver colliding with you, but it is precisely you who makes the knots in the climbing rope and packs your parachute correctly.
And, given so small probabilities, even if you tell yourself that you are mitigating the risk by not drinking&driving, it is virtually impossible for you to know that you do not have some other habit that increases your death probability to the average levels or above. Are your reaction times slower (or faster) than other peoples? Do you hitch a hike always in a car that is properly taken care of? Etc.
And if you do not own a car, your lifetime cycling quota in the US cycling environment is less than 2 000 miles[0]. Even less mitigating other peoples' mistakes than in a car.
[0] http://www.mrmoneymustache.com/2013/06/13/bicycling-the-safe...