Easy solution is to set a momentum (or kinetic energy?) limit, rather than a speed limit.
Consider getting hit in the head with a fastball pitched by a major league baseball pitcher. A baseball weighs ~0.145 kg, and a fastball might be going 45 m/s.
That gives a momentum of 6.5 kg m/s, and a kinetic energy of 146.8 J, and a serious possibly life threatening injury.
Now consider getting hit by a fully loaded B-52 (220000 kg) being towed at 0.5 m/s.
That has a momentum of 110000 kg m/s, and a kinetic energy of 27500 J. That's ~17000 times the momentum of the baseball and ~190 times the kinetic energy, but all it will do is knock you over. (You might subsequently get run over by it, which would do a lot damage, but that injury will be from neither the momentum nor the kinetic energy).
Kinetic energy is not a good measure for a car or B-52, as after the collision the large object has essentially the same kinetic energy, and a very small fraction has been transferred to you.
A bicycle is more toward the small object side, as a pedestrian is going to absorb a good fraction of the kinetic energy of the rider, which is much larger than the kinetic energy of the bicycle itself.
NB: I didn't do the math, so this is probably mildly hyperbolic, but I think it illustrates the point.