Even so, what's the worst-case scenario in biasing towards lower speed and assisting your vision system with a knowledge base?

You perform the task of vision 100% of the time driving. From a purely probabilistic standpoint, even assuming highly accurate ML, the relatively infrequent database updates still make for a sensible prior.

This is somewhat adjacent to what Gary Marcus has been arguing, and I think it makes a lot of sense; there seems to be no compelling reason to rely exclusively on primitives (i.e. vision) when good priors are easily accessible.

65 in a 55 is much less dangerous than 85 in a 35. It wouldn’t be common that road conditions would permit very high speeds while at the same time having an artificially lowered speed limit (e.g construction).

This shouldn’t be a problem, in general, with a couple extra sanity checks.

I would expect the correct solution to more commonly have the opposite problem: too slow in a faster zone.