I remember the saying that from 90% to 99% is a 10x increase in accuracy, but 99% to 99.999% is a 1000x increase in accuracy.
Even though it's a large10% increase first then only a 0.999% increase.
Even though it's a large10% increase first then only a 0.999% increase.
90% -> 1 error per 10
99% -> 1 error per 100
99.99% -> 1 error per 10,000
That can help to see the growth in accuracy, when the numbers start getting small (and why clocks are framed as 1 second lost per…).
I guess it's the same problem with the mind not intuitively grasping the concept of exponential growth and how fast it grows.
Humans struggle with understanding exponential growth due to a cognitive bias known as *Exponential Growth Bias (EGB)*—the tendency to underestimate how quickly quantities grow over time. Studies like Wagenaar & Timmers (1979) and Stango & Zinman (2009) show that even educated individuals often misjudge scenarios involving doubling, such as compound interest or viral spread. This is because our brains are wired to think linearly, not exponentially, a mismatch rooted in evolutionary pressures where linear approximations were sufficient for survival.
Further research by Tversky & Kahneman (1974) explains that people rely on mental shortcuts (heuristics) when dealing with complex concepts. These heuristics simplify thinking but often lead to systematic errors, especially with probabilistic or nonlinear processes. As a result, exponential trends—such as pandemics, technological growth, or financial compounding—often catch people by surprise, even when the math is straightforward.
From 90% to 99% is a 10x reduction in error rate, but 99% to 99.999% is a 1000x decrease in error rates.
Imo we got to the current state by harnessing GPUs for a 10-20x boost over CPUs. Well, and cloud parallelization, which is ?100x?
ASIC is probably another 10x.
But the training data may need to vastly expand, and that data isn't going to 10x. It's probably going to degrade.