Comment by hyperthesis | Hacker News Reader

Comment by hyperthesis | Hacker News Reader

Just adding an obvious corollary regarding parallelization, to "leverage computation" and "massive computation became available and a means was found to put it to good use":

  Parallelizable methods win (e.g. DL), by using the computation available

Further: progress will occur on real-world problems that can be solved by those methods, making those applications dominant throughout society. (i.e. everything looks like a nail to someone with a hammer... but if it's a really good hammer, that may be the best approach)

Yes. Prior to this though in the AI community there was a lot of debate about the pros and cons of "weak search" (ie generalized methods) and "strong search" (ie methods which make use of domain knowledge). The bitter lesson is more or less that in the long run weak search always wins and the only real judgement call is whether strong search can give you a transient benefit in a particular case before weak search surpasses it and whether that is worth it.

In my head this always parallelled the "premature optimization" conversation in programming. Most programmers would say that inlining etc was only justified when you've benchmarked so you know how the benchmarks are performing but the experience of whole program optimization in things like hotspot jvm and llvm seems to suggest that even benchmarked optimizations can be premature because only a vm can do optimizations on the real world use case.

Reply on news.ycombinator.com