Hacker News Reader

Comment by plaidfuji | Hacker News Reader

This all makes sense, but at the same time feels a bit paradoxical to me. We’re developing a first-principles theory to understand the mechanics of massive empirical models. Isn’t that kind of ironic?

If the problem we’re solving is “faster iteration and design of optimal deep learning models for any particular problem”, I would have thought the ML-style approach would be to solve that.. through massive empiricism. Develop a generalized dataset simulator, a language capable of describing the design space of all deep learning models, and build a mapping between dataset characteristics <> optimal model architecture. Maybe that’s the goal and I haven’t dug deep enough. Just feels funny that all of our raw computing power has led us back to the need for more concise theory. Maybe that’s a fundamental law of some kind.

View on HN

I think this is to be expected. The oscillation between first-principles and empirical models falls out of the scientific method: see a few datapoints, develop a predictive theory, try to prove the theory wrong with new datapoints, and reiterate for alternative explanations with fewer assumptions, greater predictive power, etc...

This happens even in pure mathematics, just at a more abstract level: start with conjectures seen on finite examples, prove some limited infinite cases, eventually prove or disprove the conjecture entirely.

Current DL models are so huge they've outpaced the scale where our existing first-principles tools (like linear algebra) can efficiently predict the phenomena we see when we use them. The space has gotten larger, but human brains haven't, so if we still want humans to be productive, we need to develop a more efficient theory. Empirical models explaining empirical models might work, but not for humans.

We had a manager with Theoretical Physics education from a top school who seriously suggested to ultimately solve QA by building a system to run the program through all the possible combinations of branches/etc.

QA?

For some types of system, searching through all possible rules for the behaviour that you want isn't out of the question, and can sometimes lead to finding a solution faster than wandering around in parameter-space (either along a directed path or randomly).

A recommendation of this 'search-based' approach (where it can be used) instead of always constructing solutions is one of the 'new' aspects of Wolfram's "New Kind of Science".

I believe that this ability to tolerate searching for a solution is a component of what can allow neural-nets to find solutions better than programmers would ever write. A NN will happily explore nonsensical or illogical bits of parameter-space where there are still some good solutions to be found in places where the 'rational' or 'logical' human mind wouldn't (or even couldn't?) go.

Is your point that, taking it literally, it's infeasible, or that the suggestion amounted to fuzzing, which goes back decades?

Sometimes you can!

Checkout: https://randomascii.wordpress.com/2014/01/27/theres-only-fou...

From it:

Floating-point math is hard, but testing these functions is trivial, and fast. Just do it.

The functions ceil, floor, and round are particularly easy to test because there are presumed-good CRT (C RunTime) functions that you can check them against. And, you can test every float bit-pattern (all four billion!) in about ninety seconds. It’s actually very easy. Just iterate through all four-billion (technically 2^32) bit patterns, call your test function, call your reference function, and make sure the results match. Properly comparing NaN and zero results takes a bit of care but it’s still not too bad.

Sufficient understanding will always trump guesswork when the understanding exists (kinda tautologically). Hence the push to find it.

Reply on news.ycombinator.com