424 karma · joined May 16, 2017
It seems like they use a fixed-distribution controller for training. It’d be nice to see why it’s worth deviating from the original RL paradigm.
Since you mentioned different layers, one question to ask is if your poetry example is “just a combination of existing rare patterns,” one that’d be addressed by a bigger model with more data.
And as you say the integration of new information does not seem to rely on any fundamentally new principles, just more engineering work.
I think then the question to ask would be, well, how much more data to x% more faithful of the rules of Othello?
This rate of learning matters, and can be the difference between “in our lifetime” and “never in hundreds of years”. As ML practitioners our jobs are often to improve these rates through better formulation of the learning problem.
I think there’s a connection here that amounts to “how quickly you learn”, and it has implications on causality: https://vladfeinberg.com/2019/12/01/metaphysics-of-causality...
Your reference, "The Nature and Meaning of Perturbations in Geometric Computing" itself refers to "On degeneracy in geometric computations" (https://dl.acm.org/doi/10.5555/314464.314474) which I think points to a less tame reality where the answer to my question (continuity claims about CG outputs) is "it depends on the CG problem".
Say you have a spiral-shaped but simple polygon that's wrapped around a query point. Even if your query point is far from the innermost "loop" of the spiral, a raycast could be arbitrarily confused if the sprial has many "thin" loops on the outside. (However, notably, the bits precision required would be a reasonable function of the spiral's specification. My point is just that fuzziness in general can be a compounding problem as a CG algorithm "progresses".)
In numerical linear algebra, problems have an intrinsic condition number, and most algorithms admit backward or forward stability in terms of that condition number, implying a kind of Lipschitz continuity in computation: you can't be too far off from the true exact solution, and if your input was rounded, then it's still close enough to the exact solution.
The documentation seems to imply that CG problems don't have this kind of regularity ("no such arsenal"), hence the need for fallback to exact.
But if this regularity doesn't exist in CG (i.e., you can create natural geometric constructs for which low-bit perturbations result in large-scale changes in the output), then why would exact numerics even save us here? Wouldn't it be possible to construct examples where we could make the amount of precision required super-polynomial in the input bitwidth? Or are there different regularity theorems which limit the complexity of the output (but still require, say, polynomially-increased precision).
https://mathoverflow.net/questions/12511/what-is-yonedas-lem...
I guess in retrospect we asked it to continue the music in a likely way, not be novel. And it definitely convinced me enough to be impressive. An NN that composes completely fresh music, whatever that means (I’m sure most modern human music has a hefty dose of cross song sampling), would certainly be a good next goal post.
It didn’t get to train on the test set, if that’s what you’re implying, and I find it hard to believe the assertion that continuations are copies of the train set (if that’s your claim).
Maybe that’s convex if you smooth enough. But depends on the buffer sizes.
Even within this vertical it seems you’d need a lot more than 15 engineers.
1. Creating a few data points (3) for scaling laws (Figure 8). These behave similar to language models, as gwern puts it [1], but across three data points, it's a bit tough to draw a power-law conclusion (eyeballing the figure, they increase params 4.5x and 3.2x and see about 20% relative performance improvement from each jump).
2. What I find more interesting than the scaling is the out-of-distribution (OOD) generalization results (Figure 9). They test the performance of the agent on a completely unseen task (though possibly from within the same domain, i.e., they might train on a fixed physics engine from the DeepMind Control Suite [2], but never let the agent look at the cartpole task). They compare this to various ablations: from-scratch training with the same architecture, pretraining only with same-domain data, and pretraining only on non-control data (presumably unsupervised contrastive-learning based data).
The results from (1) are impressive and from (2) are mixed (but no less interesting as a contribution!) in terms of the additional training data actually helping with generalization. The reason OOD generalization performance is most interesting is because it really tests whether control-based pretraining helps the agent in a truly new situation. And certainly, there are a couple tasks at which the zero-shot performance improves over the ablations (but there are others where it hurts).
What I'd find exciting to see in coming research is further investigation into variants of Figure 9.
- How does scaling affect the impact of control-data pretraining vs non-control data pretraining?
- The authors used a custom fine-tuning schedule for the few-shot evaluation on unseen tasks. It's possible the schedule needs to be changed for the ablated versions of the agents to give them the best performance, too. What would Figure 9 look like with the "best" training setup for each ablation individually? I.e., can we tease apart how much, if at all, it's a matter of low-level modality-specific features helping zero-shot adaptation vs some kind of truly generalized "control pretraining"?
[1] https://news.ycombinator.com/item?id=31356155 [2] https://arxiv.org/abs/1801.00690
Select the deterministic subset by just hashing some identifier for each weight.
For any reasonable number of copies, there's a pretty unique subset between all your copies sharing a large amount of bits flipped in the same direction among this subset.
> we have infinite sampling
We don't. You're taking the learning out of the process you're discussing by using an infinite sample. Of course, then, the machine learning part becomes trivial, and it's just an optimization problem for minimal training error!
If we have all the data, like infinite data, then we have the classification for every one of the (finite) number of possible images already. At that point, why stop at kNN? Use a hashmap!
Using Euclidean norm over pixels on imagenet will not get you anywhere.
By recasting classification into a kNN lens you're basically (in the large data regime) recasting the problem to kernel learning under some underspecified RBF kernel with an unknown "proximity" metric instead of Euclidean.
In grad school, someone in my lab actually tried scaling kernel methods directly to imagenet too. I don't think that ended up working naively (an interesting neural variant worked for CIFAR10, though [1]).
If all NNs are doing is efficiently learning an appropriate kNN metric (or, more idiomatically and generally put, learning kernel parameters for some implicit neural kernel), that's still really powerful and all we've done is just renamed "learning representations."
But if you needed to do this on a GPU or TPU, ideally with native and transparent SIMT, such as the case for the SO question inspiring the post (an unsupervised centroid-based loss for a deep learning setting), would you have to write a custom C++ kernel to do this?
Free SIMT may even make this worthwhile in the few-centroid setting.