Contrastive Representation Learning
lilianweng.github.io
lilianweng.github.io
So bottom line, I think CL is a specific instance of finding a simple rule or pattern that we can use to label select features that work for many tasks, but that's pretty much all it is. I think it's good progress to be able to find some of these simple core rules about what ML models are really noticing.
There's already been great progress, but the better we're able to create meaningful latent spaces, the better we're going to get at CL (maybe that's a self-evident statement :P ).
For example, "entangled" neurons have been shown to be a necessity to some degree when applying an informational bottleneck, and that they just tend to encode features that don't overlap very much.
I'm currently of the mind that "encouraging" latent spaces to abide by certain expected distributions is often a nice way to do this (of course depending on the distribution). Historically, we've been limited to the rather vague and often inappropriate Gaussian (i.e. VAEs), but with adversarial VAEs and MMD, we're increasingly able to impose arbitrary distributions.
Now of course, I'd agree that too narrow a distribution is probably not the right direction. But doing subtle things like half-continuous, half-discrete codes, or modelling rare entities with Poisson distributed codes could be interesting.
In the past we have been looking for one representation/specification language that works for all. I think that's not the most efficient approach even if these modes of representations have uniform approximation property. "Can represent" does not translate to "shortest representation" or "learnable representation" or "easy to manipulate". These properties are very useful and cannot be ignored.
At the same time, it seems like one encounters a fundamental problem with going from a deep learning paradigm to a learning paradigm.
For regular deep learning, you gather enough data to allow you to massive, brute-force curve-fitting that reproduces the patterns within the data. But even with this, you encounter problem of finding bogus patterns as well as useful patterns in the data and also the problem of the data changing over time.
Now, in adding "learning to learning" approaches to deep learning, you are also do brute-force, curve-fitting to discover the transformation between data-pair or similar things that are involved in change, new-stuff arriving. But this too is dependent on the massive data-set, it might learn wrong-things and the kind of change involved might itself change. But that's a more fundamental problem for the learning-to-learn system, because these systems are the one that are expected to deal with new data.
I've heard one-shot/zero-shot learning still hasn't found many applications for these reasons. Maybe the answer is systems using truly massive dataset like Gpt-3.
Maybe it took millions of years to come up with this algorithm but it seems like the approach is more than just some long incremental thing.
Deer are the product of millions of years of evolution also. Deer never learn to look both ways before cross a highway, though they can learn a significant number of other things.
Won't necessarily, A data driven approach won't necessarily work.
I'm sure that given the proper learn and virtually any "program" or algorithm, there's version data-driven training that would be able install such a program., so in the abstract data-driven training would able to do "anything". But you'd need to know what to do.
But poster above said "Human learning is based on millions of different learning strategies that were encoded into the neurons of each of our ancestors". And it seems like many of those millions of years actually produced more narrow (if extremely powerful and impressive) behaviors while the generality of human intelligence is relatively more recent.
Of course, human beings "learned to learn" somehow. It's just that I think it's plausible that a specific set of circumstances were necessary. I would say the evolution of both language and complex social interactions were part of this.
Natural selection wields significant influence. Too much too soon and it may go extinct. If its just right, the intelligence to cross roads might give them a reproductive advantage.
The only thing that comes to mind is embedding that preserves distance, such as MDS (https://en.wikipedia.org/wiki/Multidimensional_scaling#Metri...)
That’s a label-free approach which should give you a space with nice properties for eg nearest-neighbor approaches, and there’s, it follows, some reason to believe then that it’d be a generally useful feature space for downstream problems.
Note that most sample pairings, especially for images, is done through augmentations currently, so the implicit labeling you're doing is still weak on priors.
Of the methods mentioned in the article, BYOL (and even more the follow-up SimSiam [1]), have the weakest assumptions and work surprisingly well despite their simplicity.
I say this, since there are also cases of constrastive sampling like ideas with truly unsupervised data. For example, Graph Embedding, where a graph implies structural features of similarity and distance that the representations should capture.