Why Momentum Works
distill.pub
distill.pub
Y[i+1] = B * Y[i] + (1-B) * X[i+1]
Note that if the sequence X becomes a constant after some point, the sequence Y will converge to that constant (as long as B != 1).
For giving the gradient short term memory, the article's approach is of the form:
Y[i+1] = B * Y[i] + X[i+1]
Short term memory doesn't really seem to describe what this is doing. There is a memory effect in there, but there is also a multiplier effect when in regions where the input is not changing. So I'm curious how much of the improvement is from the memory effect, and how much from the multiplier effect? Does the more usual approach (the B and 1-B weighting as opposed to a B and 1 weighting) also help with gradient descent?
[EDIT] As per halfling's comment, this is just a change of the learning rate by (1-beta)
http://math.mit.edu/~freer/papers/PhysRevLett_110-168702.pdf
How Life (and Death) Spring From Disorder: https://www.wired.com/2017/02/life-death-spring-disorder/
Original article : https://www.edge.org/response-detail/27023
Until Xi, not a single variable is defined prior to inclusion into an expression. Even Alpha and Beta are only defined in the header diagram rather than the body of the text. Also, why are the iteration notations in the superscript rather than subscript?
And before someone chimes in and says that rudiments are necessary to understand this work, no they aren't. The logical steps here are exceptionally simple (and intuitive - as the introduction might lead you to believe) once you get past the delivery. This is a fantastic article that could become very accessible with the proper notation housekeeping.
but if you've studied linear systems a bit, the math is laid out really well for understanding and turning into practical applications (which seems to be the whole point of distill.pub).
many math articles, in contrast, seem to obfuscate the "why" and the "what for" and concentrate on the theory and derivation (e.g., many wikipedia math articles). that might be great for mathematical elegance (or cynically, oneupsmanship), but not so great for people trying to build things using that valuable knowledge.
And how would it perform for non-positive-definite systems?
Just to be clear, though, CG doesn't use the negative gradients as search directions, as steepest descent would.
We've been reading through the policies of lots of journals that seem thoughtfully run and borrowing good ideas. :P
Edit: I'm referring to the journal, not the author.
I can see this type of interactive journal becoming very popular in other fields, but not if the author has to create the diagrams him/herself.
I think these visualizations are deceptively easy to create. Javascript is a powerful language with many libraries, and in my experience, it just took a few nudges at exactly the right spots from Shan to go from an idea in my head to fully fledged diagram in distill. The tricky part has always been figuring out what to visualize, and if you're a researcher with an clever idea for a visualization, I recommend you reach out to the distill team.
https://github.com/distillpub/post--momentum/issues/1
We've also had a few designers volunteer to work with researchers on visualization. So, in special cases, we may match-make researchers with designers to produce a great article.
- When hovering over some notes I see the citations perfectly, but when there is math involved like for example the one about spectral decay the math is not rendered (I'm using Ubuntu, I have tested both in Firefox and Chrome).
- I see you can open issues on github to send corrections. I wonder if there are other channels of communication also to, for example, address problems like the previous one and also to discuss the article.
- I see people like a lot the figures. I think too they are great but what I really love is the writing and the math exposition.
- I think that a default animation for the figures, if they are meant to be manipulated, would be great.
I will say it again: great article. I have bookmarked the journal and I will proceed to read everything published which looks very promising too.
Would it be on-topic? (After changing style accordingly.)
In brief, Distill needs to see three things to publish an article: outstanding communication, advancing the dialogue, and scientific integrity. Distill often works with authors to help them bring their articles up to our standards.
Additionally, as a primary publication, Distill will not republish content already published elsewhere, or publish "translations" of papers where someone rewrites the content of a previous paper. (This relates to advancing the dialogue.)
If you reach out to editors@distill.pub, we're happy to discuss pre-submission inquiries about journal scope and related topics.
(For some reason I though that this t-SNE article was published elsewhere. Now I see that it was on Distill, but just before its big start.)
(Distill needs to be extra careful about a lot of this stuff because we're trying to build legitimacy for a kind of work that many people are inclined to not treat as academic contributions. So on some things, like being a primary publication, we may end up taking a more defensive posture than we would in an ideal world.)
I totally understand that you need to be set high reference level at the very beginning, even if at the cost of being a bit "conservative" (I am not sure if the best word here).
Overall -- huge fan!
</rant>
Some points not touched on in the article. If the individual dimensions are of different scales, this problem can be easily fixed with a diagonal preconditioner. Even something like ADAM or Adagrad (unconventional, I know, in this domain) can be used.
There's also a small industry around more sophisticated preconditioners for the linear systems in PDEs, see Multigrid, for example, or preconditioned conjugate gradient.
This could be worth looking into for you.
With discourse, I think there will be more noise and a lot of time will be wasted scrolling through unnecessary replies. What's good about the thread-like nature of HN/Reddit is that you'll have proper context and the rating system does its job so everyone's time won't be wasted.
Questions can be answered here and on Reddit too. I think Reddit can be sometimes more helpful than HN when it comes to answering 'easy' questions so I would go there if you have any simple questions.
[1]: https://www.reddit.com/r/MachineLearning/comments/63f3uk/r_w...
These articles on distill I believe could greatly benefit from this. Let the community help distill.
And if there is a question that requires more control, like math formatting, etc. I would actually suggest posting to cross validated[1] and then linking it here.
momentum = mass * velocity
force = mass * acceleration
momentum = (force / acceleration) * velocity
If you want to make your analogy work, the momentum algorithm adds mass to an object. In terms of literal acceleration at any point in time it is neutral, but the added mass causes you to get through difficult areas much faster.
That said, much of this article is moving away from that physical analogy. :)
The added inertia smooths out variation in velocity, dampening oscillations and causing us to barrel through narrow valleys, small humps and local minima. Keeping our speed steadier, we arrive at the global optimum faster.
Note the alliteration :-)
> momentum accelerates convergence
Here it's more clear that momentum is accelerating convergence, not the "heavy ball" itself.
> inertia acts both as a smoother and an accelerator
On the second read, it's more clearly a contradiction. Speed can't be both held steadier and accelerated simultaneously. If you meant that momentum alternately smooths and accelerates, then it's even more strange. For that behavior, some sort of motor would be a more appropriate metaphor.
velocity = 1/2*acceleration*time^2
momentum = 1/2*force*time^2 = 1/2*mass*acceleration*time^2
There's two ways to change momentum -- velocity or mass. If mass increases, acceleration decreases. If velocity increases, acceleration remains the same.
Edit: You've got a bit of an endogeneity problem in your equation. And I guess I do, too.
momentum = 1/2 * mass * acceleration * time ^ 2
So... I think "momentum" here is leading us a bit in the wrong direction. What the author should have been saying is "mass". After all, the metaphor was "a heavy ball", emphasizing the weight. The initial velocity was unchanged across the comparison.
It bugs me that so many domains misuse the term "momentum". Business people confuse momentum for velocity. Now machine learning folks are confusing momentum for mass. It's common for people to use a component of the whole as a stand-in term, like "my wheels" instead of "my car". Less common to go the other way: "my arm hurts" instead of "my elbow hurts". In this machine learning case the imprecision of the metaphor harms understanding.
In general, this is a nicely written article, though. Good work!
under-relax, converge real slowly
over-over-relax, oscillate
just-right-over-relax, get fast convergence