m ← αm + (1-α)g // momentum
c ← βm + (1-β)g // new part
θ ← θ + η sign(c) // update of parameters
The parameter update is via the sign of the weighted average of the momentum and the gradient.
For comparison, the gradient of abs(x) is sign(x), which is the update with L1-norm.
To be fair, I don't think that this operation will fundamentally change the behavior of the algorithm. In fact, I don't think their algorithm has a fundamentally different behavior than SignSGD with momentum, whose convergence behavior is well-understood [1, 2, 3]. Actually, the algorithm you described is equivalent to SignSGD with momentum. In the algorithm you described, m is a convex combination of m and g (with coefficient alpha), then c is a convex combination of updated m and g (with coefficient beta). This is exactly the same as setting c as a convex combination of m and g with coefficient alpha*beta. And that is exactly SignSGD with momentum. The only difference between the algorithm you described (equivalent to SignSGD) and LION is that they further modify the value of the momentum after computing the update with an additional convex combination operation. Again, I am skeptical that this operation will fundamentally change the behavior of the algorithm. But that's just my guess.
[1] https://arxiv.org/abs/1802.04434
I suspect it is equivalent up to slightly different α and β?
c ← 0.9m + (0.1)g
There doesn't seem to be a significant difference between their code:
θ ← θ + η sign(0.9m + (0.1)g)
and just not having the 2nd line at all:
θ ← θ + η sign(m)