Hacker News Reader

Comment by carbocation | Hacker News Reader

I'm a bit confused as to why a Mixture of Experts (MOE) isn't one of the comparators. That seems like the most relevant direct comparator, rather than the several other paradigms that they cited.

View on HN

MoE is a type of neural network architecture, not an approach to lifelong learning. As they say " We propose a new Shared Knowledge Lifelong Learning (SKILL) challenge, which deploys a decentralized population of LL agents that each sequentially learn different tasks, with all agents operating independently and in parallel." What they propose is effectively an MoE with separately learned routing as far as I can tell, with the heads of the LLM being task specific and there being a task mapper to assign training labels. The baselines based on Parameter-Isolation methods are sort of similar MoEs, in that additional weights are added and trained on each tasks (somewhat as experts would).

There are network architectures for MoE, but afaik the concept of MoE is separate from them. MoE at least significantly predates the last decade's neural network boom.

afaik the term MoE is generally used in the context of neural networks, with the term mixture models being more general. But you are right it's not exclusive to NNs.

> What they propose is effectively an MoE with separately learned routing as far as I can tell, with the heads of the LLM being task specific and there being a task mapper to assign training labels.

What you are describing would indeed be an MoE model, but that's not where it ends. The manuscript continues [1]: "all agents become identical after all tasks have been learned and shared, and they all can master all tasks."

That is a substantial divergence from the MoE model!

To the extent that there is an MoE-like stage in their training, I think it's odd that the term "mixture of experts" is not mentioned nor is the literature on the topic cited.

1 = https://openreview.net/pdf?id=Jjl2c8kWUc

Good point - it's a loose analogy training wise, and not applicable to inference time.

Reply on news.ycombinator.com