Comment by icyfox | Hacker News Reader

Comment by icyfox | Hacker News Reader

Parent Full thread

Neat paper. Planning on reading more in-depth over the weekend, but more fundamental than just applications to GPT their insights are:

- Existing pruners were written for models that are order-of-magnitudes smaller than any in the modern GPT family. They grow in linear time with the amount of input parameters so they're unequipped to work on current architectures. The best existing pruner performs takes 4.3h for a 1.3B model

- The core issue to scale is time to calculate the Hessian during prune analysis (effectively a matrix of second-order derivatives, famously computationally intense to calculate)

- They follow the existing literature and use a local approach to each layer. By doing this (and doing it well), it can preserve the input/output contract for surrounding layers, which makes the whole thing paralellizable across machines

- Their solution approximates reconstruction loss by approximating a quadratic loss and then running a OBS update (with a few other optimizations on ordering and iteration on the side)

I'm particularly excited for these smaller models, mostly for inference efficiency gains in realtime applications. The general con of weight pruning is they still require incredibly large training clusters / investment in training resources upfront to get the original parameter weight. But if the lottery ticket hypothesis holds true, this might be the best way we have at the moment to get models with same performance and lower longterm operational costs.

Another random thought: Most of these general purpose pruning approaches rely on randomly calculating the X vector for which they want to measure output loss of the layer. In theory it's possible to feed actual datasets into these models as well, which could be another way to get a sparse model that's more acutely optimized towards one task. The original model produces the X activations on each layer, and these are used as the optimization criteria for the pruned version.

It might be able to provide performance similar to fine-tuning but without the weight skew that you'll necessarily see in parameter values.

I believe you are correct, I worked on a summer research project at NYU in 2018 based on https://arxiv.org/abs/1805.12185

As part of that project I constructed an API that took a small dataset and a model, launched a K8s pod and ran something like this from the paper:

> The pruning defense works as follows: the defender exercises the DNN received from the attacker with clean inputs from the validation dataset, D_valid, and records the average activation of each neuron. The defender then iteratively prunes neurons from the DNN in increasing order of average activations and records the accuracy of the pruned network in each iteration. The defense terminates when the accuracy on the validation dataset drops below a pre-determined threshold. We note that pruning has been proposed in prior work for n

Obviously this wasn't on transformers but the idea is similar.

Statisticians have been using L1 regularization to estimate sparse models for a while, it seems reasonable to assume that you could fine tune the model on a data set while also pushing weak parameters to zero in a natural way in this domain as well.

> lottery ticket hypothesis

For those that, like me, didn't know the reference: https://arxiv.org/abs/1803.03635

This is a great breakdown. I don’t know much about LLM internals but I could follow this easily.

Reduction in working memory for sparse models seems pretty huge.

Reply on news.ycombinator.com