It might be able to provide performance similar to fine-tuning but without the weight skew that you'll necessarily see in parameter values.
It might be able to provide performance similar to fine-tuning but without the weight skew that you'll necessarily see in parameter values.
As part of that project I constructed an API that took a small dataset and a model, launched a K8s pod and ran something like this from the paper:
> The pruning defense works as follows: the defender exercises the DNN received from the attacker with clean inputs from the validation dataset, D_valid, and records the average activation of each neuron. The defender then iteratively prunes neurons from the DNN in increasing order of average activations and records the accuracy of the pruned network in each iteration. The defense terminates when the accuracy on the validation dataset drops below a pre-determined threshold. We note that pruning has been proposed in prior work for n
Obviously this wasn't on transformers but the idea is similar.