Vera: Vector-Based Random Matrix Adaptation

dkopi.github.io

1 pointegnehots3 comments

VeRA makes LoRA ~10x more parameter efficient while retaining the same performance.

It's somewhat like a recursive LoRA scheme, where the LoRA A and B matrices are also decomposed using two small trainable vector parameters.

Is the code available anywhere

Highly skeptical paper 1) Authors havent released code 2) They claim its published in ICLR , but i wasnt able to find the paper in openreview https://openreview.net/group?id=ICLR.cc/2024/Conference#tab-... 3) GPT-4 to evaluate performance. It seems to be grading the responses rather arbitrarily and not really considering how well the adapted model has learned the particular behavior it was being tuned for.

This just screams 0% code 0% peer review 100% Hocus Pocus.

Reply on news.ycombinator.com