I turned a looped transformer into a decision model that scores answer options [pdf]

Hello! I just released wyat-2, a fine-tune of Ouro-1.4B (a looped transformer) trained on a single RTX 4070 Ti.

The link is the report. As far as I know it’s the first decision model built on a looped transformer: not the best, but the first.

I realized I would need heavy data work to make it the best, so I’d prefer not to spend too much time on it for now. Maybe later, if it catches interest.

Model: https://huggingface.co/wyat-io/wyat-2-1.4b