Hacker News Reader

The stats shown are pretty compelling that this transformer + RNN approach works. Looking back at the Llama paper (https://arxiv.org/pdf/2302.13971.pdf), many of the results are comparable between llama and RWKV. E.g. Llama 13bn scores 80.1 on PIQA and RWKV scores 77.5

I wonder if RLHF would boost performance for this architecture in the same way

View on HN

I personally think the way these chat assistants are created using these stateless models is extremely flawed. I'm glad that the technique used in RWKV seems successful, I wonder if this transfomer + RNN combination will be adopted elsewhere.

Transformers scale in training mainly because they are stateless!

They have a long lookback window (30,000 tokens in GPT4!) so the issues are smoothed over.

LSTMs end up being very finnicky to train and their sequential nature makes model scaling bottlenecked in parallelism.

RWKV training can be parallelized just like transformers. It's not LSTM.

Right, I meant why transformers exploded over LSTM-based LMs in 2018 was because of those training issues that transformers solved.

I'm excited about the next iteration of models focusing on inference. RWVK seems promising on this end. Also love the citizen science aspect of RWVK coming out.

Reply on news.ycombinator.com