RWKV is a language model design that trains in parallel like a transformer but runs like a recurrent network, so its memory use stays constant however long the text gets.
In plain words
Transformers compare every token with every earlier token, so cost grows quickly with length. Older recurrent networks carry a fixed-size memory forward, which is cheap to run but hard to train in parallel and scale up. RWKV mixes the two using a form of linear attention and needs no standard attention at all. It has been scaled to 14 billion parameters and is now a Linux Foundation AI project.
A quick example
Imagine a chatbot running on a small device during a very long conversation. A transformer's memory needs would keep growing with every message, while an RWKV model keeps a fixed-size state, so each new reply costs about the same.
Where you'll meet it
- Efficient text generation on limited hardware.
- Research on attention-free alternatives to transformers.
Related
Sources
- PaperRWKV: Reinventing RNNs for the Transformer Era, Peng et al. (arXiv) · read 29 Sept 2026
- DocsRWKV-LM, BlinkDL (GitHub) · read 29 Sept 2026
- DocsRWKV Language Model, RWKV · read 29 Sept 2026
Checked by WhatIsAI against the sources above · 29 Sept 2026. Spotted an error? Report it.