| | --- |
| | library_name: RAT |
| | language: |
| | - en |
| | license: mit |
| | datasets: |
| | - HuggingFaceFW/fineweb-edu |
| | tags: |
| | - efficient architecture |
| | - recurrence |
| | - attention |
| | - pretraining |
| | metrics: |
| | - perplexity |
| | - accuracy |
| | --- |
| | |
| | ## Description |
| | Models trained from [RAT Paper](https://arxiv.org/abs/2507.04416). |
| |
|
| | ## Citation |
| | If you find it useful, please consider citing the paper: |
| | ``` |
| | @article{wei2025rat, |
| | title={RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modeling}, |
| | author={Wei, Xiuying and Yadav, Anunay and Pascanu, Razvan and Gulcehre, Caglar}, |
| | journal={arXiv preprint arXiv:2507.04416}, |
| | year={2025} |
| | } |
| | ``` |
