Recent ARC-AGI systems have shown that pure vision can solve abstract grid reasoning without language backbones, but most strong models still rely on fixed-depth feed-forward computation. We study whether recurrent computation is a better scaling axis for this setting. We propose Loop-ViT, a vision-native recurrent model that reuses a weight-tied Hybrid Block across multiple steps, with local convolutions for image-token refinement and attention for global rule induction. We also use an entropy-based exit rule to reduce unnecessary iterations at inference time. On ARC-AGI-1, Loop-ViT improves the parameter-efficiency trade-off over fixed-depth vision baselines and reaches 65.8% accuracy with an 18M model. Overall, the results suggest that repeated computation is a useful inductive bias for ARC-style visual reasoning. The code is available at https://github.com/WenjieShu/LoopViT.

Transformers have revolutionized almost all natural language processing (NLP) tasks but suffer from memory and computational complexity that scales quadratically with sequence length. In contrast, recurrent neural networks (RNNs) exhibit linear scaling in memory and computational requirements but struggle to match the same performance as Transformers due to limitations in parallelization and scalability. We propose a novel model architecture, Receptance Weighted Key Value (RWKV), that combines the efficient parallelizable training of transformers with the efficient inference of RNNs. Our approach leverages a linear attention mechanism and allows us to formulate the model as either a Transformer or an RNN, thus parallelizing computations during training and maintains constant computational and memory complexity during inference. We scale our models as large as 14 billion parameters, by far the largest dense RNN ever trained, and find RWKV performs on par with similarly sized Transformers, suggesting future work can leverage this architecture to create more efficient models. This work presents a significant step towards reconciling trade-offs between computational efficiency and model performance in sequence processing tasks.