Tiny Recursive Models — arXiv Paper
Co-authored study of TRMs on ARC-AGI-1. 3 citations.
Co-authored "Tiny Recursive Models on ARC-AGI-1: Inductive Biases, Identity Conditioning, and Test-Time Compute", analyzing the behavior of Tiny Recursive Models (TRMs) on the ARC-AGI-1 benchmark. Performed empirical ablations and efficiency analyses to isolate the impact of test-time compute, puzzle-identity conditioning, and recursion depth on model performance. Benchmarked TRMs against a QLoRA-fine-tuned LLaMA 3 8B baseline.
Overview
This paper presents an analysis of Tiny Recursive Models (TRMs) and their performance on the ARC-AGI-1 benchmark, a challenging dataset designed to test abstract reasoning. The study investigates how architectural choices and training strategies impact performance, with a focus on test-time compute, identity conditioning, and recursion depth.
Key findings
Test-time compute
Increasing test-time compute significantly improves performance on ARC-AGI-1, suggesting recursive models benefit from additional computation during inference.
Identity conditioning
Puzzle-identity conditioning is a crucial factor: accuracy dropped from 40.00% to 0.00% when puzzle IDs were blanked or randomized under the verification protocol.
Recursion depth
Effective recursion is shallow: step 1 already reached 38.25% Pass@1 (94.4% of final), with accuracy saturating by step 4 and unchanged at step 6.
My contributions
- Built reproducible data pipelines for ARC-AGI-1 experiments, processing and evaluating 400+ tasks at scale.
- Engineered a 1,000-sample data augmentation pipeline, improving Pass@1 accuracy by 10.75 percentage points.
- Ran data analysis and model ablation experiments, identifying puzzle-ID embeddings as a critical performance dependency.
- Benchmarked efficiency on H100: TRM (7M) used 2.4 GB VRAM at 31.3 samples/s vs Llama 3 8B QLoRA at 6.1 GB and 0.24 samples/s.
Methodology & stack
- Empirical ablations to isolate individual factors
- Efficiency analyses comparing computational requirements
- Systematic evaluation across multiple model configurations
- Python, PyTorch, ARC-AGI-1 benchmark, QLoRA