Learning to (Learn at Test Time)
I’m excited to share a project I’ve been working on for over a year, which I believe will fundamentally change our approach to language models. We’ve designed a new architecture, which replaces the hidden state of an RNN with a machine learning model. This model compresses context through actual gradient descent on input tokens. We call our method “Test-Time-Training layers.” TTT layers directly replace attention, and unlock linear complexity architectures with expressive memory, allowing us to train LLMs with millions (someday billions) of tokens in context. Our instantiations, TTT-Linear…
In plain words
Learning to (Learn at Test Time) is a language model architecture that replaces traditional attention mechanisms with Test-Time-Training layers, which compress context through gradient descent on input tokens. Designed for researchers and developers building large language models, it enables linear complexity while maintaining expressive memory, allowing training with millions of tokens in context. The method achieves performance comparable to or better than Transformers and Mamba while offering improved efficiency for handling extended sequences.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
I’m excited to share a project I’ve been working on for over a year, which I believe will fundamentally change our approach to language models. We’ve designed a new architecture, which replaces the hidden state of an RNN with a machine learning model. This model compresses context through actual gradient descent on input tokens. We call our method “Test-Time-Training layers.” TTT layers directly replace attention, and unlock linear complexity architectures with expressive memory, allowing us to train LLMs with millions (someday billions) of tokens in context. Our instantiations, TTT-Linear and TTT-MLP, both match or beat the strongest Transformers and Mamba. Arxiv: https://arxiv.org/abs/2407.04620 --- Sequence models store historic context into a hidden state. RNN layers, like Mamba, compress into a state of fixed size across time. They’re efficient, but performance is limited by expressivity. Attention has a KV cache, which grows over time. This state doesn’t compress any historic context, but becomes costly as context length increases. Why don’t we compress context into the weights of a model – just like LLMs do with internet data? This “hidden state model” still has fixed size over time, but a lot more expressivity. We use self-supervised learning to update the hidden state weights, taking a gradient descent per on each token. After forwarding on a sequence, the state has been “trained” on the tokens in its context window. And remember, the hidden state lives in just 1 layer of the end-to-end architecture. The other components, like the QKV projection matrices, are learned during pre-training with the standard cross entropy objective. So, the end to end architecture is meta-learning the best way to compress context, such that it helps for next-token prediction. We are “Learning to (Learn at Test Time)” If it’s hard to believe that this actually works, our paper does a great job explaining the method in depth. We also cover some theoretical equivalences – this process of self-supervised learning is actually equivalent to self attention (if the hidden state model is a kernel). Our own instantiations, TTT-Linear and TTT-MLP, both match or beat the strongest Transformers and Mamba on perplexity performance. Plus, TTT-Linear is already faster than the fastest SSMs, and has strong scaling in size and context. The search space inside this framework is huge, and our paper has only taken a baby step. All our code, including training loops and datasets, is available in JAX (https://github.com/test-time-training/ttt-lm-jax) and PyTorch (https://github.com/test-time-training/ttt-lm-pytorch) --- Twitter thread: https://x.com/karansdalal/status/1810338845659131940
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 27d ago · cactuscompute.com
Launched alongside, July 2024
the whole month →
- IC
Many years ago, I made VJ softwares (to mix live visuals in clubs) for unexpected platforms like the Game Boy Advance, the Playstation 2 and the Raspberry Pi. This year, I’m back with a new web-app: Pikimov. Inspired by Photopea (a free Photoshop clone), I created this web-based motion design & video editor as an alternative to After Effects, to fill empty void. It's free, without signup, without cloud uploads (your files stay on your machine), and your projects are not used for AI models training.
AI · 2024 · pikimov.com
