Learning to (Learn at Test Time)
I’m excited to share a project I’ve been working on for over a year, which I believe will fundamentally change our approach to language models. We’ve designed a new architecture, which replaces the hidden state of an RNN with a machine learning model. This model compresses context through actual gradient descent on input tokens. We call our method “Test-Time-Training layers.” TTT layers directly replace attention, and unlock linear complexity architectures with expressive memory, allowing us to train LLMs with millions (someday billions) of tokens in context. Our instantiations, TTT-Linear…
Learning to (Learn at Test Time) launched on July 8, 2024 with 13 votes, #1,242 of 5,184 launches that month and more than 75% of that year's launches.
In plain words
Learning to (Learn at Test Time) is a language model architecture that replaces traditional attention mechanisms with Test-Time-Training layers, which compress context through gradient descent on input tokens. Designed for researchers and developers building large language models, it enables linear complexity while maintaining expressive memory, allowing training with millions of tokens in context. The method achieves performance comparable to or better than Transformers and Mamba while offering improved efficiency for handling extended sequences.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
I’m excited to share a project I’ve been working on for over a year, which I believe will fundamentally change our approach to language models. We’ve designed a new architecture, which replaces the hidden state of an RNN with a machine learning model. This model compresses context through actual gradient descent on…
Does a similar job
all alternatives →
I built a tiny LLM to demystify how language models workApr 2026 · github.com · ▲915Built a ~9M param LLM from scratch to understand how they actually work. Vanilla transformer, 60K synthetic conversations, ~130 lines of PyTorch. Trains in 5 min on a free Colab T4. The fish thinks the meaning of life is food. Fork it and swap the personality for your own character.
Gan.AI TTS Model & API Playground2024 · ▲333First TTS model to support all 22 Indic languages + English
Duplicate 3 layers in a 24B LLM, logical deduction .22→.76. No trainingMar 2026 · github.com · ▲265I replicated David Ng's RYS method (https://dnhkng.github.io/posts/rys/) on consumer AMD GPUs (RX 7900 XT + RX 6950 XT) and found something I didn't expect. Transformers appear to have discrete "reasoning circuits" — contiguous blocks of 3-4 layers that act as indivisible cognitive units. Duplicate the right block and the model runs its reasoning pipeline twice. No weights change. No training. The model just thinks longer. The results on standard benchmarks (lm-evaluation-harness, n=50): Devstral-24B, layers 12-14 duplicated once: - BBH Logical Deduction: 0.22 → 0.76…
Inworld TTS2025 · inworld.ai · ▲318Hey HN, this is Igor, one of the engineers behind Inworld Runtime, which we're releasing today in public preview. We built it to solve the common problem we and our customers had: engineers spend more time on AI ops and plumbing than on actual feature development. This was often due to the challenge of using Python for I/O-bound, high-concurrency workloads and complexity maintaining pipelines with streams that use always-changing ML models. Our solution is a high-performance runtime written in C++ with the core idea of defining AI logic as graphs. For instance, a basic voice-to-voice…
A perceptron with stochastic gradient descent in Go2017 · github.com · ▲94A single / multi layer / recurrent neural network written in Golang. - made2591/go-perceptron-go
- ASA surprisingly effective way to predict token importance in LLM prompts2023 · heatmap.demos.watchful.io · ▲14
We explored a novel method to gauge the significance of tokens in prompts given to large language models, without needing direct model access. Essentially, we just did an ablation study on the prompt using cosine similarity of the embeddings as the measure. We got surprisingly promising results when comparing this really simple approach to integrated gradients. Curious to hear thoughts from the community!
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 21d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 29d ago · world.pixverse.video
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, July 2024
the whole month →
Many years ago, I made VJ softwares (to mix live visuals in clubs) for unexpected platforms like the Game Boy Advance, the Playstation 2 and the Raspberry Pi. This year, I’m back with a new web-app: Pikimov. Inspired by Photopea (a free Photoshop clone), I created this web-based motion design & video editor as an alternative to After Effects, to fill empty void. It's free, without signup, without cloud uploads (your files stay on your machine), and your projects are not used for AI models training.
AI · 2024 · pikimov.com
