The Sub-Quadratic Showdown: Mamba, Jamba, Titans, and the End of Transformer Dominance
A transformer reading 1,000,000 tokens does not read 1,000 times more work than a 1,000-token prompt. Naive attention asks roughly 1 trillion pairwise questions instead of 1 million. That square is the tax. Mamba, Jamba, and Titans are three attempts to stop paying it without giving up the one thing attention does well: exact retrieval from the past.
The fight is not "attention versus no attention." It is a fight over memory shape. Should a model keep every token visible, compress the past into a state, mix the two, or update a separate memory while it reads?
The wall
Attention computes a score between each query token and each key token:
cost ~ n^2
At 4,096 tokens, that is about 16.8 million pairwise scores. At 128,000 tokens, it is about 16.4 billion. Engineering can hide some of the pain with FlashAttention, KV caching, sparsity, and better kernels, but the basic geometry remains: every new chunk wants to look backward across a growing table.
State-space models take the older path:
h_t = A_t h_{t-1} + B_t x_t
y_t = C_t h_t
The model carries a state h. Mamba's 2023 move was to make the transition terms depend on the input token, so the model can decide what to keep and what to forget. That turns recurrence from a blind conveyor belt into a selective memory system.
Three bets
| Architecture | First paper | Core bet | Hard number |
|---|---|---|---|
| Mamba | 2023 | Replace attention with selective recurrence | Reported 5x higher inference throughput than transformers in Gu and Dao 2023 |
| Mamba-2 | 2024 | Show attention and SSMs meet inside structured matrices | Reported 2-8x faster core layer than Mamba in Dao and Gu 2024 |
| Jamba | 2024 | Use Mamba most of the time, keep attention where it pays | Up to 256K token context in the AI21 paper |
| Titans | 2025 | Add a neural long-term memory that learns during inference | Tested beyond 2M context window in Behrouz, Zhong, and Mirrokni 2025 |
Mamba is the cleanest bet. It says the past can be compressed if the compression is token-aware. Mamba-2 makes the stranger claim: a subset of state-space models and a subset of attention are two views of related structured matrix operations. The line between recurrence and attention is less metaphysical than it looked in 2017.
Jamba is the operator's compromise. AI21 interleaves Transformer and Mamba blocks, then uses mixture-of-experts capacity so only part of the parameter count is active per token. The public Jamba paper reports a model that fits on a single 80GB GPU and handles 256K-token contexts. That is not a funeral for attention. It is attention demoted from monarch to specialist.
Titans changes the question again. Instead of forcing the entire past into attention or one recurrent state, it adds a neural memory module that learns to memorize during test time. The paper frames attention as short-term memory and neural memory as long-term memory. That is a different bet: the model should not only attend to context; it should decide what becomes memory.
Where the transformer still bites back
The transformer has one clean advantage: it keeps the evidence table explicit. If the prompt says "Alice = 48172" on page 2 and asks for Alice on page 80, softmax attention has a direct retrieval path. A recurrent state has to have decided that "Alice = 48172" was worth preserving.
That is why the strongest sub-quadratic claims often split by task. Long-context throughput, genomics, audio, and time series fit recurrence well. Exact key-value recall and short-context tool use still favor attention more often than the hype cycle admits.
The honest framing is simple: transformers waste compute; recurrent models risk forgetting the one token you later need.
What's contested
The live question is not whether sub-quadratic models can win benchmarks. They can. Mamba, Jamba, and Titans all report wins in specific settings.
The contested question is whether those wins survive three pressures at once: frontier scale, tool-heavy instruction following, and adversarial retrieval. Academic long-context tests often measure needle finding; production agents need citation grounding, scratchpad discipline, and exact state over hours. A 2M-token context window is not the same object as a reliable memory.
There is also a naming problem. "Linear time" sounds decisive, but GPU reality cares about kernels, batch size, memory movement, and active parameters. Big-O notation is a map, not the machine.
Why this has to do with other realms
This page sits in computing, but the deeper question is biological: what should a memory system forget? A brain does not keep every photon-level input visible. It keeps a working state, stabilizes some episodes, routes attention through gates, and lets most of yesterday decay. Titans makes that analogy explicit; Jamba arrives at a similar structure through hardware pressure.
That is why concept thalamo cortical attention matters here. If cortex looks attention-like and the hippocampus looks memory-like, then concept neuromorphic computing is not only about chips that spike. It is about whether future models should look less like a giant lookup table and more like a nervous system with several memory speeds.
Open question
If a model learns during inference, when does memory become behavior rather than context?
Key Sources
- Vaswani et al. (2017), "Attention Is All You Need", arXiv:1706.03762 - the transformer baseline and the attention mechanism this page reacts against.
- Gu and Dao (2023), "Mamba: Linear-Time Sequence Modeling with Selective State Spaces", arXiv:2312.00752 - selective SSMs, hardware-aware scan, reported 5x inference throughput claim.
- Dao and Gu (2024), "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality", arXiv:2405.21060 - Mamba-2 and the structured state-space duality argument.
- Lieber et al. (2024), "Jamba: A Hybrid Transformer-Mamba Language Model", arXiv:2403.19887 - hybrid Mamba, attention, and MoE design with 256K-token evaluations.
- Behrouz, Zhong, and Mirrokni (2025), "Titans: Learning to Memorize at Test Time", arXiv:2501.00663 - neural long-term memory and 2M-context experiments.
Further Reading
- concept transformer architecture - the 2017 baseline that made attention the default grammar of language models.
- concept thalamo cortical attention - the biological rhyme: routing, attention, and layered memory.
- "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness" by Dao et al. (2022) - why the memory wall is also an IO problem.
- "Efficiently Modeling Long Sequences with Structured State Spaces" by Gu et al. (2021) - the S4 ancestor behind Mamba.
- concept physarum memory - a non-neural organism that stores useful paths by changing its own body.
See Also
- concept transformer architecture
- concept thalamo cortical attention
- concept neuromorphic computing
- concept halting problem
- concept arrow of time
- concept physarum memory
Abhishek's take
The part that grabs me is not that transformers may lose. It is that every serious replacement starts looking like an argument about memory discipline. I care less about a 2M-token demo than about the missing rule: what deserves to persist after the prompt is gone?
Tags: #transformers #mamba #ssm #state-space-models #attention #long-context #memory