What is Mamba:
Definition: Mamba is a selective state space model designed to process sequences with near-linear scaling in sequence length and without a Transformer-style KV cache. Its key idea is selection: the model decides which information to keep, update, or discard as it moves through a sequence. This makes long-context inference efficient, but exact recall and in-context learning can still favor attention.
Mamba in 2026: What Held Up
When we first published this list in August 2024, Mamba research was exploding across text, vision, audio, and biology. Two years later, the picture has become clearer: pure Mamba remains compelling for efficient sequence modeling, but the strongest large models increasingly combine Mamba layers with attention rather than replacing attention altogether.
We no longer ask if Mamba is going to replace the Transformer. The real interest lies in where state space layers help, where attention remains necessary, and which combinations survive beyond a research benchmark. The papers and models below give us a much better answer than we had in 2024.
What Mamba Research Showed
The first wave of Mamba research carried the architecture into almost every modality. Some studies produced durable ideas; others mainly showed where selective state space models could be tested. This table separates the strongest result from the important caveat.
Paper or model | What it showed | Where it fell short |
|---|---|---|
Speech Slytherin (2024) | Tested Mamba across speech separation, recognition, and synthesis; its advantage was clearest on longer speech sequences. | Less efficient on shorter speech and on joint text-speech processing. |
Mamba (2023) | Introduced selective state space models for content-aware sequence processing with linear scaling in sequence length. | Later controlled studies found weaknesses in exact copying, in-context learning, and some long-context reasoning. |
Connected state space models and attention through structured state space duality; the Mamba-2 layer was reported two to eight times faster than Mamba. | Pure SSM variants still trailed attention on copying and in-context learning; realized speed depends on optimized kernels and hardware. | |
Mamba-3 (2026) | Adds a more expressive recurrence, complex-valued state updates, and a multi-input, multi-output design for better state tracking and decoding efficiency. | The paper evaluates models only up to 1.5B parameters, so large-scale production behavior is still unproven. |
Jamba2 (2026) | AI21’s open family combines Mamba and attention in 3B and 52B-total MoE models, with a 256K context window. | It is a hybrid, not evidence that pure Mamba can replace attention; several performance claims come from AI21’s own evaluations. |
Falcon-H1 (2025) | TII combines Transformer attention and Mamba layers across six open-weight model sizes, from 0.5B to 34B. | Results are vendor-reported, and the models use the TII Falcon License rather than a standard permissive license. |
Nemotron-H and Nemotron 3 (2025–2026) | NVIDIA scaled hybrid Mamba-Transformer models from dense Nemotron-H to the MoE-based Nemotron 3 family, prioritizing inference throughput. | They use attention, MLP, MoE, and enormous training budgets, so gains cannot be attributed to Mamba alone. |
MambaVision (2024) | Adapted Mamba to vision with a redesigned mixer and attention blocks for long-range spatial relationships. | Its strongest variants are hybrid; the results do not show that a pure SSM is uniformly better for vision. |
VSSD (2024) | Introduced a non-causal state-space formulation for vision classification, detection, and segmentation. | Evidence is vision-specific and benchmark-dependent; it does not establish a general replacement for attention. |
Audio Mamba (2024) | Used bidirectional state space modeling for audio representation learning without self-attention. | It was competitive or better on selected benchmarks, not uniformly superior across audio tasks and operating conditions. |
BioMamba (2024) | Applied Mamba to biomedical language representation and reported gains over biomedical encoder baselines. | The evidence is narrow and based on biomedical NLP benchmarks; it does not establish clinical reliability or general medical reasoning. |
MUSE (2024) | Used Mamba for efficient multi-scale text-video retrieval, where long visual sequences make linear processing attractive. | It is a specialized retrieval system, not a general video-understanding or generation model. |
What Changed After 2024
Mamba-2 connected state space models and attention through structured state space duality. Its central contribution was not simply another recurrent layer, but a clearer account of how efficient state space computation relates to attention. The paper reported a Mamba-2 core layer two to eight times faster than the original Mamba layer, although realized speed still depends on kernels, hardware, and the surrounding model.
Mamba-3, released in 2026, makes the recurrence more expressive through complex-valued state updates and a multi-input, multi-output design. It is an important architectural advance, but the paper evaluates models only up to 1.5B parameters. It is too early to treat those results as proof of frontier-scale behavior.
The clearest production trend is hybridization. AI21’s Jamba2, TII’s Falcon-H1, and NVIDIA’s Nemotron-H and Nemotron 3 combine Mamba-style layers with attention, MLPs, or mixture-of-experts components. That is not a retreat from Mamba. It is a practical recognition that efficient sequence processing and precise retrieval solve different problems.
The original summer 2024 papers remain useful as a record of the field’s first experiments. Their lasting lesson is that state space models gave researchers another way to think about memory, throughput, and long sequences.
FAQ
What is Mamba in AI?
Mamba is a selective state space model for processing sequences. Instead of comparing every token with every other token through self-attention, it updates a compact state as it moves through the sequence. That gives it near-linear scaling in sequence length and avoids the growing KV cache used by Transformer decoding.
Is Mamba better than a Transformer?
Not universally. Mamba can be faster and more memory-efficient on long sequences, but attention remains stronger for some forms of exact recall, copying, and in-context learning. This is why many practical systems now use hybrid architectures: Mamba layers handle efficient sequence processing, while attention layers preserve precise access to earlier information.
What is the difference between Mamba, Mamba-2, and Mamba-3?
Mamba introduced input-dependent selective state space layers. Mamba-2 reframed the architecture through structured state space duality and improved hardware efficiency. Mamba-3 adds a more expressive recurrence, complex-valued state updates, and a multi-input, multi-output formulation designed to improve state tracking without increasing decode latency.
Which current models use Mamba layers?
Current examples include AI21’s Jamba2 family, TII’s Falcon-H1 models, and NVIDIA’s Nemotron-H and Nemotron 3 families. These are hybrid systems rather than pure Mamba models: they keep some attention, MLP, or mixture-of-experts layers because different components solve different weaknesses.
Is Mamba open source?
The original Mamba implementation, Mamba-2, and Mamba-3 code and research checkpoints are publicly available. Models built with Mamba layers use different licenses, however. Jamba2 is released under Apache 2.0, while Falcon-H1 uses the TII Falcon License. Always check the license of the specific model, not only the architecture.
