What is attention in AI? This question may sound simple, but there are actually many aspects worth a deep discussion.

In the previous episodes, we explored what a token in AI is and followed tokens as they became embeddings: dense vectors, that are also points in a learned space and coordinates with semantic structure. Embeddings are where meaning lives, but coordinates alone are not enough. The next natural step in our series is attention. So what is its role in the workflow?

Attention solves a harder task than previous stages.: for this token, in this sentence, at this exact moment, which other tokens are relevant enough to change its meaning? The overall mechanism lets one token attend to another token, decide how relevant it is, and pull in exactly the information it needs. At this moment a sequence stops being a row of vectors and becomes context.

This mechanism became an indispensable foundation and changed AI once and forever, especially becoming the central computation part of Transformers.

So let’s unpack where attention comes from, how it works, the core components and types of this mechanism, and why attention isn’t the same as understanding. It’s foundational knowledge.

In today’s episode:

Before attention became the center of the Transformer described in one of the most influential papers in AI – Attention is All You Need from Google, it had appeared three years earlier as a solution for a practical problem in neural machine translation.

Early encoder–decoder models encoded an entire source sentence into a single fixed-length vector – a dense numerical representation that captures the sentence’s meaning – and decoded the translation from that compressed summary. Dzmitry Bahdanau, KyungHyun Cho and Yoshua Bengio in their Neural Machine Translation by Jointly Learning to Align and Translate paper (2014) argued that this fixed-length context vector created a bottleneck, because the model had to compress all relevant information into one representation. They proposed a model that could align source and target words during decoding, letting the decoder softly search over different parts of the input sentence while generating each new word. The decoder dynamically computes a context vector as a weighted combination of source annotations, focusing on the most relevant parts of the input for the current prediction. This was the moment when context became adaptive and target-dependent.

Then in 2015, Stanford researchers published Effective Approaches to Attention-based Neural Machine Translation by Minh-Thang Luong, Hieu Pham and Christopher D. Manning that extended this idea with practical attention mechanisms. They introduced global attention, where the decoder attends to all source positions, and local attention, where it focuses only on a smaller window of source words at each step. Plus, they proposed input-feeding approach, where the model feeds earlier attention information back into later steps so it can remember what parts of the source it has already focused on.

Then finally came the main breakthrough – the architecture built around attention itself. Yes, it is about Transformers and Attention Is All You Need paper (2017) by A. Vaswani et al. The Google researchers removed recurrence and convolutions and made self-attention layers the core foundation of Transformers. They also introduced the formulation of attention and the language of queries, keys, and values that became the canonical way to describe how models retrieve and combine contextual information. One of the most influential architectures built on this foundation is BERT — see our deep dive on BERT and its modern variants.

This is just the story of how attention became the centerpiece of modern models. It’s not enough to understand the basics. Let’s go through every part of the workflow step-by-step.

The usual Transformer model starts with token embeddings (dense numerical vector representations of tokens that encode semantic and syntactic information in a continuous vector space), combined with positional encodings, because the word order matters a lot in this architecture. At this stage, these vectors are still not deeply contextual. They have information about token identity, and with positional encoding they “know” something about location, but they do not yet define what matters to what.

So these “token embeddings + positional encodings” vectors serve as the inputs to the attention mechanism, entering the Transformer’s attention layers.

There, each token representation is transformed into queries, keys, and values. These are linear projections of embeddings or hidden states – context-aware vector representations of tokens as they are processed layer by layer inside the model. Embeddings provide the raw material from which attention builds its comparisons. Without embeddings, the mechanism would have nothing meaningful to compare or route through the network.

Here is the essential vocabulary that everyone needs to know to understand the full mechanism and attention formulation.

After embeddings and positional information enter the model, each token vector is multiplied by learned weight matrices to produce three different versions of itself:

But why can’t we just use ordinary embeddings, and why do we need to split them into these three vectors?

The answer is simple: a token does not have just one job. A model may need one representation for searching, another for matching, and another for carrying information forward. One word can simultaneously be a requester, a candidate match, and a carrier of content. Without this separation for Q, K, and V, attention would collapse multiple roles into one vector and lose its flexibility.

Now it’s time to move on to the core aspect – computation. A classic formulation for attention looks like this:

Let’s figure out what is so important here.

In decoder self-attention, the model also applies masking, preventing tokens from attending to future positions. This preserves the autoregressive behavior of text generation, where each token can depend only on earlier tokens.

So this is the workflow in easy words: a token enters as a vector, asks a question, scans the sequence for relevant keys, gathers weighted values, and exits as a richer vector. The process repeats for every token and across every Transformer layer. We can also call attention a lookup mechanism that learns relevance in a high-dimensional space. And this is the moment where embeddings become contextual representations refined layer by layer.

Interestingly, in a Multi-Head Attention (MHA) variant, this workflow is performed several times in parallel using different learned projections of Q, K, and V. Different attention heads (independent attention operation) can focus on different kinds of relationships and representation subspaces, like local dependencies, long-range dependencies, and structural patterns. The outputs of all heads are then fused and projected again into a single representation.

In Transformers, text generation basically works one token at a time, and when generating the next token, the model still needs access to all previous tokens. If it recomputes their keys and values at every step (for example, for token 501 it recomputes everything for tokens 1-500 from scratch), this would make the decoding very slow.

So there is a solution: systems store the previously computed key and value vectors, forming KV cache. Models compute only new query, key, and value vectors for the current token, then attends over the cached representations from previous positions. In practice, past queries aren’t needed.

But in this case the main tradeoff becomes memory: KV cache grows with sequence length, batch size, number of layers, and in standard multi-head attention, with the number of attention heads, since each head maintains its own cached key and value states.

This is why so much attention research focuses on making KV storage more efficient. Multi-Query Attention (MQA) shares one key-value head across all query heads. Grouped-Query Attention (GQA) uses several KV groups, keeping much of MQA’s speed while staying closer to full multi-head quality. Cross-Layer Attention (CLA) shares KV activations across neighboring Transformer layers, reducing KV cache memory use by up to 2x compared to MQA and GQA. And Multi-Head Latent Attention (MLA) compresses KV states into lower-rank latent vectors. It was first introduced with the release of DeepSeek-V2, reducing KV cache by 93.3% in comparison to the previous DeepSeek model.

If you want to explore the core and most popular attention mechanisms, including FlashAttention algorithm optimized for the hardware (GPUs) that uses fast on-chip memory and the ones that we have already mentioned here, we recommend you to check out this list with research papers provided →

And here, we’ll focus on some of the most interesting recent attention variants instead.

So the core directions in attention research are moving toward more aggressive reduction of KV cache and specifically adapted for the very long context. If you look deeper, this trend makes a lot of sense: model reasoning chains are now much longer, agents keep much more context in memory, context windows keep growing, but at the same time we still want the speed and efficiency of much lighter models. And probably, we are still waiting for a new breakthrough in this field.

Before we finish, let’s clarify one more important point. Attention is often described using human metaphors, but AI attention is about the mechanics and compute. It does not “understand” information the way humans do, relying on memory, awareness, and intent. Attention in AI just computes which parts of the input should influence the output more strongly.

If a model sees the sentence “The big red ball was under the table” and is asked “Where was the red ball?”, attention helps it focus on the relevant parts – “red ball” and “under the table.” But if asked “Where was the blue balloon?”, the model may or may not detect the inconsistency depending on its training and internal representations, while a human immediately notices the mismatch.

That is why attention is not the same as understanding. Attention helps the model stay grounded in context, but it doesn't guarantee correctness. It is just a part of the stack that makes models more relevant. Of course, without attention, models would never have reached this level of performance. However, attention is still ultimately a mechanism – a tool for prioritizing input that helps models stay on track.

Resources From Turing Post: