All writing
AI research

How attention gives a language model context

A walk from an ambiguous sentence to token representations, attention weights, training, and next-token generation.

An illustrative image representing artificial intelligence
Conceptual illustration from Unsplash.

A naturalist writes: The ranger saw the bat by the bank. The sentence contains two polysemous words. The noun bat may denote a nocturnal flying mammal or an item of sports equipment, whereas bank may denote a sloping river edge or a financial repository. A subsequent sentence resolves the ambiguity: Its wings folded against the reeds as water reached the muddy shore. Readers immediately settle on the animal and the riverbank because subsequent syntactic and semantic cues reshape the interpretation of the earlier clauses. In modern neural language architectures, the attention mechanism performs an analogous computational role: it enables representations at each sequence position to dynamically pool contextual information from across the entire sequence.

In a Transformer model, attention operates as a parameterized routing mechanism that computes data-dependent weighted averages over representation vectors. That operation forms the computational core of contemporary autoregressive language models, providing the dynamic context that feed-forward layers and projection heads subsequently transform into probability distributions over vocabulary tokens.

Tokenization and embedding spaces

Before an attention layer can operate, raw text must be converted into numerical sequences through a tokenizer. Tokenization algorithms, such as Byte-Pair Encoding or WordPiece (Sennrich et al., 2016), segment text into discrete vocabulary indices representing whole words, morphological fragments, or individual characters. This subword decomposition balances vocabulary size against sequence length, allowing rare or compound words to be represented as sequences of known subword units without discarding semantic continuity.

Each discrete token index is mapped to a continuous vector through an embedding lookup table, yielding an initial coordinate array in Rdmodel\mathbb{R}^{d_{\text{model}}}. Unlike categorical class labels, dense embedding coordinates situate each token within a geometric space where inner products and Euclidean distances reflect distributional co-occurrence patterns acquired during pre-training.

Because standard dot-product operations are invariant to sequence permutation, the architecture requires an explicit mechanism to encode sequential order. Without positional information, a sequence such as bat near bank would yield an identical set of unmasked self-attention weights to bank near bat. In the original Transformer architecture, deterministic sinusoidal encodings across multiple frequencies were added directly to the input embeddings:

PE(pos,2i)=sin⁡(pos100002i/dmodel),PE(pos,2i+1)=cos⁡(pos100002i/dmodel)PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right), \quad PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)

Contemporary autoregressive models frequently replace absolute additive positions with Rotary Position Embeddings (RoPE), which apply an orthogonal rotation matrix to query and key vectors such that their inner product directly encodes the relative spatial distance between token positions.

The scaled dot-product attention calculation

Within each Transformer layer, the input representations are projected into three distinct functional roles: queries (QQ), keys (KK), and values (VV). For an input matrix X∈Rn×dmodelX \in \mathbb{R}^{n \times d_{\text{model}}}, linear projections yield:

Q=XWQ,K=XWK,V=XWVQ = X W^Q, \quad K = X W^K, \quad V = X W^V

where WQ∈Rdmodel×dkW^Q \in \mathbb{R}^{d_{\text{model}} \times d_k}, WK∈Rdmodel×dkW^K \in \mathbb{R}^{d_{\text{model}} \times d_k}, and WV∈Rdmodel×dvW^V \in \mathbb{R}^{d_{\text{model}} \times d_v} denote learned projection matrices. The query vectors specify what informational features a given position seeks, the key vectors define the indexable properties of available positions, and the value vectors supply the actual information content to be propagated.

The interaction between queries, keys, and values is formalized by the scaled dot-product attention equation introduced by Vaswani et al. (2017):

Attention(Q,K,V)=softmax(QKTdk)V\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V

The matrix product QKT∈Rn×mQK^T \in \mathbb{R}^{n \times m} evaluates pairwise directional alignment across all token pairs via inner products. When the key dimensionality dkd_k is large, dot products naturally scale in magnitude. Assuming components of qq and kk are independent random variables with zero mean and unit variance, their dot product has a mean of zero and a variance of dkd_k. For large dimensions, large dot products push the softmax function into regions with extremely small gradients, impeding backpropagation. Dividing the pre-softmax logits by the scaling factor dk\sqrt{d_k} preserves unit variance across dimensionality changes and stabilizes gradient updates during optimization.

Applying the row-wise softmax operator normalizes each row of the scaled logit matrix into a valid discrete probability distribution, ensuring all attention weights are non-negative and sum to unity across available key positions:

Aij=exp⁡(qi⋅kjdk)∑l=1mexp⁡(qi⋅kldk)A_{ij} = \frac{\exp\left(\frac{q_i \cdot k_j}{\sqrt{d_k}}\right)}{\sum_{l=1}^m \exp\left(\frac{q_i \cdot k_l}{\sqrt{d_k}}\right)}

The final output is computed by multiplying the attention weight matrix AA by the value matrix VV. For any single query position ii, the output vector is a convex combination of value vectors:

outputi=∑j=1mAijvj\text{output}_i = \sum_{j=1}^m A_{ij} v_j

To trace this computation with minimal concrete numbers, consider a query vector that distributes normalized scalar weights of 0.70, 0.20, and 0.10 across three key positions. If the corresponding value vectors are (2.0,0.0)(2.0, 0.0), (0.0,1.0)(0.0, 1.0), and (1.0,2.0)(1.0, 2.0), the resulting attention output evaluates to:

0.70(2.0,0.0)+0.20(0.0,1.0)+0.10(1.0,2.0)=(1.4,0.0)+(0.0,0.2)+(0.1,0.2)=(1.5,0.4)0.70(2.0, 0.0) + 0.20(0.0, 1.0) + 0.10(1.0, 2.0) = (1.4, 0.0) + (0.0, 0.2) + (0.1, 0.2) = (1.5, 0.4)

The attention distribution acts as a selective routing filter, directing the coordinate blending that produces the updated state.

Multi-head projections and layer integration

A single attention distribution can blend representations along only one set of projected dimensions at a time. To capture multiple distinct relational dependencies simultaneously, Transformers deploy multi-head attention. Instead of computing a single attention function with dimension dmodeld_{\text{model}}, the queries, keys, and values are linearly projected hh times with separate parameter matrices into subspaces of dimension dk=dv=dmodel/hd_k = d_v = d_{\text{model}} / h:

MultiHead(Q,K,V)=Concat(head1,…,headh)WO\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h)W^O
headi=Attention(QWiQ,KWiK,VWiV)\text{head}_i = \text{Attention}(Q W_i^Q, K W_i^K, V W_i^V)

where WiQ∈Rdmodel×dkW_i^Q \in \mathbb{R}^{d_{\text{model}} \times d_k}, WiK∈Rdmodel×dkW_i^K \in \mathbb{R}^{d_{\text{model}} \times d_k}, WiV∈Rdmodel×dvW_i^V \in \mathbb{R}^{d_{\text{model}} \times d_v}, and WO∈Rhdv×dmodelW^O \in \mathbb{R}^{h d_v \times d_{\text{model}}}. This modular projection allows individual heads to focus on distinct linguistic phenomena, such as local syntactic dependencies, coreference chains, or long-range associative pairings, before recombining their outputs through the linear projection WOW^O.

Each attention sub-layer is embedded within a standard residual block incorporating layer normalization:

x(1)=LayerNorm(x+MultiHead(x,x,x))x^{(1)} = \text{LayerNorm}(x + \text{MultiHead}(x, x, x))
x(2)=LayerNorm(x(1)+FFN(x(1)))x^{(2)} = \text{LayerNorm}(x^{(1)} + \text{FFN}(x^{(1)}))

The position-wise feed-forward network (FFN\text{FFN}) consists of two affine transformations with a non-linear activation function (such as GeLU or SwiGLU) and an inner expansion factor typically four times dmodeld_{\text{model}}:

FFN(z)=max⁡(0,zW1+b1)W2+b2\text{FFN}(z) = \max(0, z W_1 + b_1) W_2 + b_2

Residual connections preserve the identity gradient highway through deep networks, preventing gradient degradation and allowing early representations to bypass or mix smoothly with refined contextual states.

Architecture, masking, and autoregressive generation

The original 2017 Transformer architecture was designed as an encoder-decoder network for sequence transduction tasks like machine translation. In that framework, the encoder computes bidirectional self-attention, allowing every input token to attend freely to preceding and succeeding positions. The decoder contains two distinct attention operations: masked self-attention over the generated prefix, and cross-attention where queries originate from the decoder while keys and values originate from the encoder output.

Contemporary generative large language models, including GPT-3 (Brown et al., 2020), adopt a decoder-only architecture optimized for autoregressive next-token prediction. In autoregressive models, causality must be strictly preserved: when predicting the continuation of a sequence, position ii must not access information from future positions j>ij > i. Causal decoding enforces this restriction via an upper-triangular masking matrix M∈Rn×nM \in \mathbb{R}^{n \times n}:

Mij={0for j≤i−∞for j>iM_{ij} = \begin{cases} 0 & \text{for } j \le i \\ -\infty & \text{for } j > i \end{cases}

Adding MM to the pre-softmax logits, softmax((QKT/dk)+M)\text{softmax}((QK^T / \sqrt{d_k}) + M), sets the pre-softmax exponent to exp⁡(−∞)=0\exp(-\infty) = 0, guaranteeing that attention weights for future tokens are strictly zero.

During autoregressive training, causal masking enables fully parallel computation across the entire context window using teacher forcing. The model computes loss over all positions simultaneously in a single forward pass, with cross-entropy penalizing the discrepancy between predicted logits and true next-token labels.

During inference, token generation becomes sequential. The model processes the initial prompt, computes the next-token probability distribution via a softmax output projection over the vocabulary, and samples or selects the next token according to a decoding policy (such as greedy search, top-pp nucleus sampling, or temperature scaling). The newly generated token is appended to the input context, and the process repeats. To avoid recomputing keys and values for earlier tokens at each step, inference engines cache the key-value tensors (KV caching), reducing sequential generation time from quadratic to linear in sequence length.

Computational complexity and context limitations

Dense self-attention exhibits quadratic computational and memory complexity with respect to sequence length, scaling as O(n2dmodel)\mathcal{O}(n^2 d_{\text{model}}). Computing the complete n×nn \times n attention matrix requires storing n2n^2 activations per attention head, creating substantial memory bandwidth bottlenecks during backpropagation and inference. Modern implementations such as FlashAttention address these memory bottlenecks by tiling attention computations within high-speed GPU SRAM, evaluating the softmax reduction without materializing the full intermediate n×nn \times n matrix in high-bandwidth memory (HBM).

A model's context window specifies the maximum number of tokens it can ingest within a single forward pass. However, expanding context capacity does not imply uniform utilization across all positions. Systematic evaluations of multi-document question answering and key-value retrieval demonstrate that retrieval accuracy frequently degrades when target information is placed in the interior of long input sequences (Liu et al., 2024). This phenomenon, known as the lost-in-the-middle effect, reflects how positional encodings and optimization dynamics bias attention toward the immediate beginning and end of long prompts.

Interpretability and explanatory limits of attention

Heatmaps displaying attention matrices are widely used to visualize model behavior. Inspecting an attention map reveals how scalar weights were distributed across key tokens for a chosen query and head. However, assigning direct causal explanations to these weights requires methodological caution.

The interpretability debate initiated by Jain and Wallace (2019) and Wiegreffe and Pinter (2019) highlighted critical nuances in equating attention weights with explanatory importance. Attention weights are intermediate scalar multipliers, not final feature attribution scores. A high attention weight between position ii and position jj indicates that value vector vjv_j contributes heavily to the convex combination, but if the value projection maps that token into a low-magnitude vector or an orthogonal subspace, its net influence on subsequent layer representations may be minimal. Furthermore, residual connections carry substantial unmixed information across layers, and downstream feed-forward transformations systematically alter representation coordinates.

To evaluate whether an attention head causally governs a model prediction, researchers employ controlled counterfactual interventions: holding model weights and prompts fixed while perturbing specific attention patterns or ablating candidate heads. Grounding claims about model reasoning requires validating hypotheses against both the mathematical structure of the attention mechanism and empirical interventional data.

References

  • Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., ... Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877-1901. https://arxiv.org/abs/2005.14165
  • Jain, S., & Wallace, B. C. (2019). Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 3543-3556). https://aclanthology.org/N19-1357/
  • Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157-173. https://aclanthology.org/2024.tacl-1.9/
  • Sennrich, R., Haddow, B., & Birch, A. (2016). Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 1715-1725). https://aclanthology.org/P16-1162/
  • Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems, 30. https://arxiv.org/abs/1706.03762
  • Wiegreffe, S., & Pinter, Y. (2019). Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 11-20). https://aclanthology.org/D19-1002/