Skip to content

Transformers

Hood Under Attention

1. Attention

Some feelings about how can these attention-powered transformers work so well in Gen AI:

Natural language, or any sequence data, is literally like a more advanced Solitaire game.

Each position can — and should — look way beyond just the previous position.

🧱Encoder Diagram

Encoder reads the whole source at once, without any mask. Then it hands its output over as KK & VV for decoder to reference.

Encoder outputK, V for cross-attentionTo cross-attention ofevery decoder blockEncoder block × NAdd & NormFeed-forwardAdd & NormMulti-head self-attentionQ, K, V: all from input · no maskEmbedding + positional encodingInput tokens, all at onceRuns once, allpositions in parallelNo mask: each tokensees every position

3️⃣ Roles Inside Context

To better look deeper beneath context, each word is treated in 3 ways:

  • Query QQ: which topic is searched for.
  • Key KK: how each candidate is labeled.
  • Value VV: what content each candidate holds.

So — if each KK pairs with a VV, then how about we use QQ & KK matching results to decide how much each VV plays?

Stage ⓵: Scaled Dot-Product

  • Q=Rn×dk,K=Rn×dk,V=Rn×dv.Q = R^{n \times d_k}, K = R^{n \times d_k}, V = R^{n \times d_v}.

  • Attention(Q,K,V)=Softmax(QKTdk)V\text{Attention}(Q, K, V) = \text{Softmax} (\frac{ QK^T }{ \sqrt{d_k} }) V.

  • (QKT)i,j=Qi,:⋅(kj,:)T(QK^T)_{i, j} = Q_{i, :} \cdot (k_{j, :})^T.

🤿Mask If Required

After doing QKTdk\frac{ QK^T }{ \sqrt{d_k} }, you have a square matrix M=Rn×nM = R^{n \times n}, showing how much relative attention each token pays to other tokens.

But…..what if at times you aren’t totally free to pay attention?

Like, you can only pay attention to past and present. Not future.

Straightforward: given any ii, set Mi,j=−∞  ∀  i<jM_{i, j} = -\infty \; \forall \; i < j to have e−∞=0e^{-\infty} = 0.

Which is just to let Softmax produce 0, representing no attention, on disabled entries.

ones_matrix = torch.ones_like(qk_product)  # Shape: (n, n).

# Diagonal = 1: don't mask entire diagonal as well, cuz present can be referenced.
mask = torch.triu(ones_matrix, diagonal=1).bool()

qk_product.masked_fill_(mask, float("-inf"))

⬆️This code helps us mask out all upper triangular entries before entering Softmax.

🗞️Dot Product Comes

  • P=Softmax(QKTdk)=Rn×nP = \text{Softmax}(\frac{ QK^T }{ \sqrt{d_k} }) = R^{n \times n}: with ∑j=1nPi,j=1\sum_{j = 1}^n P_{i, j} = 1, Pi,jP_{i, j} is ithi^{th} token’s relative proportion of attention to jthj^{th} token.

  • Be sure to set axis=-1 when doing Softmax. Along columns 😉😏

  • V=Rn×dv:V = R^{n \times d_v}: each Vi,:=RdvV_{i, :} = R^{d_v} means ithi^{th} token’s value representation.

  • PVPV yields a matrix of Rn×dvR^{n \times d_v}, where each ithi^{th} row vector is a weighted sum of token vectors.

  • Weights are based on ithi^{th} token’s attention to all nn tokens.

🧮More Softmax Details: Visit

Stage ⓶: MHA — Multi-Head

Simple intro: hh sets of 3 linear projections before scaled dot-product.

Philosophy behind: have more matrices track how each token behaves in a certain semantic space.

Detailed Maneuvers 🥘

Now Q=K=V=Rn×dmodel.Q = K = V = R^{n \times d_{model}}. Each lthl^{th} head owns 3 weight matrices called:

  • WlQ=Rdmodel×dkW_l^Q = R^{d_{model} \times d_k} — Help produce Ql=QWlQ=Rn×dkQ_l = Q W_l^Q = R^{n \times d_k}.

  • WlK=Rdmodel×dkW_l^K = R^{d_{model} \times d_k} — Help produce Kl=KWlK=Rn×dkK_l = K W_l^K = R^{n \times d_k}.

  • WlV=Rdmodel×dvW_l^V = R^{d_{model} \times d_v} — Help produce Vl=VWlV=Rn×dvV_l = V W_l^V = R^{n \times d_v}.

So Ql,Kl,VlQ_l, K_l, V_l all fit into expected input shapes of scaled dot-product.

lthl^{th} scaled dot-product attention is Al=Rn×dvA_l = R^{n \times d_v}. We will have hh such matrices.

With each row referring to a token, we of course can only concatenate attention along columns.

By doing so, we obtain O=Concat(A1,...,Ah)=Rn×hdvO = \text{Concat}(A_1, ..., A_h) = R^{n \times h d_v}.

Using another weight matrix WO=Rhdv×dmodelW^O = R^{h d_v \times d_{model}}, we can reach OWO=Rn×dmodelO W^O = R^{n \times d_{model}}.

Yes, back to original shape of Q,K,VQ, K, V, but carrying learned attention.

🤔Subtle & Vital Guideline

For MHA, we need extra 3h+13h + 1 weight matrices:

  • First 2h2h matrices — all Rdmodel×dkR^{d_{model} \times d_k} that deal with Q,KQ, K.

  • Middle hh matrices — all Rdmodel×dvR^{d_{model} \times d_v} that deal with VV.

  • Last 1 matrix — Rhdv×dmodelR^{h d_v \times d_{model}} that brings back to original shape.

All these 3h+13h + 1 matrices share a common: dimensions contain no nn.

At the End of the Day

Attention is to grasp semantic relationship among nn tokens.

dmodel,dk,dv,hdvd_{model}, d_k, d_v, h d_v are all semantic spaces’ dimensions. nn isn’t.

These 3h+13h + 1 weight matrices are born to travel around semantic spaces. 😉😏

🕰️Time Complexity

QKT:O(n2×dk)QK^T: O(n^2 \times d_k)

  • Rn×dk⋅Rdk×nR^{n \times d_k} \cdot R^{d_k \times n} — n2n^2 pairs of RdkR^{d_k} doing inner product, with O(dk)O(d_k) per pair.

dk\sqrt{d_k} Scaler & Softmax: O(n2)O(n^2)

  • They both perform constantly on all n2n^2 entries inside QKTQK^T.

V Multiplication: O(n2×dk)O(n^2 \times d_k)

  • Rn×n⋅Rn×dkR^{n \times n} \cdot R^{n \times d_k} — n×dkn \times d_k pairs of RnR^n doing inner product, with O(n)O(n) per pair.

So total is O(n2×dk)+O(n2)+O(n2×dk)=O(n2×dk).O(n^2 \times d_k) + O(n^2) + O(n^2 \times d_k) = O(n^2 \times d_k).

2. Generation 🏳️‍🌈

How to make good use of these learned attention?

Like writing a research paper:

  • Ensure no topic deviation — look at already written parts.
  • Extend relevant content — make reference to other papers.

Although I’ve never written a research paper 😎

🧱Decoder Diagram

Decoder writes one token at a time:

  • Dashed parts — cross attention related things — exist only in encoder-decoder models.
  • Decoder-only LLMs keep just masked self-attention.

Teal boxes mark where KK & VV come from or get stored.

Next-token probabilitiesLinear + softmaxDecoder block × NAdd & NormFeed-forwardAdd & NormCross-attentionQ: decoder · K, V: encoderAdd & NormMasked self-attentionQ, K, V: all from output so farEmbedding + positional encodingOutput tokens generated so farEncoder outputK, V computed onceK, VDashed: dropped indecoder-only LLMsKV cachegrows each step

🎭Masked Attention: No Deviation

Like I said, to prevent topic deviation, we must look at already written parts.

So in each step, already generated output is naturally the sources for our QQ, KK and VV.

The adjective “masked” means only previous and current steps’ output is eligible.

⚔️Cross-Attention: Make Reference

Where can we let generated output make reference to incorporate relevant content?

Encoder’s learned attention, aka encoder stack. Thus, this time:

  • Encoder stack attention serves as sources of KK & VV, since it gets referenced.

  • We are allowed to look at every step. So no masks at all.

Tip: Output Is Topic Regardless

AttentionQQ sourceKK & VV sourcesMasked
MaskedOutput.Output.Yes.
CrossOutput.Encoder stack attention.No.

🎯Output Head: Linear & Softmax

Decoder diagram’s top is a “Linear + softmax” box: it turns each position’s vector into a probability over whole vocabularies.

  • Decoder stack output: Rn×dmodelR^{n \times d_{model}}.

  • Linear layer uses W∈Rdmodel×∣V∣W \in R^{d_{model} \times |V|} to transform decoder stack output into logits of Rn×∣V∣R^{n \times |V|}.

  • Softmax along vocabularies index axis=-1 makes each row of Rn×∣V∣R^{n \times |V|} sum to 1.

Training: Everything Matters

Each row compares to its true next token, tracking loss such as cross-entropy. Padding positions are left out of loss calculations.

Inference: Just Last One Matters

Only last row matters, since earlier tokens are already fixed. Each decode step:

  • Takes ∣V∣|V| probabilities’ last row.
  • Decodes: choose next token from them — argmax, top k, temperature, etc.
  • Appends it to already generated sequence.
  • Stops at EOS or a length limit.

Which Row Is Last? 🧐

“Last row” isn’t always the last real token, depending on:

  • Right padding — pads the future, so causal mask already hides them. But sequence ii’s last real token is at index Li−1L_i - 1, found by attention-mask length.

  • Left padding — pads the past. Last column is always a real token. But pads are now “past” and thus visible, so attention mask must hide them, with position ids starting from first real token.

3. KV Cache

Now comes a discussion: can we optimize time or space complexities?

This brings up a quite fundamental topic indeed: KV Cache.

It’s mentioned in decoder diagram above. For details, please see: KV Cache.

4. Positional Encoding

“Attention Is All You Need” uses sinsin and coscos to encode positions.

But there’s another interesting way to do so as well.

Second thought — decided to make an exclusive page for positional encoding only.