Skip to content
Softmax Saturation

Softmax Saturation

How Softmax Behaves

🦦Continued From Transformers Notes

Why does attention divide QKTQK^T by dk\sqrt{d_k}? This page covers:

  • What saturation is.
  • Why QKTQK^T spreads out as dkd_k grows.
  • Why 1dk\frac{1}{\sqrt{d_k}} cancels that spread.

☠️What Saturation Looks Like

  • Softmax: pi=ezi∑jezjp_i = \frac{e^{z_i}}{\sum_{j} e^{z_j}}.

  • Its derivative: ∂pi∂zj=pi(δij−pj)\frac{\partial p_i}{\partial z_j} = p_i (\delta_{ij} - p_j); δij=1\delta_{ij} = 1 if i=ji = j, else 00.

When inputs are far apart, one pk≈1p_k \approx 1 and the rest ≈0\approx 0 — literally a one-hot vector.

Then, pi(1−pi)≈0p_i (1 - p_i) \approx 0 and −pipj≈0-p_i p_j \approx 0 for every entry, so almost no gradient flows back through Softmax. This is saturation.

Example — scores with spread 8 (what dk=64d_k = 64 gives, as derived below), before and after dividing by 8:

Logits zzSoftmax Output WeightsMax Gradient pi(1−pi)p_i (1 - p_i)
(8,0,−8)(8, 0, -8)(0.99966,0.00034,0.0000001)(0.99966, 0.00034, 0.0000001)3.4×10−43.4 \times 10^{-4}
(1,0,−1)(1, 0, -1)(0.665,0.245,0.090)(0.665, 0.245, 0.090)0.220.22

Same ordering, but gradient signal is about 664× larger after scaling.

📏Why QKTQK^T Spreads Out

Each entry (QKT)i,j(QK^T)_{i, j} is a sum of dkd_k products. More terms, more spread. How much exactly?

☢️HIM — Heavy Incoming Math

Suppose all entries of QQ and KK are i.i.d. from a distribution of mean 0 and variance 1, implying several things:

  • E[Qi,h×Kj,h]=Cov(Qi,h,Kj,h)+E[Qi,h]×E[Kj,h]=0+0×0=0.E[Q_{i, h} \times K_{j, h}] = Cov(Q_{i, h}, K_{j, h}) + E[Q_{i, h}] \times E[K_{j, h}] = 0 + 0 \times 0 = 0.

  • ∀a,b∈R,E[Qi,ha×Kj,hb]=∫∫Qi,haKj,hbf(q,k)  dq  dk=∫∫Qi,haKj,hbfQ(q)fK(k)  dq  dk=∫Qi,hafQ(q)dq×∫Kj,hbfK(k)dk=E[Qi,ha]×E[Kj,hb]\forall a, b \in R, E[Q_{i, h}^a \times K_{j, h}^b] = \int \int Q_{i, h}^a K_{j, h}^b f(q, k) \; dq \; dk = \int \int Q_{i, h}^a K_{j, h}^b f_Q(q) f_K(k) \; dq \; dk = \int Q_{i, h}^a f_Q(q) dq \times \int K_{j, h}^b f_K(k) dk = E[Q_{i, h}^a] \times E[K_{j, h}^b].

  • ☝️f(q,k)=fQ(q)fK(k)f(q, k) = f_Q(q) f_K(k) decouples into an implied fact — Q2Q^2 and K2K^2 are independent.

And this heaviest one:

Var(Qi,h×Kj,h)=E[(Qi,h×Kj,h)2]−E[Qi,h×Kj,h]2Var(Q_{i, h} \times K_{j, h}) = E[(Q_{i, h} \times K_{j, h})^2] - E[Q_{i, h} \times K_{j, h}]^2

= E[Qi,h2×Kj,h2]−02=E[Qi,h2]×E[Kj,h2]=1×1=1.E[Q_{i, h}^2 \times K_{j, h}^2] - 0^2 = E[Q_{i, h}^2] \times E[K_{j, h}^2] = 1 \times 1 = 1.

So we can see that our (QKT)i,j(QK^T)_{i, j} distribution has:

I. Expectation: 0

E[(QKT)i,j]=E[Qi,:⋅(kj,:)T]=E[∑h=1dkQi,h×Kj,h]E[(QK^T)_{i, j}] = E[Q_{i, :} \cdot (k_{j, :})^T] = E[\sum_{h = 1}^{d_k} Q_{i, h} \times K_{j ,h}]

= ∑h=1dkE[Qi,h×Kj,h]=∑h=1dkE[Qi,h]×E[Kj,h]⏟All entries of Q and K are independent.\underbrace{ \sum_{h = 1}^{d_k} E[Q_{i, h} \times K_{j ,h}] = \sum_{h = 1}^{d_k} E[Q_{i, h}] \times E[K_{j ,h}] }_{\text{All entries of Q and K are } \textbf{independent.}}

= ∑h=1dk0×0=0\sum_{h = 1}^{d_k} 0 \times 0 = 0, so E[QKT]=0E[QK^T] = 0.

II. Variance: dkd_k

Var((QKT)i,j)=Var(Qi,:⋅(kj,:)T)Var((QK^T)_{i, j}) = Var(Q_{i, :} \cdot (k_{j, :})^T)

= Var(∑h=1dkQi,h×Kj,h)=∑h=1dkVar(Qi,h×Kj,h)⏟Each h uses its own entries of Q and K, so dk products are independent and variances add.\underbrace{ Var(\sum_{h = 1}^{d_k} Q_{i, h} \times K_{j ,h}) = \sum_{h = 1}^{d_k} Var(Q_{i, h} \times K_{j ,h}) }_{ \text{Each } h \text{ uses its own entries of Q and K, so } d_k \text{ products are } \textbf{independent} \text{ and variances add.}}

= ∑h=1dk1=dk\sum_{h = 1}^{d_k} 1 = d_k, so Var(QKT)=dkVar(QK^T) = d_k.

III. Assembly

Scale QKTQK^T by 1dk\frac{1}{\sqrt{d_k}} to ensure Var(QKTdk)=Var(QKT)dk=dkdk=1\text{Var}(\frac{QK^T}{\sqrt{d_k}}) = \frac{\text{Var}(QK^T)}{d_k} = \frac{d_k}{d_k} = 1:

  • When input follows a less extreme distribution, Softmax output tends to be less spiky.

  • Less spiky means less one-hot encoding — away from saturated vanishing-gradients areas.

🧪Simple Trials

One query against 16 keys, entries i.i.d. standard normal, 20,000 trials:

import numpy as np

def softmax(logits: np.ndarray) -> np.ndarray:
    max_col_entries = logits.max(axis=-1, keepdims=True)  # Shape: (N, 1).
    logits = logits - max_col_entries  # For overflow prevention. Shape: (N, D).
    
    softmax_output = np.exp(logits)
    softmax_output /= softmax_output.sum(axis=-1, keepdims=True)
    return softmax_output

rng = np.random.default_rng(0)
n_keys, trials = 16, 20000

for d_k in (16, 64, 256, 1024):
    q = rng.standard_normal((trials, 1, d_k))
    K = rng.standard_normal((trials, n_keys, d_k))
    
    scores = (q * K).sum(-1)  # Shape: (trials, n_keys).
    scaled_scores = scores / np.sqrt(d_k)

    print(
        d_k, scores.var(), scaled_scores.var(),
        softmax(scores).max(-1).mean(), softmax(scaled_scores).max(-1).mean()
    )
dkd_kVar(QKTQK^T)Var(QKTdk\frac{QK^T}{\sqrt{d_k}})Avg. Max pp: UnscaledAvg. Max pp: Scaled
1616.071.000.6850.244
6463.941.000.8400.244
256256.571.000.9230.246
10241023.131.000.9620.246

Avg. max pp: how much weight for the biggest Softmax entry:

  • Uniform over 16 keys would be 0.06250.0625.
  • Unscaled weights drift toward one-hot as dkd_k grows.
  • Scaled weights aren’t affected by dkd_k.