Softmax Saturation
How Softmax Behaves
🦦Continued From Transformers Notes
Why does attention divide by ? This page covers:
- What saturation is.
- Why spreads out as grows.
- Why cancels that spread.
☠️What Saturation Looks Like
Softmax: .
Its derivative: ; if , else .
When inputs are far apart, one and the rest — literally a one-hot vector.
Then, and for every entry, so almost no gradient flows back through Softmax. This is saturation.
Example — scores with spread 8 (what gives, as derived below), before and after dividing by 8:
| Logits | Softmax Output Weights | Max Gradient |
|---|---|---|
Same ordering, but gradient signal is about 664× larger after scaling.
📏Why Spreads Out
Each entry is a sum of products. More terms, more spread. How much exactly?
☢️HIM — Heavy Incoming Math
Suppose all entries of and are i.i.d. from a distribution of mean 0 and variance 1, implying several things:
.
☝️ decouples into an implied fact — and are independent.
And this heaviest one:
=
So we can see that our distribution has:
I. Expectation: 0
=
= , so .
II. Variance:
=
= , so .
III. Assembly
Scale by to ensure :
When input follows a less extreme distribution, Softmax output tends to be less spiky.
Less spiky means less one-hot encoding — away from saturated vanishing-gradients areas.
🧪Simple Trials
One query against 16 keys, entries i.i.d. standard normal, 20,000 trials:
import numpy as np
def softmax(logits: np.ndarray) -> np.ndarray:
max_col_entries = logits.max(axis=-1, keepdims=True) # Shape: (N, 1).
logits = logits - max_col_entries # For overflow prevention. Shape: (N, D).
softmax_output = np.exp(logits)
softmax_output /= softmax_output.sum(axis=-1, keepdims=True)
return softmax_output
rng = np.random.default_rng(0)
n_keys, trials = 16, 20000
for d_k in (16, 64, 256, 1024):
q = rng.standard_normal((trials, 1, d_k))
K = rng.standard_normal((trials, n_keys, d_k))
scores = (q * K).sum(-1) # Shape: (trials, n_keys).
scaled_scores = scores / np.sqrt(d_k)
print(
d_k, scores.var(), scaled_scores.var(),
softmax(scores).max(-1).mean(), softmax(scaled_scores).max(-1).mean()
)| Var() | Var() | Avg. Max : Unscaled | Avg. Max : Scaled | |
|---|---|---|---|---|
| 16 | 16.07 | 1.00 | 0.685 | 0.244 |
| 64 | 63.94 | 1.00 | 0.840 | 0.244 |
| 256 | 256.57 | 1.00 | 0.923 | 0.246 |
| 1024 | 1023.13 | 1.00 | 0.962 | 0.246 |
Avg. max : how much weight for the biggest Softmax entry:
- Uniform over 16 keys would be .
- Unscaled weights drift toward one-hot as grows.
- Scaled weights aren’t affected by .