Teknosys learning library

Self-attention: queries, keys, values, and masks

Self-attention lets each token construct an input-dependent mixture of information from other tokens in the same sequence. “Self” describes the shared source of queries, keys, and values; it does not mean attending only to the current token. For a missing-charger ticket, the representation of “missing” might benefit from information associated with “charger”, but the weights are learned interactions rather than hand-written grammar rules.

Three projections, three roles

A hidden-state matrix X is projected into Q = X W_Q, K = X W_K, and V = X W_V. A query represents what a receiving position uses to match information; a key participates in that match; a value supplies the information to combine. These are useful computational roles, not literal questions, dictionary keys, or human-readable facts.

scores  = Q K^T / sqrt(d_k)
weights = softmax(scores + mask)
output  = weights V

For one head with T positions:
Q, K:    T x d_k
scores:  T x T
V:       T x d_v
output:  T x d_v

The scale factor moderates dot-product magnitude as key width grows. Softmax runs across permitted key positions for each query row. Larger scores receive more weight, and each non-empty row sums to one. The output is a weighted combination of values, not a weighted combination of token IDs.

A numerical example you can run

The following standard-library example uses fabricated already-scaled scores and two-dimensional values. It demonstrates one attention row, not a trained head or a full transformer. Subtracting the largest score before exponentiation improves numerical stability without changing the probabilities.

from math import exp

scores = [2.0, 1.0, 0.0]
values = [(1.0, 0.0), (0.0, 2.0), (3.0, 1.0)]
shift = max(scores)
unnormalized = [exp(score - shift) for score in scores]
weights = [value / sum(unnormalized) for value in unnormalized]
result = [
    sum(weight * value[axis] for weight, value in zip(weights, values))
    for axis in range(2)
]
print([round(weight, 3) for weight in weights])
print([round(value, 3) for value in result])
# [0.665, 0.245, 0.09]
# [0.935, 0.579]

Masks and multiple heads

A padding mask removes artificial positions. A causal mask prevents a generation position from seeing later target tokens. Masking logits before softmax differs from simply zeroing probabilities afterward: the remaining permitted weights must still be normalized. If a query has no allowed keys, ordinary softmax over all negative-infinity scores is undefined; avoid such rows or use an implementation that handles them explicitly.

Multi-head attention computes several projected views, concatenates their outputs, and applies an output projection. Heads can learn complementary interactions, but a head number does not come with a guaranteed linguistic job. Attention plots can help diagnose masks and patterns; they are not, on their own, causal explanations of a final prediction.

Source: scaled dot-product and multi-head attention in the original paper.

Educational material. Verify examples against current documentation and applicable licenses. Report a correction.