Teknosys learning library

Layer normalization and residual connections

Deep transformer stacks repeatedly alter the scale and content of hidden states. Layer normalization and residual connections address different aspects of this process. Normalization rescales features within a representation; a residual connection lets a sublayer contribute an update without replacing every part of the existing representation. Neither is a substitute for a suitable objective, learning rate, or well-prepared data.

Normalize features, not the batch

For the common hidden-state shape (batch, tokens, hidden_width), layer normalization over hidden_width computes a separate mean and variance for each token vector. It subtracts the mean, divides by the square root of variance plus a small epsilon, then applies learned scale and bias parameters. The biased variance uses division by the number of normalized features.

from math import sqrt

x = [1.0, 2.0, 3.0]
mean = sum(x) / len(x)
variance = sum((value - mean) ** 2 for value in x) / len(x)
normalized = [(value - mean) / sqrt(variance + 1e-5) for value in x]
print([round(value, 3) for value in normalized])
# [-1.225, 0.0, 1.225]

This example leaves learned scale at one and bias at zero. After training, those parameters can change the output's mean and variance, so normalized states do not always have exactly zero mean and unit variance. Epsilon also matters when the input variance is small.

The residual path

A residual update takes the form x + sublayer(x), possibly with dropout on the update. It preserves an additive path from earlier states and helps gradients propagate through a deep network. Shapes must match: a sublayer cannot return a different feature width and still be added directly. The update may carry new information, while the old representation remains accessible.

Post-normalization:
    x_next = LayerNorm(x + Sublayer(x))

Pre-normalization:
    x_next = x + Sublayer(LayerNorm(x))

The original paper's blocks use post-normalization. Many later models use pre-normalization, often with an additional final normalization. The order changes the signal and gradient paths; copying pretrained weights into a differently ordered block is not a harmless refactoring. Other families use RMSNorm, which rescales using a root-mean-square statistic without the same mean-centering operation.

Practical checks

Unlike batch normalization, ordinary layer normalization does not rely on running batch statistics at evaluation time. It still computes statistics from the current input. A small batch therefore does not invalidate it, though small-batch training can have other optimization effects. In mixed precision, verify how your implementation computes variance and how it avoids overflow or loss of accuracy.

For a debugging exercise, feed two batches with identical first token vectors but different unrelated samples into a standalone layer-normalization module. The first token's normalized result should match when you normalize only its feature axis. If it changes, you may be reducing across the batch or sequence accidentally. Normalization should not silently mix examples or allow future-token information to leak into a causal model.

Sources: PyTorch LayerNorm documentation; the original residual and normalization design.

Educational material. Verify examples against current documentation and applicable licenses. Report a correction.