Teknosys learning library

The feed-forward layer: transforming each token

Attention moves information between positions. A position-wise feed-forward network changes how the features at each position are represented. Both are needed: distributing a useful clue across a ticket does not by itself supply the nonlinear transformation that turns that clue into a feature useful for classification or next-token prediction.

Expand, activate, project back

The original transformer applies two learned affine transformations with a ReLU activation between them. The first usually expands the hidden width to an intermediate width, and the second maps it back so that the result can be added to the residual stream. The same learned parameters are reused at every token position within that layer.

FFN(x) = activation(x W_1 + b_1) W_2 + b_2

Example widths:
input:        64 features
intermediate: 256 features
output:       64 features

The sequence length does not change.

Expansion provides more intermediate features; the activation allows a nonlinear mapping. Without an activation or another nonlinear operation, two consecutive affine transformations collapse mathematically into one affine transformation. Expansion alone would not create the same expressive computation.

A tiny, transparent calculation

Here is a single-vector example using explicit numbers. It is not a trained model and does not use the usual large intermediate width. It makes the two projections and the removal of negative intermediate values visible.

x = [1.0, -2.0]
w1 = [[1.0, -1.0, 0.5], [0.0, 1.0, -1.0]]
b1 = [0.0, 0.0, 0.0]
hidden = [
    max(0.0, sum(x[i] * w1[i][j] for i in range(2)) + b1[j])
    for j in range(3)
]
w2 = [[1.0, 0.0], [0.0, 1.0], [0.5, -0.5]]
output = [
    sum(hidden[i] * w2[i][j] for i in range(3))
    for j in range(2)
]
print(hidden)  # [1.0, 0.0, 2.5]
print(output)  # [2.25, -1.25]

Modern variations and their consequences

BERT uses GELU rather than the original ReLU. Other architectures use gated feed-forward networks, where one projection modulates another before the final mapping. Some models replace a dense network with a mixture of experts that routes tokens to selected expert networks. These choices affect parameter count, active computation, memory, and training behavior; “feed-forward” does not describe one universal implementation.

The feed-forward layer operates separately at each position, but its input has already been contextualized by attention. It is therefore wrong to call the entire output context-free. Conversely, the feed-forward transformation alone does not inspect another position unless the architecture includes additional token-mixing operations.

What to check when implementing it

Preserve the leading batch and sequence dimensions, apply the activation in the intended place, and return the hidden width expected by the residual addition. Disable dropout for deterministic evaluation where it is present. When comparing models, include feed-forward width and activation in the configuration record; matching only the number of attention heads leaves a large part of the model unspecified.

Sources: position-wise feed-forward networks; BERT activation and intermediate-width configuration.

Educational material. Verify examples against current documentation and applicable licenses. Report a correction.