Teknosys learning library

The transformer decoder: generating one token at a time

A transformer decoder predicts an output sequence from an available prefix. In an encoder-decoder translator it also reads encoded source information. In a typical decoder-only language model, the prompt and generated tokens form one sequence, and there is no separate source encoder. These designs share causal generation behavior but differ in the presence of cross-attention.

The causal mask protects the objective

At a target position, masked self-attention allows the current input token and earlier prefix positions, but excludes later ones. This prevents training from using the answer it is supposed to predict. For a three-position prefix, the allowed connections form a lower-triangular pattern.

Key positions:      0  1  2
Query position 0:   1  0  0
Query position 1:   1  1  0
Query position 2:   1  1  1

Here 1 means allowed, not a universal library mask convention.

Suppose the target is a start token followed by “Please send photos” and an end token. Teacher-forced training supplies the earlier gold tokens as decoder inputs and supervises the corresponding next tokens. Libraries differ in whether label shifting occurs in a collator, model method, or loss. Shift once in the intended place, not twice.

Cross-attention when an encoder is present

In the original decoder, queries come from decoder states and keys and values come from the encoder output. That gives a target position access to source evidence, such as the ticket details needed for a reply. Source positions are not masked causally just because target generation is causal; source padding still needs to be excluded.

The decoder also contains a position-wise feed-forward network, normalization, and residual paths. Its language-model head maps the hidden width to vocabulary logits. Those logits define a next-token distribution, not a verdict that one complete answer is true.

Training parallelism and inference latency

All known target positions can be evaluated together during teacher forcing. At generation time, the chosen token changes the next input, so output production is sequential. A key-value cache avoids recomputing earlier keys and values for every new token. It saves repeated work but consumes memory and does not remove the dependency between successive output tokens.

Greedy decoding selects the highest-scoring token each step. Sampling with temperature or other filters changes the distribution from which tokens are chosen. Neither setting fixes unsupported claims. Bound generation length, handle end-of-sequence tokens, and evaluate whether the reply actually follows the input instead of judging fluency alone.

A useful leakage test

With dropout disabled, changing tokens strictly after a selected position should not change that position's causal hidden state, apart from numerical or implementation details. If it does, inspect mask orientation, normalization axes, and any noncausal token-mixing operation. A successful test verifies one structural property, not the overall correctness of a language model.

Sources: original decoder design; GPT-2 causal generation and cache documentation.

Educational material. Verify examples against current documentation and applicable licenses. Report a correction.