Teknosys learning library

The encoder-decoder branch: T5, BART, and sequence transformation

The encoder-decoder branch keeps source processing and target generation as connected but distinct stages. The encoder reads the supplied input with its permitted context. The decoder predicts a target sequence causally while cross-attending to encoder states. This is a natural structure for transforming a document into a summary, a sentence into a translation, or a ticket into a constrained reply.

Two sequences, three attention patterns

Encoder self-attention operates among source positions. Decoder self-attention operates among permitted target-prefix positions. Cross-attention uses decoder queries and encoder keys and values, connecting the two sequences. Source and target lengths can differ, so a cross-attention score matrix has target positions on one axis and source positions on the other.

Source ticket:      24 tokens
Target reply:       12 tokens during teacher-forced training

Encoder scores:     24 x 24 per head
Decoder self scores:12 x 12 per head, causally masked
Cross scores:       12 x 24 per head

These are conceptual matrix sizes, excluding batch/head dimensions.

The source must be encoded, but its states can be reused throughout generation. Decoder caching can also reuse prior target keys and values. The output still depends on previous generated tokens, so separate source processing does not make autoregressive inference fully parallel.

T5: tasks expressed as text

T5 frames many tasks with text inputs and text targets. A task prefix can indicate translation, summarization, or another operation. Its pretraining includes span corruption, in which removed spans are represented with sentinel tokens and the target contains the missing content. A task prefix is useful only in the context of the checkpoint's training; inventing a new prefix does not guarantee a new skill.

BART: reconstructing corrupted input

BART combines a bidirectional encoder with an autoregressive decoder and learns to recover original text from a corrupted version. Its denoising setup differs from T5's design choices. Fine-tuned BART summarizers are examples of source-conditioned generation, but neither pretraining strategy guarantees faithful summaries of documents outside its training distribution.

Training and evaluating a source-target workflow

Prepare genuine pairs: source text and the desired output, with clear rules about what may be included. Keep duplicate documents and their near-identical summaries out of different dataset splits. Target padding should be ignored by the loss, and the decoder's shifted inputs must align with the supervised labels. Confirm whether the framework prepares those shifted inputs automatically.

For a support reply, evaluate whether facts are grounded in the source, whether required steps are mentioned, and whether unsupported promises are absent. For summarization, overlap scores such as ROUGE can help compare experiments but should be supplemented with factuality checks and human review. A high overlap score can coexist with a wrong name, date, or causal claim.

Choose this branch when separate source conditioning fits your workflow, not because “two stacks” sounds intrinsically better. Compare it against causal prompting and simpler extraction on the same inputs. Document token limits on both source and target, and handle truncation explicitly so omitted source evidence does not silently become an invented answer.

Sources: T5 documentation; BART documentation; original cross-attention architecture.

Educational material. Verify examples against current documentation and applicable licenses. Report a correction.