Teknosys learning library

3. Transformer Anatomy: a guided chapter

Chapter 3 · Transformer Anatomy

A transformer is not a single model or a synonym for a chatbot. It is a family of neural-network designs that turn sequences of vectors into useful representations and predictions. Understanding the moving parts makes it easier to choose a model, debug a training run, and recognize what a product description does not tell you. This chapter follows one running example: a support team wants to classify incoming tickets and also draft short replies.

Start with the two different outputs

The ticket “My parcel arrived, but the charger is missing” needs a label such as missing-item, not an open-ended paragraph. A contextual encoder followed by a classifier is a natural candidate. Drafting a reply, however, means producing a sequence whose length is not fixed. That points toward a decoder-based generation model or an encoder-decoder system. Both applications can use attention, but their masks, heads, objectives, and evaluation are different.

We will first trace the shared machinery: token embeddings, attention, position-wise feed-forward networks, residual connections, normalization, and position information. We then attach a classification head, inspect a decoder, and compare the encoder-only, decoder-only, and encoder-decoder branches. The original 2017 transformer is our baseline; modern models often change its normalization, positional method, activations, and attention implementation.

Read the chapter in order or choose a topic

1. The architecture and tensor shapes
2. The encoder
3. Self-attention
4. The feed-forward layer
5. Layer normalization
6. Positional embeddings
7. A classification head
8. The decoder
9. Meet the transformers
10. The tree of life
11. The encoder branch
12. The decoder branch
13. The encoder-decoder branch

How to use the examples

Shapes use batch, sequence, and feature dimensions in that order. Small numerical examples are deliberately artificial: they explain an operation, not the behavior of a trained model. Python examples using only the standard library need no downloaded weights. Where a framework or checkpoint is discussed, verify its current documentation, license, and resource requirements before running it. No exercise requires sending real customer tickets to an external service.

By the end, you should be able to explain why a padding mask differs from a causal mask, why a classifier returns label logits instead of vocabulary logits, and why longer context or a larger parameter count alone does not guarantee better answers.

Sources and further reading: Vaswani et al., Attention Is All You Need; Hugging Face architecture guide. This chapter uses original explanations and examples rather than reproducing those publications.

Educational material. Verify examples against current documentation and applicable licenses. Report a correction.