Teknosys learning library

7. Transformer NLP: tokens, attention, and generation

A tokenizer splits text into token IDs. Embeddings and position information represent them; attention combines relevant context, and feed-forward layers refine representations.

An encoder can read both sides of a token, while an autoregressive decoder masks future tokens. An encoder-decoder model uses cross-attention to generate output conditioned on input. Generation selects tokens from predicted distributions using methods such as greedy decoding, sampling, or beam search.

Context windows and truncation limit what the model can see. Tokenization, chat templates, and decoding settings must match the checkpoint.

Educational material. Verify examples against current documentation and applicable licenses. Report a correction.