Teknosys learning library

The transformer encoder: building contextual representations

An encoder converts a visible input sequence into contextual token representations. Unlike a simple embedding lookup, its output for “charge” can change depending on whether the ticket discusses a battery or a credit-card payment. The tokenizer may split that word into multiple pieces, but the principle is the same: each output reflects the permitted context, not just the identity of one input token.

Bidirectional context is a masking choice

A standard text encoder allows an input token to attend to both earlier and later non-padding tokens. “The charge was reversed by the bank” provides useful evidence after the ambiguous word. This is often called bidirectional attention. It does not mean reading the sentence twice, and it does not grant access to information that was never in the supplied input.

In a batch of differently sized tickets, padding positions should be excluded as attention keys. The exact mask convention depends on the API: a one can mean “keep” in one interface, while a Boolean true can mean “block” in another. Consult the method you actually call instead of transferring a mask unchanged between libraries.

What happens inside a layer?

Attention builds weighted mixtures of value vectors from visible positions. Multiple heads offer different learned projections, and an output projection recombines them. A feed-forward network then transforms the feature vector at each position. Residual connections and normalization wrap these sublayers, with their ordering determined by the particular architecture.

Input hidden states:   batch x tokens x hidden_width
Self-attention:        information moves between token positions
Feed-forward network:  features change independently at each position
Output hidden states:  batch x tokens x hidden_width

After several layers, the model still returns a sequence of vectors. Those vectors are not automatically label probabilities or a ready-made search embedding. A downstream head supplies the task-specific interpretation: one label for the whole ticket, one label per token, or start/end scores for a span.

From encoder output to a useful feature

For ticket classification, a special summary token or masked pooling can provide a single vector. For search, a model trained for sentence similarity is usually a better starting point than arbitrary pooling of an untuned language encoder. A masked average should divide by the number of real tokens, not the padded length. Otherwise, adding padding changes the representation's scale.

Try this diagnostic: encode “bank loan” and “river bank” with the same encoder, then compare the hidden states associated with the relevant token pieces. Differences show contextualization, but they do not prove correct word-sense disambiguation on all examples. Evaluate the actual classifier or retrieval system on labeled, held-out data.

Limits that matter in deployment

The input window can truncate evidence near the end of a long ticket. Bidirectional context does not remove domain mismatch, ambiguous labels, or data leakage. Keep related tickets and duplicate templates out of separate training and test partitions, and assess performance on new customers or time periods where possible. A pretrained encoder is a starting representation, not a guarantee of a reliable business decision.

Sources: BERT documentation; encoder architecture in Attention Is All You Need.

Educational material. Verify examples against current documentation and applicable licenses. Report a correction.