Teknosys learning library

9. A numerical weight update and a backtracking example

Consider one weight with prediction y_hat = w * x and squared loss L = (y_hat - y)^2. With input 2, target 4, weight 1, and learning rate 0.1:

Prediction: 1 * 2 = 2
Loss: (2 - 4)^2 = 4
Gradient: dL/dw = 2 * (w*x - y) * x = -8
New weight: w - 0.1 * gradient = 1.8
New prediction: 1.8 * 2 = 3.6
New loss: (3.6 - 4)^2 = 0.16

This is a toy gradient-descent step, not a realistic full LLM. Real models update many parameters and often use optimizers such as AdamW.

In contrast, imagine filling a puzzle: save the board, try value A, detect a contradiction, restore the board, then try value B. That is backtracking; no neural-network gradient is involved. A line-search optimizer may also shrink an unsuccessful step, but that is a separate algorithm from backpropagation.

Educational material. Verify examples against current documentation and applicable licenses. Report a correction.