Attention Mechanisms
An attention mechanism is a neural network technique that lets a model weigh how relevant every part of its input is to every other part. Instead of treating all inputs equally, the model learns which words, image regions, or time steps to focus on when producing each part of its output.
For each element of the input, the model computes a score against every other element, turns those scores into weights, and builds a weighted combination of the inputs. Elements with higher weights have more influence on the result. Self-attention applies this within a single sequence, so each word or image patch is interpreted in the context of all the others.
In a vision AI model inspecting a product image, attention can link a scratch in one corner to the edge of the part it sits on, even when the two are far apart in the image. In a language model, attention connects a pronoun to the noun it refers to several sentences earlier.
Attention is a building block, not a complete model. Transformers are the architecture built mainly from attention layers, while older recurrent networks and convolutional networks can also add attention to improve how they handle long-range relationships.