Not every word in a sentence matters equally. When translating the phrase bank account, the model needs to know that bank here means finance, not riverside. Attention lets a model weigh different parts of the input when producing each output, rather than treating everything with equal importance.

The mechanism computes scores between a query and a set of keys, converts those scores into weights, and uses them to combine values. The result is a weighted summary that emphasizes relevant information. Self-attention applies the same idea within a single sequence.

Common attention variants

• Scaled dot-product attention
• Multi-head attention with parallel views
• Self-attention within one sequence
• Cross-attention between two sequences

The transformer architecture built entirely on attention replaced recurrent networks for most language tasks. It parallelizes better and captures long-range dependencies more effectively, which is why nearly every large language model uses it.