Every word in a sentence can relate to every other word. Self-attention computes those relationships within a single sequence, letting each position gather information from all the others. It is the core operation inside the transformer.

The mechanism projects each token into a query, key, and value. Scores between queries and keys determine attention weights, which combine the values into a new representation. Multi-head attention runs several of these in parallel to capture different types of relationships.

Key properties

• Parallel computation across positions
• Direct paths between distant tokens
• Learned attention patterns
• Scalability with sequence length

Cost grows quadratically with sequence length, which limits how much text a model can process at once. Sparse attention, sliding windows, and linear approximations reduce the burden at some cost in expressiveness.