The attention mechanism, visualized
Attention is the idea that lets a model decide which other words in a sentence matter when it processes a given word. Instead of reading strictly left to right, the model looks at every token at once and assigns each pair a weight — how much this token should 'pay attention' to that one.
This single mechanism is what powers transformers, the architecture behind GPT, Claude, and essentially every modern large language model. Seeing the attention weights as a heatmap makes the abstract idea concrete.
Click a query token (a row)
"sat" attends most to
"cat" (44%)
Each row is a softmax — the query's attention split across all tokens, summing to 100%. Content words matter; filler doesn't.
Darker cell = stronger attention. This one mechanism, run in parallel across many "heads", is what replaced RNNs.
Free · runs entirely in your browser · nothing to install
How to use it
- Click any token to make it the query.
- Read the heatmap row: brighter cells are the tokens it attends to most.
- Try a token whose meaning depends on context and watch where the attention lands.
What you'll take away
- What 'attention weights' are and why they're a matrix.
- Why attention lets models handle long-range relationships in text.
- How this scales up into multi-head attention inside a transformer.
Want to actually build this?
This demo is one moment inside a full Math to Machine lesson — predict, build, and explain the concept, with an AI tutor that gives hints, not answers. The first five lessons are free.
FAQ
- What is the attention mechanism in AI?
- Attention is a way for a model to weigh how relevant every other token is to the one it is currently processing. These weights let the model pull in context from anywhere in the input, which is the core trick behind transformers.
- Why is attention important for LLMs?
- It lets the model relate distant words to each other in one step, capturing grammar and meaning that older sequential models struggled with. It's also highly parallelizable, which made training very large models practical.