Attention Mechanism Demo
Interactive visual explainer of neural network attention
This is a simplified demonstration of self-attention. Real transformer attention patterns are learned and much more complex, with multiple heads and layers.
Input Sentence
Attention Patterns
Click a token to see what it attends to. Brighter colors = stronger attention.
Related Tools
Cosine Similarity Calculator
Calculate similarity between two vectors or text embeddings
LLM Temperature Simulator
Visualise how temperature and top-p sampling affect next-token probabilities
AI Image Prompt Enhancer
Expand simple ideas into detailed, descriptive prompts for better image generation
Technical AI
Deep dive into the technical details of AI
AI Pose Prompt Gallery
Browse and discover prompt keywords for character poses and camera angles
Darken Color
Make colors darker by reducing their brightness and luminosity
How to use
- Enter a short sentence or keep the sample text.
- Watch how attention weights highlight relationships between tokens.
- Click tokens to inspect which other tokens they attend to.
- Use the demo to build intuition before diving into real transformer math.
This is a simplified educational demo that runs entirely in your browser.
What is Attention?
Attention is the mechanism that allows transformer models to weigh the importance of different parts of the input when processing each token. It's the key innovation behind models like BERT, GPT, and modern LLMs.
Self-attention lets each token "look at" all other tokens and decide which ones are most relevant for understanding its context.
How Attention Works
Query, Key, Value
Each token creates three vectors: Query (what am I looking for?), Key (what do I contain?), Value (what information do I have?).
Compute Scores
Each token's Query is compared with all Keys to compute attention scores (usually via dot product).
Softmax
Scores are normalized to probabilities using softmax.
Weighted Sum
Values are weighted by attention scores and summed to produce the output representation.
Attention Patterns
Self-Attention
Tokens attend to themselves and nearby context.
Long-Range Dependencies
Important tokens can attend to distant relevant tokens.
Positional Patterns
Early layers often attend locally, later layers more globally.
Pro Tip: Multi-Head Attention
Real transformers use multiple attention "heads" that learn different patterns. GPT-3 has 96 heads per layer! This demo shows a simplified single-head view.
Frequently Asked Questions
Why is attention computationally expensive?
Standard attention has O(n²) complexity — every token attends to every other token. This is why context lengths were limited. Newer architectures use sparse attention or other tricks.
What is causal/masked attention?
In autoregressive models like GPT, tokens can only attend to previous tokens (not future ones). This is enforced with a triangular mask during training.
Related Tools
Tokenization Visualizer
See how text becomes tokens.
Embedding Visualizer
Explore token embeddings in 2D.