Attention Mechanism

A technique in neural networks that allows the model to focus on the most relevant parts of the input when producing output. Self-attention enables transformers to weigh the importance of different words in a sentence relative to each other, dramatically improving language understanding.