Attention, the move that made transformers
Every word asks a question; every word offers an answer. The match is meaning.
Before attention, a model read a sentence like a queue; after it, the sentence became a room where every word could turn and look at every other.
Take the sentence "the trophy didn't fit in the suitcase because it was too big." You know instantly that "it" is the trophy, not the suitcase. The model has no such intuition handed to it; it must compute the link, fresh, from the words alone. Attention is the mechanism that lets "it" reach back across the sentence, weigh both candidates, and bind to the right one. It is, quite literally, how meaning gets wired together inside a transformer.
Query, key, value
Each token is turned into three short vectors. The query is the question this token is asking; roughly, "what am I looking for?" The key is the label every token wears: "here is what I am." To decide how much one token should care about another, the model compares the first's query against the second's key; a strong alignment means a high score. Those raw scores are passed through softmax, which turns them into a tidy set of weights that sum to one, a distribution of attention spread across the sentence.
Then comes the value, the actual content each token offers up. A token's new representation is a weighted blend of everyone's values, mixed in proportion to those attention weights. It listens loudest to the tokens it scored highest and barely hears the rest. Do this for every token at once and the whole sentence rewrites itself in a single parallel step, each word now coloured by exactly the others that matter to it.
A query is a question, a key is an answer, a value is what gets passed along when they agree. Attention is just that handshake, run between every pair of words at once.
Why long context costs so much
The power and the price come from the same fact: every token compares itself against every other token. For n tokens that is n×n comparisons, n², so the work, and the memory to hold all those scores, grows with the square of the length. Going from a thousand tokens to ten thousand isn't ten times the work; it's closer to a hundred. This is the real reason long context windows are expensive, and why "just feed it the whole codebase" runs into a wall.
Almost every famous optimization is an assault on that square. Flash attention reorganizes the arithmetic so it never stores the full grid; sparse and sliding-window schemes let each token look at only a relevant neighbourhood; KV-caching avoids recomputing keys and values for tokens already seen. The headline trick of modern AI is simple enough to draw on a napkin: a handshake between every pair of words. The frontier is mostly the engineering it takes to afford that handshake at the length of a book.