Attention Is All You Need: the translation paper that invented modern AI

In June 2017, eight Google researchers published a fifteen-page paper about machine translation. It never mentions "artificial general intelligence". Today it sits among the most cited papers of the century, and the T in GPT comes from it. I read the whole thing to tell you what it says — no math required.

The problem was translation

It helps to start with what the paper was not. It wasn't a plan for chatbots or a manifesto about the future. "Attention Is All You Need" attacked a concrete, almost boring problem: translating English into German a little better and a lot cheaper than the systems of the day.

Those systems — recurrent networks — read the way we do: one word at a time, in order, carrying a summary of what came before. That design had two problems. First: in long sentences, the beginning fades; by word thirty, word two is a blurry memory. The second is less intuitive but proved decisive: if every word depends on the previous one, nothing can be processed in parallel. And GPUs — the chips that make all of this possible — are precisely machines for doing millions of things in parallel. The hardware was underused by design.

The idea: look at everything at once

The paper's proposal is quietly radical: remove sequential reading altogether. Instead of moving word by word, the model looks at the whole sentence at once, and for each word computes how much it should "attend" to every other one. That mechanism is called attention — hence the title: attention is all you need.

An example. In "the dog didn't cross the street because it was tired", what was tired? You know it's the dog, not the street. To know that, you connected "was tired" with "dog", skipping everything in between. That is exactly what attention learns to do: which words explain each other, regardless of the distance between them.

And the remarkable part is that nobody programs this: it emerges from training. No one taught the model grammar. It saw millions of sentences and figured out on its own which connections help translate well.

You can see it

My favorite part of the paper is in the appendix, and almost nobody mentions it: the authors drew what the model looks at. In this sentence, the verb "making" needs to complete itself with "more difficult", ten words away. The lines show attention connecting them over everything in between:

And in this one, the pronoun "its" points firmly at "Law" and "application": the model resolved what the pronoun refers to — one of the hardest things in translation.

Next time someone tells you these models "just predict the next word", this is what lives inside that prediction: a web of relations between every word in the context, learned without anyone writing it down.

The whole machine

The architecture that assembles all of this is called the Transformer, and the paper summarizes it in a diagram that became iconic. You don't need to understand every box — just the overall shape: two towers. The left one (the *encoder*) reads the source sentence and builds a rich representation of what it means. The right one (the *decoder*) writes the translation word by word, consulting that representation at every step.

Two design details are worth knowing:

1. Attention has eight heads. Instead of one attention mechanism, eight run in parallel, and each specializes on its own: one learns grammar relations, another tracks pronouns, another watches the neighboring word. Nobody assigned those roles.
2. Order is added, not assumed. Since the model sees everything at once, it lost track of which word comes first. The fix was to tag each position with a mathematical signal — the positional encoding: the model sees the whole sentence and also knows where each piece sits.

The result that changed the game

The Transformer didn't just translate better than everything published before. It did so training in a fraction of the time: 3.5 days on 8 GPUs for the big model, when the best previous systems needed an order of magnitude more compute. Same hardware — an architecture that finally uses it.

And that, more than translation, is the real legacy: the Transformer scales. Give it more data and more chips and it improves, predictably, with no ceiling in sight. Recurrent networks didn't scale that way; that's why they fell behind. Everything that came after — GPT, BERT, Claude, Gemini, image generators, AlphaFold in biology — is at heart the 2017 recipe with more zeros on the bill.

We are excited about the future of attention-based models and plan to apply them to other tasks: images, audio and video.

Last page of the paper, June 2017

They wrote that in 2017, and it read like wishful thinking. It was literal.

What happened to the eight

The epilogue is very tellable. The paper ranks seventh among the most cited papers of the 21st century according to Nature — above physics and medicine discoveries. And all eight authors left Google: today they found or lead Cohere, Sakana AI, Essential AI, Character.AI, NEAR — several are unicorns. Google published the key to the decade, for free, with explicit permission to reproduce its figures (which is why I could put them here), and then watched the people who forged it walk out the door.

There's a lesson in there about publishing open research, and a simpler one, which is what brought me to write this: primary sources can be read. This paper is fifteen pages. Half is math you can skip, and what remains is a clear idea told by the people who had it. Just like reading the building code instead of a summary of it: the original is almost always better than its reputation.