A (fast) history of NLP

We Taught
Rocks to Talk

From Markov chains (1913) to autonomous agents (2026)

Flint Barrow Fractional CTO · Flintspark Labs AI/ML & NLP · Founder · Inventor

CONNECT ON LINKEDIN

Intro & Bio: Long-time coder, passionate about AI & NLP. Started in gaming, founded multiple companies.
LinkedIn Connection: Mention the QR code on screen and invite the room to connect on LinkedIn.
Literal Title: We taught silicon (rocks) to talk — from basic arithmetic to autonomous agents.
113-Year Progression: Walking through the historical timeline from 1913 Markov chains to 2026 Agents.
Bridge to Next Slide: Step back and establish the big-picture map of Artificial Intelligence.

Additional Context

Intro & Bio: Long-time coder, passionate about AI & NLP. Started in gaming, founded multiple companies.
Literal Title: We taught silicon (rocks) to talk — from basic arithmetic to autonomous agents.
113-Year Progression: Walking through the historical timeline from 1913 Markov chains to 2026 Agents.

The Big Picture

What is Artificial Intelligence?

AI covers Vision, Robotics, and more.
Machine Learning includes NLP.
LLMs are one small part.

Blueprint diagram: Venn diagram of Artificial Intelligence showing Computer Vision, Robotics, Expert Systems, Planning, Speech, Machine Learning, NLP, and LLMs

What is AI? What is NLP?

Language used to be a small corner of AI, and now it's the part everybody means.

AI is

software that does a job we would call thinking.

NLP is

the part of it aimed at human language.

The classic jobs

translationsentimentclassificationsearch summarisationquestion answeringspeech recognitionnamed entities parsingtaggingcoreferencetopic modelling

Once separate fields.
Whole careers spent on one.
Now, prompts for one model.

Define NLP: Software aimed at human language — reading, writing, translating.
12 Classic Tasks: Point out 2-3 examples (translation, sentiment, search) — each used to be a separate career.
arXiv Research Explosion: Highlight data: 1.6% of AI papers in 2015 vs 32% in 2025 (343× jump in papers).
Task Unification: Today's LLMs turn 12 separate research fields into 12 prompts on one model.
Bridge to Next Slide: To see how we got here, let's look at our 113-year historical roadmap.

Additional Context

[PACE — two definitions and a list. Slow, this is the ground floor.] Two words first, because I'm going to use both all night. AI is software doing a job we'd call thinking. NLP, natural language processing, is the part of it pointed at human language. Reading it, writing it, translating it. [Point at the grid, pick three. Don't read twelve.] And that's the shopping list. Translation. Sentiment, which is just asking whether this review is happy or angry. Classification, or categorisation, which is spam or not spam. Parsing is working out a sentence's grammar, tagging is labelling which word is the verb, coreference is working out who "he" refers to. Here's the thing. Ten years ago, if you said AI, you almost certainly meant machine learning on numbers. Recommendations, fraud scores, photos. Language was the poor cousin. [Point at the bars.] In 2015, out of every hundred AI papers, one and a half mentioned a language model. Last year it was a third of them. And look at the bottom line, because the pile itself went from under seven thousand papers to a hundred and thirteen thousand. [FACT GUARD — say "AI papers on arXiv", not "all AI research". And do NOT say language is the biggest field now: general machine learning and computer vision still publish more.] So the word flipped. AI used to be the big tent and NLP was one small corner of it. Now somebody says AI and they mean a thing you type a sentence into. [REVEAL] And every one of these was somebody's entire career. One dataset, one model, one leaderboard. Now they're twelve things you ask the same model to do. COMING UP: the roadmap, all twenty-three stops. Don't preview any of the history here.

What you're getting

A hundred and thirteen years, in twenty-three stops.

Where we start

A Russian mathematician counting letters in a novel by hand, because there isn't a computer anywhere on Earth yet.

Where we end

Software that reads your codebase, makes the change and runs the tests while you go get a coffee.

The one thing to hold

Almost none of this was invented from scratch. Count how often the new thing is somebody's old idea.

You don't need every detail.
Lost? Just hang on.
Big year on screen? We're starting new.

The Contract: 113 years of history organized across 5 major eras (1913 to 2026).
Whirlwind Pace: Set expectations for a quick-pass tour — giving overall context before diving into individual years.
Bridge to Next Slide: Lock in the key vocabulary terms before starting the historical timeline.

Additional Context

[PACE — this is the contract with the room. Unhurried.] So here's what you're getting. One walk, in order, 1913 to tonight — twenty-three stops. [Point at the two rows.] And look at the years, because this is the bit I love. The top row is twelve stops and it covers ninety years. The bottom row is eleven stops and it covers thirteen. Same length of line, same number of stops, and one of them is basically the last five minutes of history. We start with a man counting letters in a novel by hand, because there is no computer to do it for him. We finish with software that reads your codebase, makes the change and runs the tests while you go and get a coffee. And if you hold onto one thing tonight, make it this one. Almost nothing we're about to look at was invented from scratch. Nearly every breakthrough is somebody picking up an idea that had been lying around for decades, and finally having enough text, or enough compute, or a big enough sandbox to make it pay. So keep a count as we go, because by the end I want to show you the list. [REVEAL — say this properly, it matters.] Last thing. Some of this gets genuinely technical. There are three or four slides tonight where I'll show you the actual mathematics, and if you lose the thread there, that's completely fine and it isn't you. Everybody loses the thread somewhere. Here's your handrail: whenever a big year fills the screen, we've started something new, and you can climb straight back on. You never need the previous bit to understand the next one.

Words you'll hear tonight

Six words, and where each one turns up.

TERM IN PLAIN ENGLISH WHERE YOU'LL MEET IT NLP getting a computer to work with language the whole talk corpus the pile of text you learn from 1913 token one unit of text, historically a word 1913 language model anything that predicts the next word 1948 context window how much text it sees in one go 1948 embedding / vector numbers standing in for a word 2003

Reference Vocabulary: Point to key terms — Token, Context Window, Embedding, Language Model.
Tokens & Embeddings: Token = unit of text; Embedding = representing text as an array of numbers.
Bridge to Next Slide: Launch our story in 1913 Russia, three decades before electronic computers.

Additional Context

[PACE — don't read the table. Point at three rows, then move on. It's a reference slide, not a teaching slide.] Quick one before we start, because I'm going to use these words all night and I'd rather you had them up front than guessed at them. [Point, don't read.] Corpus is the pile of text you learn from. A token is one unit of it, basically a word. A language model is anything that predicts the next word, which sounds far too simple to be worth a name, and it turns out to be the whole talk. The bottom two are the ones you'll hear on the news. Context window is how much text the thing can look at in one go. An embedding, or a vector, is just numbers standing in for a word. Both of those take a while to earn, so don't worry if they mean nothing to you yet. [Point at the last column.] The useful part is the right-hand column. That's the year each word shows up in the walk, so if one goes past you and you want it back, that's the address. [DON'T make the n-gram and context-window connection here. That's 154's payoff, after the Transformer. Same for what an embedding is actually for.] COMING UP: 010, the first year card. 1913, Markov, counting letters by hand. Don't start the story here.

Rocks are just rocks.

1913

Andrey Markov works out the maths for things that depend on what came before.

Three decades before there's a machine to run it on.

Free Will & Independence: The big debate in 1913 Russia was about free will, independent events, and the Law of Large Numbers.
The Prevailing Belief: Mathematicians claimed the Law of Large Numbers required independent events — using that to argue for human free will.
Markov's Mission: Andrey Markov set out to disprove this by showing that dependent events also obey statistical laws.
Bridge to Next Slide: How Markov proved this by testing dependent events on a famous novel.

Additional Context

We start well before the computers. It's 1913, we're in Russia, and there's no machine anywhere in this story, because the first computer is thirty years away. Rocks are still just rocks. Andrey Markov is a mathematician, and he's working out how to handle sequences where each thing depends on the thing before it. Which sounds dry, right up until you notice it's the foundation of every prediction machine in this talk, including the one you used this morning. [Meanwhile card] Same year here: Henry Ford starts the first moving assembly line, which is worth noticing, because Ford is about to make everything in the world faster except this. [Beat, then move on — the demonstration is the next slide, don't give it away here.]

1913 · how he proved it

He counted 20,000 letters of a novel by hand.

Probability assumed independent events.
Markov proved laws survive dependence.
He tested on 20,000 letters.
He invented the Markov chain.

letters o n e g i n

marked V C V C V C

8,638 vowels · 11,362 consonants · every row sums to 1

Dependent, yet predictable.
Past changes future odds.

Nekrasov's Assertion: Pavel Nekrasov claimed that observing convergence to a fixed probability was mathematical proof of free will (independent choices).
Markov's Counter-Proof: Markov set out to prove that dependent events (where step N depends on step N-1) still show the Law of Large Numbers.
The Vowel & Consonant Test: Hand-counted 20,000 letters of Pushkin's Eugene Onegin, marking each letter as Vowel or Consonant.
The Proof: Discovered transition odds (0.13 V→V, 0.87 V→C, 0.66 C→V) settling at 43% vowels — proving dependent events obey statistical laws.
Bridge to Next Slide: How we use this approach as a simple application to experimentally determine dependent chains.

Additional Context

So how do you go about proving a thing like that in 1913? You need real data where each event genuinely depends on the one before it, and there are no datasets, because there's basically no data at all. So he uses a novel. He takes Pushkin's Eugene Onegin, strips out the spaces and the punctuation, and works through the first twenty thousand letters one at a time, marking every one of them vowel or consonant, by hand. Then he counts which kind follows which kind. [REVEAL] Twenty thousand letters, marked by hand, to win an argument about probability. [REVEAL table] And here's what he got. 8,638 vowels and 11,362 consonants. After a vowel the next letter is a vowel only about thirteen percent of the time, but after a consonant it's a vowel about two thirds of the time. [REVEAL] Which is exactly his point. Letters are not independent, and they are still perfectly predictable. Both halves of that matter. Now, be precise about what's actually new here, because it's easy to overclaim. He didn't invent probability, or its notation. His own teacher was Chebyshev; this is a well-developed field by 1913. What he invented is the object: a chain of events where each one depends on the last, governed by a fixed little table of odds, and — the genuinely hard part — provably still obeying the big theorems. Before him, "statistics" more or less assumed "independent events". After him it doesn't. That object gets his name, so let's have a look at what one of them actually is.

Markov chains · 1913

The states can be anything, not just letters.

Simulate weather with sunny and rainy.
Real weather clumps.

1 · States

S = { sunny, rainy }

2 · Transitions

P(rainy | sunny) = 0.2

3 · The Markov property

P(X_n+1 | X_n, X_n-1, …, X₁) = P(X_n+1 | X_n)

You need only the current state, never the whole history.

→ sunny→ rainy sunny0.80.2 rainy0.40.6

Every row sums to 1. That's the only rule.

Swap weather for words.
Same maths runs PageRank.

Experimental Application: Using sequence counts to experimentally determine hidden dependent chains in any system (weather, text, biology).
State Transition Graph: Representing states (weather, words) with arrows showing exact transition probabilities.
The Markov Property: Future state depends ONLY on current state — ignoring deep history to make computation tractable.
Foundation of NLP: This exact method of measuring dependent chains is the foundation for all n-grams, autocomplete, and language modeling.
Bridge to Next Slide: Fast-forward 35 years to Claude Shannon and Information Theory in 1948.

Additional Context

[PACE — slow down here. This is a load-bearing explanation and much of the room has English as a second language. Say the numbers slowly and pause after each reveal.] Here's why that letter-counting matters, and it's not really about letters. Markov's two states were "vowel" and "consonant". But nothing in the maths cares what the states are; they could be anything that follows anything at all. So let's make them something you can picture. Say you're building a game and you need weather. The lazy version is a coin flip every morning, and it looks immediately wrong, because what you get is sun, rain, sun, rain, sun. Real weather clumps, because rainy days come in runs. So instead of flipping a coin, you make tomorrow depend on today, and that's a Markov chain. It's three things and nothing else. One: a set of states, so sunny or rainy. Two: a probability for every move between them. If it's sunny there's an 80% chance tomorrow is sunny too, and a 20% chance it turns. That's the table, and every row sums to 1, which is the only rule it has to obey. [REVEAL property] Three, and this is the one that matters: to predict tomorrow you only need today. Not last week, not last year. Formally, the probability of the next state given the entire history equals the probability of the next state given just the current one, because the past is already baked into the present. That's the Markov property. It's also a lie, but a really useful one. [If asked what "hidden" adds: you can't see the state, only something it emits, so you don't see "sunny", you see whether someone carried an umbrella. Park it; we come back to hidden Markov models when we get to speech recognition in the eighties.] [REVEAL bridge] Swap the weather for words and you've got a language model — which is exactly what the nineties did. And this isn't niche maths, it's load-bearing infrastructure. Google's original PageRank is a Markov chain over web pages: imagine someone clicking links at random forever, and rank pages by where they end up. Your phone's predictive text. Gene-finding walks along a DNA sequence deciding which region you're in. And Markov chain Monte Carlo is how a huge share of modern statistics and physics gets computed at all, because it's how you sample a distribution you can't solve directly. [Cut-if-tight aside: Snakes and Ladders is also a pure Markov chain (your next square depends only on the square you're on), which is precisely why the game plays itself and no child has ever made a decision in it.] And note the date on this one, because it's the oldest thing in the talk: Markov publishes the chains in 1906, and in 1913 he demonstrates them on TEXT, by hand, counting vowels and consonants through the first twenty thousand letters of Pushkin's Eugene Onegin, to prove that events which depend on each other still obey statistical laws. No computer, no corpus, just a mathematician with a novel and an enormous amount of patience. The first language model is 1913.

Do rocks write?

1948

Claude Shannon generates English by picking likely next words.

In the paper that invents information theory.

Raise your hand

Who’s happy with autocomplete?

Claude Shannon: 1948 paper A Mathematical Theory of Communication at Bell Labs.
Information & Entropy: Quantified information in bits and modeled language as statistical processes.
Bridge to Next Slide: Scale Markov letter counting up to word sequences: N-grams.

Additional Context

1948. Claude Shannon publishes the paper that invents information theory, which is the reason we say "bit", and buried inside it there's a party trick. He builds sentences by choosing each word based only on the word before it, using a book and a table of frequencies. What comes out reads like this: "the head and in frontal attack on an English writer that the character of this point is therefore another method." Total nonsense, with perfect grammar and excellent manners. Which tells you something genuinely important, and it's that fluency is cheap. You can fake the SHAPE of language with counting alone. Hold that thought for the rest of the night. [REVEAL — hands. Count them, react, answer it yourself. HANDS ONLY, don't call on anyone.] Who's HAPPY with autocomplete? — [almost no hands; that's the joke, let it sit] "Two people. Two. And I don't trust either of you." Then: "Right, what's annoying you is a direct descendant of Shannon's party trick, and on the next slide I'll show you exactly why it does that."

N-grams · Shannon 1948 · predicting the next word

N words in a row, counted.

the cat sat on the ____

Raise your hand

Who just thought “mat”?

No parsing required.
Just seen them together.

Making one

…and the cat sat on the mat and…

Slide the window across billions of words of newswire, parliamentary records and eventually the web, tallying every sequence as you go. That's the training.

Blueprint: a window of three context cells feeding a NEXT cell, with a bar chart of candidate words and a probability of 0.112

P(mat | the cat sat on the) ≈ P(mat | on the)

Keep the last n words, which is your context window, and throw the rest away.

P(mat | on the) = count(on the mat) count(on the)

One count divided by another, and no grammar anywhere.

n usually stopped at five.
Modern context windows are huge.

Standard Autocomplete: We've all seen this — predicting the next word based on previous counts.
Markov Connection: Same math as 1913 Markov chains, but now each node is a sequence predicting the next word.
Context Window & Sparsity: The 'N' in N-gram is the context window (historically capped at ~5 words due to data explosion).
Bridge to Next Slide: Move to 1950 and Alan Turing's foundational question.

Additional Context

[PAYOFF — pays off 012-markov-chains. An n-gram model IS a Markov chain over words; say so.] [PACE — slow down here. This is a load-bearing explanation and much of the room has English as a second language. Say the numbers slowly and pause after each reveal.] This idea is older than almost everything else tonight. Claude Shannon's 1948 paper, the one that invents information theory, has a section where he generates English by picking each word based on the last one or two, and it reads like nonsense with excellent manners. By the eighties it's the language-model half of every speech recogniser. And here's the method, which you already run. [Read the line, leave the blank hanging.] "The cat sat on the…" [pause] [Hands — count them, react, move on. HANDS ONLY.] Who just thought "mat"? — "Almost everyone. And

Do rocks think?

1950

Alan Turing asks, in print, “Can machines think?”

Seventy-six years ago.

Raise your hand

Who already knows what the Turing Test is?

Alan Turing: 1950 paper Computing Machinery and Intelligence.
Reframing the Question: Replaced "Can machines think?" with an operational test.
Bridge to Next Slide: Examine the exact mechanics of the Imitation Game.

Additional Context

Set the dial to 1950. [Beat — let 1950 land. It's older than they expect.] Alan Turing publishes a paper in the journal Mind called "Computing Machinery and Intelligence." First line, essentially: I propose to consider the question, can machines think. And then he goes further than asking, because he proposes a way to settle it. [REVEAL — then pause and actually look at the room.] Raise your hand: who already knows what the Turing Test is? — Count them out loud ("okay, six, eight, about a third of the room"). React to the count, then answer it yourself: "so most of you have heard of it, and here's the part that usually gets left out." HANDS ONLY — do not call on anyone or ask anyone to explain it. Works either way: lots of hands → "then you already know the famous version, and I want to show you what he actually wrote." No hands → "perfect, then this next part is going to be fun."

“Computing Machinery and Intelligence” · Mind, October 1950

He asked the question, then quietly changed it.

“Can machines think?”
Hard to define thinking.

“…too meaningless to deserve discussion.”

Turing, on his own question

Testable substitute: tell them apart?
Called the imitation game.

Alan Turing, photographed in 1951 — **Alan Turing, 29 March 1951.** Elliott & Fry, London · public domain

The Problem: Defining "thinking" is a philosophical trap.
Operational Approach: Test behavior rather than internal consciousness.
Bridge to Next Slide: Look at the setup and scoring bar Turing proposed.

Additional Context

October 1950, in a journal called Mind, which is a philosophy journal and not an engineering one. The paper is "Computing Machinery and Intelligence," and it opens: "I propose to consider the question, can machines think?" Then he spends the rest of the paper refusing to answer it, because to answer it you'd first have to define "thinking", and nobody can. Ask ten people in this room and you'll get ten different answers. He says so himself, in the same paper: too meaningless to deserve discussion. [REVEAL] So he swaps it out for a question you can actually test. Not "does it think", but could an examiner tell it apart from a person, in writing. [REVEAL] He called it the imitation game, and the setup is a good deal stranger than the version you've heard. That's the next slide.

The imitation game

A judge, two hidden players and a screen.

1A judge questions two hidden players, A and B.
2Everything goes through a teleprinter, so typed words only.
3The judge says which is which.
4Now swap one player for a machine.

Fool the judge 30%.
Expected by 2000.

Blueprint: a judge's terminal connected through a screen to two player terminals, A and B

Blueprint of a teleprinter on a stand, paper feeding out of the platen

The Imitation Game: Human judge converses via teletype with a human and a computer.
Turing's Bar: Fool the judge 30% of the time after 5 minutes of conversation by year 2000.
Bridge to Next Slide: 1954 Georgetown-IBM experiment & Bag-of-Words.

Additional Context

Here's the setup, and it's stranger than the version you've heard. A judge sits alone with a keyboard, and there are two hidden players, call them A and B. The judge can ask either of them anything at all, but every word goes through a teleprinter [gesture right], so no voices, no faces and no handwriting, just text. The judge's job is to work out which is which. Now here's the detail everyone leaves out. In the actual paper, the two players are a man and a woman, and the judge is trying to tell the man from the woman. It's a Victorian parlour game. [REVEAL 4] Turing's move is to take one of them out and put a machine in that seat, and then ask: if the judge can't do any better than before, what exactly is the difference? [REVEAL] And he gave a number, which matters for later. Fool the judge 30% of the time over five minutes, and he guessed a machine would manage it by the year 2000. Remember that 30%. We'll come back to it, and by the end of tonight it's going to look quaint.

Do rocks sort?

1954

Zellig Harris: you can tell what a word means by the company it keeps.

Meaning, treated as a statistic.

1954 Georgetown-IBM Demo: First public machine translation demonstration (Russian to English).
Early Hype: Press claimed machine translation would be solved in 3 to 5 years.
Bridge to Next Slide: Look at the underlying technique: Bag of Words.

Additional Context

1954, and a linguist called Zellig Harris writes something that turns out to be load-bearing for everything after it. His claim: words that show up in the same contexts tend to mean similar things. You don't need a definition of "dog" — you need to see what words appear around it. That's called the distributional hypothesis, and two things grow out of it. The immediate one is what we're about to look at, which is treating a document as nothing but a pile of words. The other one takes another sixty years, and we'll get there, because it's how machines eventually learned meaning.

Bag of words · 1954 · sorting documents

You can spot spam without reading a word.

Message becomes a word tally.
Spam words mean bin.
Answers “what document type?”
Cost: Word order lost.

10,000 spam emails

free4,102 winner2,890 click2,751 offer1,806 meeting12

10,000 real emails

free61 winner9 click104 offer88 meeting3,806

new “Congratulations — you are a winner! Click to claim your free gift.”

Three words, and it already looks far more like the left pile than the right. No reading required.

Bag of Words (Zellig Harris): Treat text as an unordered collection of word frequencies.
Distributional Hypothesis: "Words that occur in similar contexts have similar meanings."
Bridge to Next Slide: The 1966 Machine Translation hype crash and ALPAC report.

Additional Context

While the labs are arguing about grammar, librarians have a completely different problem. Theirs isn't "what does this mean", it's "which of these ten thousand documents did I want", and they solve it by counting. The phrase is Harris's, 1954. Luhn at IBM is doing it on real documents by '58, and by the mid-seventies it's simply how information retrieval works. I'll use a modern version of the same job, because you all rely on it every single day: your email somehow knows what spam is. It doesn't read the message. It tips the words out into a bag, gives it a shake and counts them. [Point at the diagram.] "The" twice, "cat" once. The email is now just that list of numbers, and if the list has enough "free" and "winner" and "click here" in it, the thing goes in the bin. That's the whole classifier. No grammar, no meaning, nothing that understands a word of it. [REVEAL] And the same move answers any question shaped like "what kind of document is this?" Sort news into topics, route a support ticket to the right team, rank a search result. It's all counting words and comparing tallies. [REVEAL] Now the cost, and it's a big one, because the order is gone. "The cat sat on the mat" and "the mat sat on the cat" contain identical words in identical numbers, so as far as this representation is concerned they're the same document. One of those is a normal Tuesday and the other one is a cry for help, and the computer genuinely cannot tell them apart. So: crude, obviously wrong about language, and it has been running document search since before most of this room was born. Counting beat understanding, and honestly it wasn't close. [Bridge] But there's a problem hiding inside the counting, and I bet you can guess it. What's the most common word in every document ever written?

1954 → 1966 · machine translation

Declared nearly solved, then defunded.

1954: IBM 701 translates Russian.
250 words, six grammar rules.
IBM: “an unqualified success.”

“Five, perhaps three years hence … may well be an accomplished fact.”

Leon Dostert, Georgetown, at the demonstration

1966: Government reviews the progress.
Finds “no immediate prospect.”
Funding collapses. Language research dies.

The specimen punched card from the 1954 demonstration: transliterated Russian across the top, and a printed strip beneath reading THE QUALITY OF COAL IS DETERMINED BY CALORY CONTENT — **The actual card.** Russian punched across the top, *KACHYESTVO UGLYA…*, with the English printed on the strip below.

An operator punching Russian sentences onto cards at the 1954 Georgetown-IBM demonstration — Punched in.

The English translation emerging on the line printer at the 1954 demonstration — Printed out.

IBM press materials, 7 January 1954

1966 ALPAC Report: Government audit concluded machine translation was slower, less accurate, and more expensive than humans.
First AI Winter: Funding for machine translation crashed overnight.
Bridge to Next Slide: Shift from translation to conversational pattern-matching: ELIZA.

Additional Context

Around this same time, machine translation is the big push, because it's the Cold War and everybody wants Russian read fast. January 1954, IBM and Georgetown put on a demo in New York, where an IBM 701 translates Russian sentences into English in front of the press. [Point at the photos.] The Russian gets punched onto cards by an operator who didn't speak a word of it, and English comes off the line printer seconds later. IBM's own press release calls it "an unqualified success" and brags about "the breakneck speed of two and a half lines per second." Here's what it actually was. 250 words of vocabulary and six grammar rules. The showpiece sentence was about coal: "the quality of coal is determined by calory content." So, carefully chosen sentences, a tiny dictionary, and absolutely enormous headlines. Georgetown's Leon Dostert tells reporters it'll be a solved problem in, and I love the ordering here, "five, perhaps three years." [REVEAL] It wasn't. Twelve years of funding later, in 1966, a committee for the National Academy of Sciences reports back that there is "no immediate or predictable prospect of useful machine translation." That's the actual sentence, and the money stopped more or less overnight. [REVEAL] So 1966 goes down as the year a committee declared machine translation a dead end. Hold onto that year, because in those same twelve months, over at MIT, somebody was building the most convincing talking program anyone had ever seen, and it had nothing to do with translation at all. That's next. [Do NOT tell the "vodka is good but the meat is rotten" story — it's almost certainly made up.]

Do rocks listen?

1966

Joseph Weizenbaum, at MIT, publishes ELIZA.

The first convincing chatbot.

Raise your hand

Who thought the first chatbot was younger than you?

Keep it up if you thought it was younger than your parents.

1966 ELIZA: Created by Joseph Weizenbaum at MIT.
First Chatbot: Simulated a Rogerian psychotherapist.
Bridge to Next Slide: Weizenbaum's motivation and human reaction.

Additional Context

Same year. 1966 is the year the committee said language was hopeless, and it's also the year this happens. [Beat.] The first chatbot is older than the moon landing, because Apollo 11 is three years after this. It's older than colour TV in most homes, and it's older than very nearly everyone in this room. [REVEAL — hands. HANDS ONLY, never call on anyone.] Who thought the first chatbot was younger than you? — [most of the room; count it out loud] "Almost everyone." [REVEAL second line] Now keep your hand up if you thought it was younger than your PARENTS. — [most hands stay up] "Right. It is older than nearly every parent in this room." Same year: England win the World Cup, and Star Trek premieres with a computer the crew talks to. One of those three aged strangely.

“ELIZA” · Communications of the ACM, January 1966

He wrote it to show how little it took.

Script called DOCTOR (therapist).
Knows nothing, echoes your words.
People confided in it anyway.
His secretary asked him to leave.
He later argued against it.

**Joseph Weizenbaum**, 2005 · Ulrich Hansen · CC BY-SA 3.0 · via Wikimedia Commons

The Doctor Persona: Weizenbaum chose psychotherapy because it allowed asking open questions without needing real-world knowledge.
The ELIZA Effect: Users confided deeply in the bot despite knowing it was a simple script.
Bridge to Next Slide: Look at an actual transcript from 1966.

Additional Context

Joseph Weizenbaum, MIT, January 1966, Communications of the ACM. And note what he says he's doing: studying natural language communication between man and machine. He isn't trying to win Turing's game, he's trying to show how little machinery it takes to LOOK like you're playing it. The script is called DOCTOR, and it imitates a Rogerian therapist, which is the cleverest thing here. A Rogerian therapist is allowed to know nothing at all about your life; they just reflect your own words back at you. So the program's total ignorance stops being a bug and becomes the character. [REVEAL] And then the thing he genuinely didn't expect. People opened up to it. His own secretary, who had watched him write the thing and knew perfectly well there was nothing in there, asked him to leave the room so she could talk to it privately. [REVEAL] That broke something in him. He spent the rest of his career, and a whole book in 1976, arguing that we should not do this. We went ahead and did it anyway, about forty more times, on purpose.

A real conversation · Weizenbaum, January 1966

her Men are all alike.

ELIZA IN WHAT WAY

her They're always bugging us about something or other.

ELIZA CAN YOU THINK OF A SPECIFIC EXAMPLE

her Well, my boyfriend made me come here.

ELIZA YOUR BOYFRIEND MADE YOU COME HERE

her He says I'm depressed much of the time.

ELIZA I AM SORRY TO HEAR YOU ARE DEPRESSED

her It's true. I am unhappy.

ELIZA DO YOU THINK COMING HERE WILL HELP YOU NOT TO BE UNHAPPY

her I need some help, that much seems certain.

ELIZA WHAT WOULD IT MEAN TO YOU IF YOU GOT SOME HELP

her Perhaps I could learn to get along with my mother.

ELIZA TELL ME MORE ABOUT YOUR FAMILY

People queued to use it. It was copied onto home computers for the next twenty years, and practising psychiatrists proposed developing it into real therapy. Reading understanding into a machine that has none is still called the ELIZA effect.

The Dialogue: "Men are all alike" -> "IN WHAT WAY" -> "They're always bugging us" -> "CAN YOU THINK OF A SPECIFIC EXAMPLE".
Fluency vs Evidence: Fluency felt like talking to a person, but there was zero understanding inside.
Bridge to Next Slide: Peek under the hood at ELIZA's pattern matching.

Additional Context

Read a few of these out loud, her lines in a normal voice and ELIZA's flat. Don't editorialise while you read; let it do the work. Two things to point out. It's ALL CAPS, and there's no punctuation anywhere, not one question mark in the whole thing. That isn't a transcription slip, it's what the machine actually printed on a teleprinter in 1966. [REVEAL last line] "Tell me more about your family." That's the line that got it. Now, how loved was this thing? Nobody treated it as a toy. They queued for it, MIT students burned hours on it, and it got copied and re-copied for twenty years — rewritten in BASIC, typed out of magazines, shipped on home computers. That's a longer run than most software ever gets. Practising psychiatrists wrote seriously about using this as a therapeutic tool, and that's the part that really horrified Weizenbaum. Not that people were fooled, but that professionals wanted to PRESCRIBE it. He said he was startled by how quickly and how deeply people got emotionally involved with it. And the effect got named after the program — "the ELIZA effect" is still the term for reading understanding into a machine that has none. We'll be using that term again tonight.

How it works

Two kinds of rule, and neither one reads.

you I hate my job.

ELIZA WHY DO YOU HATE YOUR JOB

you …get along with my mother.

ELIZA TELL ME MORE ABOUT YOUR FAMILY

Reflect it: Swap pronouns, echo.
Match word: Fire stock line.
Recognised nothing? PLEASE GO ON.

Loop diagram: INPUT to PATTERN to SWAP to OUTPUT and back to INPUT, with a separate dashed box labelled MEANING that nothing connects to

MEMORY

Stores your “my …” sentences. When it recognises nothing, it replays one: EARLIER YOU SAID YOUR …

RANK

Keywords carry a priority. In “everybody laughed at me”, everybody outranks me.

ROTATE

Several replies per rule, used in turn — so it never answers the same shape twice running.

TIDY

don't → do not · you are → I am, so the sentence comes back grammatical.

Every one of them is a lookup. Nothing consults the dashed box.

Pattern Matching Mechanics: Keyword substitution, pronoun swapping (my -> your), and fallback canned responses.
No Machine Learning: 100% hand-typed rules by a human programmer.
Bridge to Next Slide: 1970 SHRDLU & natural language in a simulated world.

Additional Context

So how does it do that? There are two kinds of rule in there, and neither one of them involves reading. The first kind is reflection. Take the sentence apart, swap the pronouns around so my becomes your and I becomes you, then reassemble the whole thing as a question. "I hate my job" comes back at you as "why do you hate your job." The second kind is the one that catches people out, and it's pure keyword lookup. In the script, "mother" and "father" are both tagged into a family group, and any word in that group fires the same stock line: tell me more about your family. Nothing gets echoed back at all. It did not work out that mother implies family; a person typed that connection in by hand, years in advance. And when it recognises nothing at all, it stalls. "Please go on." "What does that suggest to you." [If pressed, there IS more machinery: keywords are ranked so the strongest wins, each keyword has several alternative replies so it doesn't repeat itself, and there's a memory stack that can bring back something you said earlier — "earlier you said your…". Genuinely clever engineering. Still no understanding anywhere in it.] [REVEAL — point at the dashed box.] That's the whole point. Every rule is a lookup. There is no step where the program works out what you meant. Find-and-replace in a trench coat. [LIVE DEMO — 2 minutes, you at the keyboard. Let the room feel it working before you point out that it isn't. Close with: "you all knew it was a trick, and it still felt like something."]

Do rocks understand?

1970

Terry Winograd builds a program that genuinely understands you.

Inside a world containing seven blocks.

SHRDLU · 1970

Language in, actions out, all inside a sandbox.

you Pick up a big red block.

SHRDLU OK.

you Grasp the pyramid.

SHRDLU I DON'T UNDERSTAND WHICH PYRAMID YOU MEAN.

Plans moves, changes the world.
That's tool use in 2026.

Every fact typed by hand.

Blueprint: a bounded box containing a block, a pyramid and a box, dimensioned and labelled KNOWN — The sandbox, meaning everything SHRDLU could see, name or act on.

Language in, Action Out: "Pick up a green block and put it in the box."
Full Loop: Parse sentence, reason about sandbox constraints, execute action, explain reasoning.
Bridge to Next Slide: Look at syntax parsing: Augmented Transition Networks.

Additional Context

Here's what it could do. You type "pick up a big red block" and it does it. You say "put it in the box" and it works out what "it" means. Ask why it moved something and it tells you. In 1970. And look at the SHAPE of that, because this isn't a chatbot. It isn't answering questions at all. It takes an instruction in English, plans a sequence of actions, changes the state of a world, and reports back to you. [REVEAL] That is tool use. That's an agent. It's exactly the loop every AI agent runs in 2026: instruction in, actions taken inside a sandboxed environment, result handed back. We have been doing this for fifty-five years, and the only thing that really changed is the size of the sandbox. SHRDLU's held seven blocks and a box; the one your coding agent runs in holds your entire filesystem. [REVEAL — point at the diagram.] And here's the catch that ends the whole rules era. Every single fact inside that sandbox was typed in by a person, by hand. One inch outside it and there's nothing at all: no weather, no feelings, no second kind of object. To scale this up, somebody would have to sit down and type in the entire world, and nobody can do that. [Bridge — note this hands off to ATN/LUNAR now, NOT straight to the counting era. The "gave up and started counting" line has moved to the tagging scoreboard three slides from here, which is where it actually earns itself.] Hold that thought, because down the hall from Winograd somebody is working on the narrower, more durable version of this problem.

Augmented transition networks · 1970

A machine that walks through your sentence.

Grammar as networks of states.
Augmented means memory and calls.

Phrases containing phrases: recursion can.

Woods, Communications of the ACM, October 1970. POP hands control back to whoever called you — and both NP arcs call the same network.

ATNs (Augmented Transition Networks): Recursive state graphs for parsing grammatical structure.
Hand-Crafted Rules: Complex rule trees to parse nouns, verbs, and prepositional phrases.
Bridge to Next Slide: 1971 Part-of-Speech tagging.

Additional Context

[PACE — slow down here. This is a load-bearing explanation and much of the room has English as a second language. Point at each network before you walk it.] Same era, different building. Bill Woods at BBN (that's Bolt Beranek and Newman, the company that also built the first ARPANET routers) asks a much narrower question than Winograd does. Not "can it understand", just: how do you take a sentence apart, mechanically? His answer is a picture, and the important thing is that it's TWO pictures. [Point at the top one.] That's the sentence network. And look at the first arc — to get from S to S1 you don't need a word, you need a whole noun phrase. Then a verb. Then another noun phrase. Then POP, which means you're done. [Point at the bottom one.] And that's the noun-phrase network. An article or an adjective, then as many more adjectives as you fancy — that's the little loop on top — then a noun, then POP. [REVEAL nothing yet — point at the dashed arrow.] And this arrow is the whole reason we're looking at this. When the sentence network hits its NP arc, it CALLS the noun-phrase network. Like a subroutine. So let's walk "the big dog chased the cat." Top network needs a noun phrase, so it calls the bottom network. The. Big. Dog. POP, and we're back up top at S1. Now we need a verb — chased, that's S2. Now another noun phrase, so we call the bottom network again. The. Cat. POP. POP. Parsed. [REVEAL] Now, why "augmented". That call is one of the two additions. The other is registers, which is scratch memory: it can write down what it found, that this was the subject, and it can BACK UP when a guess turns out wrong. Woods's own phrase for that is postponing decisions until more is known about the sentence. [REVEAL] And here's why the call matters so much. A network can call any network, including itself. Take "the cat that the dog chased sat." There's a whole sentence buried inside that sentence. A flat state machine, one that only walks forward, can never do that. Recursion can. So this isn't a toy: ATNs are how parsing works for the next twenty years, and they end up inside the ARPA speech programme. [Bridge — this is the setup for the next block, say it deliberately] But look at the arc labels again. ART. ADJ. N. VERB. To take an arc at all, the machine has to already know what kind of word it's holding. Woods assumed that was a solved problem. It was not solved, and it wasn't even close.

Do rocks know grammar?

1971

Somebody tries to label every word in a million-word corpus — automatically.

77% automated accuracy; the remaining 23% corrected by hand.

Part-of-speech tagging

What kind of word is this?

Tag every word in a sentence.
What it's doing here.

Raise your hand

Is book a noun?

Only the sentence can tell.
A tagger reads the neighbours.

I read a book.

book → NOUN · a thing you hold

Book me a flight.

book → VERB · an instruction

Same four letters, and nothing inside the word decides which one it is.

Read the Neighbors: Disambiguate word roles by inspecting adjacent words.
Rule-based vs Statistical: Transition from manual rules to probabilistic taggers.
Bridge to Next Slide: Scoreboard progression from 77% to 97% accuracy.

Additional Context

[PACE — slow down here. This is a load-bearing explanation and much of the room has English as a second language. Read both sentences out loud.] So let's do the boring thing properly, because everything upstream needs it. Part-of-speech tagging. For every single word in a sentence, decide what kind of word it is. Noun, verb, adjective. And note the emphasis, because it isn't what the word means in the dictionary, it's what the word is DOING, right here, in this sentence. Which sounds like homework, so here's why it isn't. [Read the first sentence.] "I read a book." [Read the second.] "Book me a flight." [REVEAL hands. HANDS ONLY — and this one has a trap in it, which is the point.] Hands up — is "book" a noun? [Count. React to whatever you get: "About half of you" / "Most of you" / "Nobody's committing, very sensible."] [REVEAL] It's both, and that's the whole lesson. There is nothing inside the word at all, not the letters, not the spelling, that tells you which one it is. The only evidence you get is the company it keeps. In "a book", coming after "a", it's a thing. In "Book me", sitting at the front with "me" after it, it's an instruction. [Optional, if the room is with you: "read" in that first sentence is doing the same trick to you right now. Present tense or past tense? You cannot tell from the letters either. English is full of this.] So the machine has to look at the neighbours. Which sounds easy and is the thing the field spends the next twenty years on. Next slide, the scoreboard.

Tagging, 1971 → 1994

Twenty-three years to read a word.

1971: Hand-written rules, 77%.
1987: Statistical context, 96–97%.

Then it stops near 97%.
The task has a ceiling.

The same numbers as errors: 1 word in 4 wrong → 1 in 29 → 1 in 36.

Accuracy Scoreboard: TAGGIT (77%) -> CLAWS (97%).
Twenty Years of Arguing: Rule-based taggers plateaued; statistical taggers broke through.
Bridge to Next Slide: 1972 TF-IDF & document search.

Additional Context

[PACE — slow down here. This is a load-bearing explanation and much of the room has English as a second language. Say each percentage slowly.] Here's the scoreboard, and it's the clearest picture in the talk of how this field actually moves. 1971, TAGGIT, by Greene and Rubin. Hand-written rules: look at the ending of the word, look at what's sitting next to it, apply the rule. Seventy-seven percent. [Point at the first bar.] And I want you to sit for a second with what the other twenty-three percent actually means. Linguists went through a million words and fixed them by hand. That took years. Human beings, word by word, for years. [REVEAL] 1987, Lancaster, CLAWS. And they change the question, because instead of arguing about what the rule ought to be, they just count what actually happens. Which tag tends to follow which tag. Ninety-six to ninety-seven percent. In error terms, because that's where you feel it: from about one word in four wrong, to one in twenty-nine. [Point at the caption.] [REVEAL] And then it stops. Look at the third bar: 1994, Brill, ninety-seven point two. It barely moves, and that's because ninety-seven percent or so is roughly where two human experts stop agreeing with each other about the right tag. The task has a ceiling and the machines have hit it. [Precision guard: say "roughly" on the human-agreement number. It's the commonly cited figure for part-of-speech agreement, not something I'd defend to a decimal place.] [The nuance, if anyone asks — and be honest about this rather than telling the clean story: Brill's tagger is RULE-based, and it beats CLAWS. So this is not simply rules losing to statistics. What died was HAND-WRITTEN rules. Brill's rules are learned — the machine finds its own errors and works out the rule that fixes them. Rules written by people lost. Rules found by counting won.] [Bridge] And that's the lesson the whole field is about to take, everywhere, all at once. If you can count it, don't argue about it. Hold onto that, because next we're going to count words in documents, and it ends up running the internet.

Do rocks find?

1972

Karen Spärck Jones works out which words actually matter.

Twenty years before there's a web to search.

TF-IDF · term frequency × inverse document frequency

The commonest words tell you almost nothing.

Common words make documents identical.
Discount words that are everywhere.

weight = times it appears here how many documents it appears in at all

Frequent here and rare everywhere else → informative. In every document → worth nothing.

1972: Decades before the web.
Powers every modern search engine.

**Karen Spärck Jones** · Cambridge · she published the idea in **1972** · Markus Kuhn · CC BY 2.5

Term Frequency × Inverse Document Frequency: Frequent words in a document that are rare overall carry high weight.
Search Foundation: Standard search algorithm powering databases for 50+ years (BM25 ancestor).
Bridge to Next Slide: 1980s Hidden Markov Models for speech.

Additional Context

So: what's the most common word in every document ever written? "The." Followed by "of", "and", "a". Which means if you rank by raw count, every document on earth looks identical and the counting was pointless. [REVEAL maths] The fix is one line, and honestly it's beautiful. Take how often a word appears in THIS document, and divide it by how many documents it appears in at all. "The" appears in everything, so the bottom of that fraction is enormous and its weight collapses to nothing. But "banana" appears in almost nothing, so when it does show up it counts for a lot. Frequent here and rare everywhere else means informative. That's the whole idea: term frequency over document frequency, TF-IDF. [REVEAL Spärck Jones] And note the date, 1972. Karen Spärck Jones, at Cambridge, right in the middle of the quiet period, twenty years before there was a web to search. Every search engine you have ever used is built on top of this, and she spent most of her career under-credited for it. She also said something worth repeating: computing is too important to be left to men. So by the mid-seventies the toolkit exists. Count the words, weight them properly. What's missing isn't the maths, it's the text, and that's what the nineties turn up with.

Do rocks hear?

1980s

Speech recognition stops trying to understand anything.

Hidden Markov Models replace rule-based acoustic parsing.

Hidden Markov models · speech

Ask what's most likely, not what it means.

Hidden words, observed noise.
Guess the most likely sequence.

No grammar, still beat rules.
Why your phone dictates.

“Every time I fire a linguist, the performance of the speech recogniser goes up.”

attributed to Fred Jelinek, IBM

Blueprint: three hidden states in a row with transition arrows, each emitting to an observed box below

Speech is the hardest possible case, because you don't get words at all, you get a pressure wave. So they flip the question over: instead of asking what the sound means, ask which sequence of words was most LIKELY to have produced this particular noise. [Point at the diagram.] The words are the hidden row and the sound is the observed row. And you already know this machinery, because it's the Markov chain from earlier with the states hidden. [REVEAL] There's no grammar in here, no parsing, nothing anybody would call linguistics, and it beat every rule-based system that had ever been built. It's why your phone can take dictation today. [REVEAL quote] There's a line attributed to Fred Jelinek, who ran the IBM group. He argued about the exact wording later on, so I'll say attributed: "every time I fire a linguist, the performance of the speech recogniser goes up." Brutal. Also more or less what the next twenty years went on to prove.

Do rocks count?

1990s

The counting finally has enough to count.

Large-scale digital corpora enable statistical methods.

1990s Statistical Era: Text exploded with the web, cheap disks, and digital archives.
More Data Beats Better Rules: Pure statistical counting on huge piles of text outperformed manual linguistics.
Bridge to Next Slide: Statistical Machine Translation at scale.

Additional Context

The nineties. And notice what does NOT change here, which is the maths. We already have all of it: counting words, weighting them by how rare they are. That's on the board already and some of it is twenty years old. What changes is the pile. Suddenly there's text everywhere and disks cheap enough to keep it on. The web shows up, first website 1991, Mosaic in '93. Digitised newspapers, parliamentary transcripts, email. For the first time there is more text than any human being could read, which is exactly the condition where counting beats reasoning. [The hands beat for this material now lives on the 1948 Shannon card, where n-grams are introduced.]

The 1990s · machine translation, again

Statistical translation scales with parallel corpora.

IBM trains on Canadian Hansard.
Same math, just more text.
No grammar. Just counting words.

A page of Hansard, the Canadian Dominion Parliament, 17 July 1895 — **Hansard.** Every debate, published in English and French, which makes an accidental translation corpus. Parliament of Canada · public domain

Parallel Corpora: Canadian Hansard (English/French transcripts) used to align translation statistics.
Data-Driven Translation: IBM Translation Models (IBM Models 1-5).
Bridge to Next Slide: 2000s Web-scale sentiment analysis.

Additional Context

Remember what a committee said in 1966: "no immediate or predictable prospect of useful machine translation." That's the sentence that killed the field's funding. Twenty-five years later IBM has another go, and instead of grammar rules they use the Canadian Hansard. Canada publishes every parliamentary debate in both English and French, by law, which means millions of sentences already translated by human beings, sitting in a filing cabinet, free. [REVEAL] And notice what is NOT in this story, which is a new idea. Markov 1913, Shannon 1948, Harris 1954, Spärck Jones 1972 — every technique the nineties reach for is already decades old. What the decade adds is the pile. Newspapers get digitised, everything lands on a disk, and then the web turns up. [REVEAL] Their system never learns a single rule of French. It counts which words tend to sit opposite which other words, across millions of pairs, and it beats twenty years of hand-written rules. Same problem, declared hopeless, solved by counting. And that's the first half of the night. Nobody taught the rocks language, they just counted it.

Do rocks care?

2000s

Putting the counting to work.

Turning 90s statistical tallies into practical outcomes: sentiment, topics, and classification.

Sentiment & topics · 2002–2003

The same counting, pointed at feelings.

Score bags for sentiment.
Cluster bags for topics.
Still can't read. Sarcasm breaks it.

Blueprint: a gauge reading 80% between NEG and POS

One number, from a pile of counts.

Sentiment Analysis (Pang & Lee 2002): Classifying text as positive or negative.
N-gram & SVM Classifiers: Using word frequencies to detect movie review sentiment.
Bridge to Next Slide: 2003 Neural Language Models (Bengio).

Additional Context

This is where we take the statistical counting built in the nineties and point it directly at outcomes. It's the same bag of words, and there are two things you can do with it. You can score the bag along one axis, so how positive is this, and that's sentiment analysis. Or you can cluster the bags into groups and see what falls together, and that's topic modelling. [REVEAL] 2002, a famous paper on movie reviews: throw bag-of-words at a basic classifier and you get about eighty percent on thumbs up versus thumbs down. That's good enough to be useful, and within a few years every company on earth is running it over their customer feedback. [REVEAL] But hold onto the limitation, because it's the reason for everything after this. It cannot read. "This film was not remotely as bad as I expected" is a positive review, and it's absolutely stuffed with negative words. Sarcasm eats it alive. To fix any of that you'd need the machine to know what words MEAN, and nobody has the faintest idea how to do that. Until somebody does.

Do rocks mean?

2003

Yoshua Bengio stops treating words as symbols and starts treating them as numbers.

Dense vector representations for neural language modeling.

The neural language model · 2003

Give every word a list of numbers.

Before: cat unrelated to kitten.
Now: words get number rows.
Kitten looks like cat.

Blueprint: words feeding a lookup table, then a hidden layer, then an output distribution

Words → a learned table of numbers → a prediction.

Learned Vectors (Embeddings): Give each word a row of numbers learned during prediction.
Similarity & Generalization: Similar words drift to similar numbers — "kitten looks like cat".
Bridge to Next Slide: Visualizing words as coordinates in Vector Space.

Additional Context

Here's the problem he's solving, and it's the one that has been sitting under this whole talk. To an n-gram, every word is just a symbol, so cat and kitten are exactly as unrelated as cat and Tuesday. There is no way anywhere in that machinery to say two words are similar. [REVEAL] So Bengio does something that sounds trivial and really isn't. He gives each word a row of numbers, a vector, and instead of choosing those numbers by hand he lets the network learn them while it learns to predict the next word. Words that get used the same way end up with similar numbers, because that's what makes the prediction better. [REVEAL] And look what that buys you. Say your model has never seen "the kitten sat." An n-gram is completely stuck: count zero, no idea. This model knows kitten's numbers look like cat's numbers, and it has seen "the cat sat" a thousand times. So evidence transfers between words for the first time ever. That's the seed, and it sits there quietly for ten years.

What the numbers are · 2003

A list of numbers is a place.

Two numbers plot a point.
Distance measures meaning.
The list is a vector.

Two numbers per word, plotted: cat and kitten sit 0.6 apart, and Tuesday is 7.6 away.

Vector Space Geometry: Words become points in space. Distance equals semantic similarity.
Distance Measurement: cat->kitten is 0.6 apart; cat->Tuesday is 7.6 apart.
Bridge to Next Slide: 2013 Word2Vec & vector arithmetic.

Additional Context

[PACE — slow down here. This is a load-bearing explanation and much of the room has English as a second language. Say the numbers slowly and pause after each reveal.] I want to stop on "a row of numbers" for a second, because everything after this depends on it. Take the simplest possible version: two numbers per word instead of three hundred. Two numbers is a point you can draw, like a coordinate on a map. [Point at the grid.] cat goes here. kitten lands right next to it, because it gets used in the same sentences. Tuesday is up here, miles away. [REVEAL] And here's what that buys you, which counting never could. You can MEASURE it. cat to kitten: 0.6. cat to Tuesday: 7.6. Thirteen times further. Say that again, thirteen times further. And nobody wrote a rule about cats, or animals, or days of the week. The numbers came out of predicting the next word, and the distances arrived for free. [Honest aside, worth 10 seconds: the two axes don't mean anything you could name. Not "furriness", not "royalty". People draw it that way and it's not true. The individual numbers are not interpretable — only the distances and directions are.] [REVEAL] Real models use around three hundred numbers per word instead of two. You cannot draw three hundred dimensions and neither can I. But nothing important breaks, because near is still near and a direction is still a direction. The maths doesn't care that you've run out of paper. And the word for that list of numbers is a vector. That's the whole definition: a list of numbers you're allowed to treat as a place. [Bridge] Hold onto "distance you can measure", because it comes back twice. Once in ten years, when the directions turn out to mean something, and once at the end of the talk when we get to RAG.

Do rocks understand?

2013

A team at Google works out how to put meaning on a map.

Mikolov et al. introduce Word2Vec and vector space arithmetic.

word2vec · 2013

Meaning, as a direction you can travel.

Directions reveal meaning.

king − man + woman = ?

Raise your hand

Who got queen?

Machine gets queen too.
Math emerges from text.

Blueprint: points labelled MAN, WOMAN, KING, QUEEN in a scatter, with two parallel arrows showing the same offset

The step from man to woman is the same step as king to queen.

The thing

word2vec: read a lot of text, and give every word numbers based on the company it keeps.

It unlocks

Similar meaning becomes similar numbers, so a machine can measure meaning instead of matching letters.

Vector Arithmetic: King - Man + Woman = Queen.
Linear Relationships: Semantic relationships correspond to consistent direction vectors in space.
Bridge to Next Slide: Table 8 relationship breakdown & failure cases.

Additional Context

[PAYOFF — pays off 041 (Harris 1954, a word is the company it keeps) and 121 (Bengio 2003). Name both years out loud.] [PACE — slow down here. This is a load-bearing explanation and much of the room has English as a second language. Say the numbers slowly and pause after each reveal.] [SAY THE NAME. It's "word2vec" — word, the digit two, vec. Google, 2013, Tomas Mikolov's team. Say it now and say it again at the end of the slide; the audit found this deck taught the idea without ever speaking its name until half an hour later.] Every word becomes a few hundred numbers, so think of it as a point in space, and words used in similar ways sit near each other. Fine, that's Bengio's idea at scale. But then somebody noticed something nobody had been looking for. [REVEAL maths — read it out, then STOP. Let the room work.] King, minus man, plus woman, equals…? [REVEAL hands. HANDS ONLY.] Who got queen? — [count] "Nearly all of you." [REVEAL] So does the model. And I want to be precise about why that's astonishing. Nobody wrote a rule about royalty or gender, and nobody built a dictionary. It read a large pile of perfectly ordinary text, gave every word some numbers, and the direction you travel from man to woman turns out to be the same direction you travel from king to queen. Meaning showed up as geometry. [REVEAL diagram — point at it.] Same arrow, twice. That's the whole result. And this is Harris's distributional hypothesis from 1954 finally paying off: you shall know a word by the company it keeps. It only took sixty years and an awful lot of text. [If challenged, the honest footnote: the standard test excludes the input words when it looks for the nearest neighbour. The regularity is real; the vector lands nearest queen, it doesn't land exactly on it.]

word2vec · Mikolov, Chen, Corrado & Dean · 2013

Analogies as vector arithmetic.

Subtract two, add one.
Nobody programmed these.

RELATIONSHIP EXAMPLE 1 EXAMPLE 2 EXAMPLE 3 France − Paris Italy: Rome Japan: Tokyo Florida: Tallahassee big − bigger small: larger ✗ cold: colder quick: quicker Miami − Florida Baltimore: Maryland Dallas: Texas Kona: Hawaii Einstein − scientist Messi: midfielder Mozart: violinist Picasso: painter Sarkozy − France Berlusconi: Italy Merkel: Germany Koizumi: Japan copper − Cu zinc: Zn gold: Au uranium: plutonium ✗ Berlusconi − Silvio Sarkozy: Nicolas Putin: Medvedev ✗ Obama: Barack Microsoft − Windows Google: Android IBM: Linux Apple: iPhone Microsoft − Ballmer Google: Yahoo ✗ IBM: McNealy ✗ Apple: Jobs Japan − sushi Germany: bratwurst France: tapas ✗ USA: pizza

Scores only 60%.
Right shape, wrong details.
This is hallucination.

Table 8, “Efficient Estimation of Word Representations in Vector Space” · skip-gram, 783M words, 300 dimensions.

Table 8 Analysis: 9 relationship types (capital cities, plurals, tenses).
60% Exact Match: Confidently wrong on 40% of queries (e.g. France -> tapas instead of Spanish).
Bridge to Next Slide: Cosine similarity and vector search.

Additional Context

[PACE — slow down here. This is a load-bearing explanation and much of the room has English as a second language. Read down the left column slowly.] Now I want to show you the actual table from the actual paper, because king-minus-man-plus-woman makes it sound like a party trick and it is not a party trick. [Point down the left column, read them out.] France minus Paris, that's the capital-city relationship. Big minus bigger, that's grammar, comparatives. Miami minus Florida, cities to states. Einstein minus scientist, people to professions. Copper minus Cu, chemical symbols. Berlusconi minus Silvio, surnames to first names. Microsoft minus Windows, companies to products. Microsoft minus Ballmer, companies to chief executives. Japan minus sushi, countries to food. Nine completely different kinds of relationship, and it's the SAME operation every single time. Subtract, add, look at what's nearest. Nobody wrote a capital-cities table, nobody typed in the periodic table. It read text. [REVEAL the crosses.] And now the part I love, because it's the honest part. Look at these. Small: larger, no, that's smaller. Uranium: plutonium, no, that's U. Putin: Medvedev, which is not his first name. IBM: McNealy, and Scott McNealy ran Sun Microsystems. France: tapas, and tapas is Spanish. [REVEAL] And they published it anyway. Here's the sentence that comes right after this table in the paper: score it on exact match, and it gets "only about sixty percent." Sit with that for a second, because it tells you exactly what kind of machine this is. It's right about the SHAPE of the world, that countries have foods and companies have bosses and metals have symbols, and it's confidently wrong about the details. Fluent about the structure, unreliable on the facts. [Bridge] Remember that sentence, because in about an hour we're going to give that exact behaviour a name, and the name is hallucination. It has been sitting here since 2013. It was never a bug somebody added later. [If challenged on the crosses: I've only marked the six that are unarguable. A few others are debatable — Mozart played violin but he was a composer, Apple's answer is a phone not an operating system. I'm not adjudicating those; the paper's own sixty percent does the work.]

Embeddings in production · 2013–2019

Almost nobody used it for analogies.

Vectors became easy.
People asked: what's nearest?
This was early RAG.

most_similar("king")

queen0.74 monarch0.69 prince0.67 throne0.61

Two lines of Python, and the values are illustrative.

1Turn every document into a vector.
2Turn the question into a vector.
3Return the documents nearest to it.

Cosine Similarity: Measuring angle between vectors to compute document relevance.
Vector Search Engine: Embed question, embed documents, return nearest items in space.
Bridge to Next Slide: 2015 Attention Mechanism.

Additional Context

Now, the six years everybody forgets, and they're the six years this stuff got into your life. Because the thing about 2013 is that Google shipped the CODE, not just the paper. And then Radim Řehůřek ported it to Python in a library called gensim, and that is the moment this leaves the labs. Stanford put out GloVe in 2014, and Facebook put out fastText in 2016, which can even handle words it has never seen before by looking at the pieces of the word. So by about 2015, any developer anywhere could have working embeddings in an afternoon. And here's what I want you to notice. Almost nobody used them for king minus man plus woman, because that's the demo. [Point at the list.] What they actually used was this, most_similar: give me the things nearest to this thing. Related products, duplicate support tickets, search results that match what you meant instead of what you typed. [REVEAL the three steps — go slowly, and let them get there before you say it.] So the recipe is: turn every document into a vector. Turn the question into a vector. Return the closest documents. [REVEAL] Some of you have already worked out what I've just described. That is RAG. That's the retrieval step, exactly, and it's running in production from about 2015, five years before the paper that names it and a decade before anybody put it on a slide. [Forward-reference, do NOT spoil it — the twist lands at the end of the talk] Hold onto that, because when we get to RAG properly I've got one detail about which method actually ended up doing the work in production, and it's funnier than this. [Illustrative numbers again — if challenged, say so. The ranking is the real behaviour; I'm not claiming those exact cosine values.]

Recurrent Neural Networks · 2010–2014

Reading word by word: the recurrent loop.

Read one word sequentially.
Pass memory to next word.
Cannot run in parallel.

The RNN Recurrent Loop: folded cell (left) and unrolled sequential chain (right).

What it did

Read text one word at a time, carrying hidden memory from word to word.

It unlocked

Variable-length input — neural networks could handle full sentences instead of fixed bags of words.

RNN Recurrent Loop: Mikolov 2010 (RNNLM) & Sutskever 2014 (seq2seq).
Sequential Dependency: Memory vector h_t passes forward one word at a time.
Bridge to Next Slide: LSTMs & the context bottleneck problem.

Additional Context

Before we get to attention, we have to look at how neural networks handled text between 2010 and 2014: the Recurrent Neural Network (RNN). Up to now, bag-of-words threw away word order completely. An RNN brings order back by reading like a human does: word one, word two, word three, strictly in a row. [Point at the diagram] Word one ("The") feeds into the RNN and produces a memory vector h1. That memory passes to word two ("cat"), which produces h2. For the first time, a neural network could read a sentence of ANY length. But notice the structural limitation: word three CANNOT compute until word two is finished. It is single-file. And that single-file line is the exact reason GPUs couldn't train them in parallel across massive text datasets.

LSTMs & Seq2Seq · 2014

The memory bottleneck.

LSTM added memory gates.
Squeezes sentence into one vector.
Fixed box drops details.

The seq2seq bottleneck: input length grows, vector size stays fixed.

What it did

Long Short-Term Memory (LSTM) gates allowed networks to remember words longer.

The flaw

Fixed-Vector Bottleneck — squeezing 50 words into a 512-D vector forced early words to fade.

LSTMs (1997) & Seq2Seq (2014): Sutskever et al. (Google) & Cho et al. (Montreal).
The Context Bottleneck: Single vector must hold entire sentence meaning.
Bridge to Next Slide: 2015 Attention (which solves the bottleneck by looking back).

Additional Context

Now we arrive at 2014, the peak of seq2seq (sequence-to-sequence). LSTMs solved basic RNN memory loss by adding "gates" that control what to keep and what to forget. In 2014, Google used them to build sentence-to-sentence translation. [Point at the diagram] An encoder RNN reads the English sentence, squeezes it down into ONE fixed-size vector, and then a decoder RNN writes the Spanish sentence from that vector alone. Notice the flaw: whether your sentence is 5 words or 50 words, it has to squeeze through that exact same fixed vector. Information at the beginning of a long sentence gets overwritten and faded. And THAT is the exact bottleneck that Bahdanau and Bengio set out to solve in 2015 with Attention.

Do rocks pay attention?

2015

A translation model is allowed to look back at any word it wants.

Bahdanau et al. introduce neural attention mechanisms.

Attention · 2015

Let it look back at everything.

Old models crushed long sentences.
Attention looks back at everything.
The window is gone.

Disambiguation across distance

“She grew up in France, moved abroad for work, lived in three different countries for twenty years, and yet at home she still speaks fluent French.”

Attention connects France (17 words back) directly to predicting French.

The thing

A score for every pair of words, saying how much one should listen to the other.

It unlocks

Distance stops costing anything, so word one and word fifty are one step apart.

Long-Range Disambiguation Example: Attention connects "France" (17 words back) directly to predicting French.
No Distance Cost: Word 1 and word 50 consult each other in a single step — context window limitation destroyed.
Bridge to Next Slide: Attention Query, Key, Value mechanics.

Additional Context

[PAYOFF — pays off 021's five-word window. The debt you asked them to remember. Say “the window is gone”.] Here's the state of the art before this. A translation model reads your whole sentence, squeezes it down into one fixed-size list of numbers, and then writes the translation from that summary alone. Which works fine for short sentences and falls completely apart on long ones, because it's the same size box no matter how much you put in, and everything that didn't fit is lost. [REVEAL] Attention throws the box away. For every single word it produces, the model can look back across the entire input and decide which parts matter for this word, right now. Translating a verb? Go and look at the subject, wherever it happens to be. [REVEAL] And this is the debt I asked you to remember. The n-gram had a five-word window and anything before it was invisible. "I grew up in France, so I speak fluent —" needs a word twelve back, and an n-gram never sees it. This one does. The window is gone. Two years later somebody asks the obvious question: if attention is doing all the work, do we actually need the rest of it? [FACT GUARD — attention is Bahdanau, Cho & Bengio, Montreal. NOT Google. Google is GNMT in 2016 and the Transformer in 2017. 145 puts Bahdanau's name on screen, so the slide will contradict you. And don't say “BERT” here; that's 2018, three stops away.]

How attention actually works · one word, one step at a time

Every word asks a question, and the others answer.

The animal didn't cross the street because it was too tired.

You know it means the animal. Nothing in the model was told that. Here's what it computes instead.

1 · Each word emits three vectors

Query: What I want.
Key: What I offer.
Value: What I contribute.

2 · Score every pair

score = query(it) · key(word)

A dot product. Two vectors pointing the same way score high. That is the entire definition of “relevant”.

3 · Turn scores into shares

Weights add to 1.
A budget of attention.

4 · Take a weighted average of the values

Illustrative weights. The new vector for it is now mostly “animal”, so the word has been rewritten to carry what it refers to.

softmax(Q·Kᵀ) · V
Score, normalise, average.
For every word at once.

Q, K, V Matrices: Query, Key, Value dot-product scoring.
Softmax Weights: Normalizing scores into attention weights summing to 1.0.
Bridge to Next Slide: Attention visualization map.

Additional Context

[PACE — slow down here. This is a load-bearing explanation and much of the room has English as a second language. Say the numbers slowly and pause after each reveal.] OPEN: this is the slide. Understand this one and you understand LLMs; everything after tonight is this at scale. [Read it.] “The animal didn't cross the street because IT was too tired.” You know “it” is the animal, instantly. A machine has no rule for that. So what does it do instead? [REVEAL 1] Every word emits three vectors, each just the word's numbers times a learned matrix. Query = what am I looking for. Key = what do I offer. Value = what I contribute if picked. [Analogy, if there's room: a room full of people, each holding a sign saying what they're about, each with a question they want answered.] [REVEAL 2] Score every pair by dot product. Query of “it” against the key of every other word. Same direction, big number. That is the entire definition of “relevant” here. No grammar, no rule about pronouns. [REVEAL 3] Softmax turns the scores into shares that add to 1. A budget of attention. [REVEAL 4 + chart] The payoff: 51% of “it”'s attention lands on animal. Weighted average of everyone's VALUE vectors, mostly animal's, and “it” gets rewritten to carry “animal” inside it. The pronoun was resolved by arithmetic. [REVEAL last] The formula everyone quotes: softmax(QKᵀ)V. Score, normalise, average. And you do it for every word at once, which is the bit that made 2017 possible. [FACT GUARD — the dot product IS the score. Order is: dot product, then softmax, then weighted average of the values. Don't say “score them and then dot product”.]

The lead-up · 2013 → 2016

Everything the Transformer needed already existed.

YEAR WHAT LANDED WHY 2017 NEEDED IT 2013 word2vec → words arrive as vectors you can do maths on 2014 encoder–decoder (seq2seq) → the overall shape: read a sequence, write one 2015 attention Bahdanau → the part that does the actual work 2015 residual connections ResNet → why you can stack a hundred layers at all 2016 layer normalisation → why that deep stack trains without falling over 2016 attention with no recurrence Parikh → proof you could drop the RNN and still win 2016 self-attention → a sentence attending to itself, not to a translation

Added three new things.
Threw the RNN away.
An assembly that changed everything.

The Bottleneck of Recurrence: RNNs process word by word sequentially (slow, unparallelizable).
The Big Question: "If attention does all the work, why keep recurrence?"
Bridge to Next Slide: 2017 "Attention Is All You Need".

Additional Context

[PACE — slow down here. Read the left column, don't read the right one; the right column is for them to scan.] Before we get to 2017, I want to show you the four years leading up to it, because the story everybody tells is that the Transformer arrived out of nowhere and it really, really didn't. [Point down the years.] 2013, word2vec — we just did that one. Words become vectors, and you can do arithmetic on them. 2014, sequence-to-sequence: the encoder-decoder shape, read one sequence and write another. 2015, attention, which is the slide we were just on — Bahdanau, Cho and Bengio. [If anyone challenges the year: the paper is on arXiv in September 2014 and published at ICLR 2015. I use 2015 to match the date card. Both are defensible, don't die on it.] Then two pieces of pure plumbing that nobody outside the field has heard of and without which none of this works. 2015, residual connections, from ResNet — that's the trick that lets you stack a hundred layers deep without the signal dying. 2016, layer normalisation, from Hinton's group, which is what makes that deep stack actually train. And then the two that really give the game away. 2016, Parikh and colleagues at Google publish a model that does attention with no recurrence at all — parallelisable, and it deliberately ignores word order. That's the Transformer's whole bet, a year early, on a smaller problem. And also 2016, self-attention: a sentence paying attention to itself rather than to a translation of itself. [REVEAL] So what does 2017 actually add? Three things. Multi-head attention — run it eight times at once. A scaling factor on the dot product, which sounds trivial and is what stops the softmax saturating. And a number representing position, which you need precisely because you've thrown away the word order. Then they delete the RNN and stack the rest. [If asked about other prior work: the paper itself cites ByteNet and convolutional seq2seq as earlier attempts to cut sequential computation. Same direction, different mechanism.] That's an assembly, not an invention. And it changed everything anyway — which is sort of the lesson of the whole night.

First page of the paper Attention Is All You Need — Vaswani et al., NIPS 2017 · reproduced with attribution

Do rocks scale?

2017

Eight people at Google publish a paper with a very confident title.

Vaswani et al. publish "Attention Is All You Need", introducing the Transformer.

Vaswani et al. · Google · June 2017

“Attention Is All You Need”

Old models read one word at a time.
Throw all that away.
Keep only attention.

Blueprint: a left panel labelled RECURRENT showing a chain of boxes in sequence, a right panel labelled PARALLEL showing boxes all connected to each other at once

Process words at the same time.
Massive parallel training unlocked.
Everything after is this architecture.

What the paper did

Took the recurrence out, and kept only attention.

It unlocks

Nothing waits for anything, so you can train it on the whole internet.

Transformer Paper: Attention Is All You Need.
Parallelization: Train on massive clusters simultaneously without waiting for token t-1.
Bridge to Next Slide: What is a Transformer in plain English.

Additional Context

Eight people at Google, June 2017, eight pages, and a title with real swagger: "Attention Is All You Need." And you've just seen the parts list, so you know this is an assembly. Here's what it assembles into. Every model up to now reads a sentence the way you do, one word at a time, left to right, with each step waiting for the step before it to finish. That's called recurrence, and it's been the shape of language models for thirty years. The paper's argument is that attention already lets you look at any word from any position, so why are we reading in order at all? Take the recurrence out and keep only attention. [REVEAL diagram] On the left, the old way, a chain where each box waits. On the right, the new way, everything connected to everything all at once. [REVEAL] And that isn't just tidier, it's the whole ballgame. If nothing has to wait, you can compute every word in parallel, which means you can throw a thousand GPUs at it, which means you can train on the whole internet. The architecture didn't only work better, it unlocked scale. [REVEAL] Everything from here to the end of tonight is this one paper. BERT, GPT, Claude, Gemini. The T in ChatGPT stands for Transformer. This is it. This is the thing. [FACT GUARD — call the Transformer the BIGGEST breakthrough. Agents later are the LATEST. Pick one word each so the two claims don't sound like a contradiction.]

The Transformer Architecture · Vaswani et al. 2017

From input text to probabilities.

Encoder on the left.
Decoder on the right.

1Positional Encoding: Adds word order numbers.
2Multi-Head Attention: 8 parallel attention heads.
3Add & Norm: Residual connections for 100+ deep stacks.

BERT uses Encoder (understanding).
GPT uses Decoder (generation).

Figure 1: The Transformer — model architecture (Vaswani et al., 2017).

The Complete Transformer (Vaswani et al. 2017): Dual Encoder-Decoder architecture for translation.
Encoder (Left): Self-Attention + Feed Forward blocks to encode input sentence.
Decoder (Right): Masked Self-Attention + Cross-Attention + Linear/Softmax to generate output tokens.
Bridge to Next Slide: Essential LLM vocabulary (Tokens, Context Window, Parameters).

Additional Context

Here is Figure 1 from the original paper, in full detail. On the left is the Encoder: inputs arrive, get positional numbers added, pass through multi-head self-attention, and edit their representations through feed-forward layers. On the right is the Decoder: outputs generate token-by-token using masked self-attention (to prevent looking at future words) and cross-attention into the encoder. Notice how BERT and GPT split this picture in half: BERT uses only the left half (Encoder) for understanding, while GPT uses only the right half (Decoder) for generation.

Words you'll hear for the rest of the night

Eight words, and seven are older than the paper.

TERM IN PLAIN ENGLISH WHERE IT CAME FROM parameter one learned number inside the model older than all of this weights all the parameters, together older than all of this token the unit it counts, roughly a word Markov's letters · 1913 embedding a list of numbers standing in for text Harris 1954 · Bengio 2003 encoder / decoder reads a sequence · writes one seq2seq · 2014 attention a score for every pair of words Bahdanau · 2015 context window how much text it can look at in one go the n-gram window · 1948 multi-head that same attention, eight times at once genuinely new · 2017

Look at the right column.
Only one word is new.
The rest were waiting.

LLM Key Terms: Parameter, Context Window, Token, Multi-Head, Encoder/Decoder.
7 of 8 Words Older Than 2017: Only "multi-head" was brand new in 2017.
Bridge to Next Slide: 2018 BERT & Bidirectional context.

Additional Context

[PACE — slow down here. This is the vocabulary the whole back half assumes, and much of the room has English as a second language.] [PAYOFF — pays off 002-roadmap's "count how often the new thing is somebody's old idea", and sets up 241-shoulders. Do not skip this slide.] Right, checkpoint. We've built the machine, and from here on I'm going to use seven words constantly, so let's pin them down. Parameter first, because I use it all night and it's the one people get wrong. A parameter is ONE learned number inside the model. That's it. All of them together are what people mean by "the weights". So when I say a hundred and seventy-five billion parameters in a few minutes, that is a hundred and seventy-five billion individual numbers. [Point down the rest, don't read every one.] Token, the unit it counts. Embedding, a list of numbers standing in for a piece of text. Encoder reads a sequence, decoder writes one. Attention, a score for every pair of words. Multi-head, that same attention run eight times over. [Point at context window — say this one properly.] Context window is how much it can look at in one go. And that is the n-gram's five-word window from 1948, grown. Same idea, same name almost, four orders of magnitude bigger. When somebody sells you a two-million-token context window, they are selling you a bigger n-gram. [REVEAL — the point of the slide.] Now look down the right-hand column, because this is the bit I love. Eight words. One of them, multi-head, was actually invented by the paper we just read. The other seven were already lying around. 1913, 1954, 2003, 2014, 2015. Keep counting. There's a list at the end.

The problem BERT solves · 2018

One word doing two jobs.

word2vec gives one vector per word.
"Bank" becomes a blur of both.
BERT gives occurrences vectors instead.

The bank robber stole money from the vault.

→ money, crime, a building

Let's go fishing on the river bank.

→ mud, water, the edge of a thing

We've been here before

This is “is book a noun or a verb” from 1971, the same four letters doing two jobs. That one was about grammar and this one is about meaning, and the answer is the same: read the neighbours.

2018 BERT: Devlin et al. at Google.
The "Bank" Problem: Static embeddings gave "bank" one vector. BERT gives occurrences vectors based on context (river bank vs money bank).
Bridge to Next Slide: BERT vs GPT architecture split.

Additional Context

[PAYOFF — pays off 075-pos-tagging. “Is book a noun or a verb”, 1971, same four letters doing two jobs. 47 years apart, same answer: read the neighbours.] [PACE — slow down here. This is a load-bearing explanation and much of the room has English as a second language. Read both sentences out loud.] OPEN: the problem BERT exists to solve, and it breaks everything we just spent ten minutes admiring. [Read both sentences out loud.] “The bank robber stole money from the vault.” / “Let's go fishing on the river bank.” A building full of money, and the muddy edge of a river. [REVEAL] word2vec gives every word ONE row of numbers. So “bank” gets averaged: one point halfway between money and mud. Slightly wrong in every sentence you'll ever use. No mechanism for being right, because the vector was fixed before it saw your sentence. [REVEAL — say this one slowly.] BERT's idea: stop giving WORDS vectors, start giving OCCURRENCES vectors. Read the whole sentence, both directions, then place THIS bank. Two sentences, two sets of numbers, same four letters. That's what “contextual” means. And it's why bidirectional matters: you can't place “bank” without “robber” before it and “vault” after it. [REVEAL callout — the payoff. Land it.] We did this in 1971. Is “book” a noun or a verb? Same four letters, two jobs. That one was grammar, this one is meaning. Forty-seven years apart, same answer both times: read the neighbours.

GPT-3 & scaling laws · 2020

Bigger stopped being boring.

More data. More compute. Same architecture.
It learned new jobs without training.
Prompting was born here.

Blueprint: bars climbing 117M, 340M, 1.5B, 175B

Same idea, four orders of magnitude.

The thing

More data and more compute buy you a predictable amount of accuracy. You can plot it.

It unlocks

You can budget for intelligence — which is why the billions started arriving.

2020 GPT-3: Brown et al., OpenAI (117M -> 1.5B -> 175B parameters).
Scaling Laws (Kaplan et al.): Predictable performance gains from scaling data & compute.
In-Context Learning (Prompting): Provide 2 examples in prompt, model performs task without retraining.
Bridge to Next Slide: 2021 Fine-tuning & LoRA.

Additional Context

Look at the shape of this. 117 million parameters in 2018, a billion and a half the next year, then a hundred and seventy-five billion in 2020. And the important bit is that the architecture doesn't change at all. It's still the same 2017 transformer, just fed more text on more computers. You'd expect that to make it a bit better at predicting words, and instead it did something much stranger. [REVEAL] It started doing tasks nobody had trained it for. Write two examples of English into French in the prompt, and it translates the third one for you, with no training run, no new model, nothing updated anywhere. They called it few-shot learning, because the model picks up the task from the prompt itself. [REVEAL] And that's where prompting comes from. For seventy years, using a language system meant writing code or training a model, and from here it means writing a sentence. This is the moment the interface changed for everybody in this room. [FACT GUARD — GPT-3 is 2020 and developer-only. ChatGPT is 30 Nov 2022, on GPT-3.5. There is no such thing as “ChatGPT 3.0”. Keep the two-year gap; it's the point of 181.]

Fine-tuning & LoRA · 2021

Change the behaviour without touching the model.

Classic fine-tuning updates every weight.
LoRA trains small side matrices.
Megabytes. Single-GPU. Swappable adapters.
Make it sound like you.

Blueprint: RAG shown as documents fed to a model, fine-tuning shown as documents changing the model itself

Fine-tuning changes the model, and RAG only changes what you hand it.

2021 LoRA (Hu et al.): Low-Rank Adaptation for efficient fine-tuning.
Freeze & Adapt: Freeze 175B base model, train tiny rank-A/B matrix pair alongside.
Democratizing Fine-Tuning: Data center job becomes single-GPU job; megabytes instead of gigabytes.
Bridge to Next Slide: 2022 RLHF & ChatGPT.

Additional Context

Fine-tuning is simple to describe: take a finished model and keep training it on your own examples until it behaves how you want. Legal tone. Your product names. Your support style. The problem is the bill. Classic fine-tuning updates every weight in the model, so for each task you end up owning a full private copy of something with a hundred and seventy-five billion parameters in it. That is not a thing a normal company does. [REVEAL] LoRA's trick is genuinely lovely. Freeze the original model, don't touch it at all, and train a small pair of matrices that sit alongside it and nudge the results. You're steering through a side channel instead of rewriting the engine, with orders of magnitude fewer numbers to train. [REVEAL] And that changes who gets to play. A job that needed a data centre now runs on one GPU. The thing you produce is a few megabytes rather than a few hundred gigabytes, small enough to email, and because the base model is untouched you can keep a shelf of adapters and swap them like plugins. One model, many personalities. [REVEAL] This is how a mid-sized company ends up with a model that sounds like them. Not by training one, by adapting one. And keep this next to the RAG slide, because people confuse them constantly: RAG changes what the model KNOWS. Fine-tuning changes how it BEHAVES. If it doesn't know your products, no amount of fine-tuning fixes it. If it won't stop being chatty, no amount of RAG fixes that.

LoRA · what “low-rank” actually means

Don't learn a new matrix. Learn two thin ones.

Model is stacked weight matrices.
Take one: 16.8 million numbers.
Fine-tuning changes every single one.

Why two thin ones work

Multiply a tall-thin ($4096 \times 8$) by a short-wide ($8 \times 4096$) and you get a full-size matrix — but one that can only vary in 8 directions. Adapting is a nudge, not a rebuild.

What you train

W never moves. Only A and B get gradients — 65,536 numbers instead of 16.8 million per matrix. That's the whole saving.

At runtime

Fold W + B·A into one matrix and it runs at original speed. Or keep them apart and swap A and B per task/customer like plugins.

Let me show you the actual matrices, because "low-rank adaptation" sounds like jargon and the idea underneath is simple enough to draw. A model is stacks of weight matrices. Pick one — say four thousand by four thousand. That's sixteen point eight million numbers in a single matrix, and there are hundreds of them. Ordinary fine-tuning computes a change for every one of those numbers. [Point at the diagram.] LoRA says: don't. Leave W frozen — dashed, untouched. Instead, learn two thin matrices. B is four thousand tall and eight wide. A is eight tall and four thousand wide. Multiply them together and — this is the bit worth pausing on — you get something the full size of the original, four thousand by four thousand. A full-size update, built from two skinny things. But here's the constraint that makes it cheap: because it came from something only eight wide, that update can only vary in eight independent directions. It's full-size but low-rank. And the counts: sixteen point eight million versus sixty-five thousand. Two hundred and fifty-six times fewer numbers to train. [REVEAL the three boxes] Why does it work? Because adapting a model to your task is a nudge, not a rebuild — the model already knows English, you're adjusting how it leans. Turns out eight directions is plenty for that. What do you train? Only A and B. W never moves — which also means you can't damage the base model. And at runtime, you can just add B times A back into W, fold it into one matrix, and it runs at exactly the original speed. Or leave them separate and swap A and B per customer. That's why these behave like plugins.

RLHF · reinforcement learning from human feedback · 2022

Somebody had to teach it manners.

Predicting text isn't helping.
RLHF teaches good answers.
Behaviour beats size 100×.

Blueprint: a stack labelled TRANSFORMER at the base, GPT-3 above it, RLHF above that, all inside a box labelled CHAT BOX

Nothing new at the bottom, and everything new at the top.

2022 RLHF & ChatGPT: Reinforcement Learning from Human Feedback.
Alignment: Base models predict text; RLHF teaches manners and helpfulness.
Behavior Beat Size: 1.3B RLHF model preferred by humans over raw 175B base model.
ChatGPT Launch: Free text box UI got 1 Million users in 5 days.
Bridge to Next Slide: 2023 Hallucination & RAG.

Additional Context

The problem with a raw language model is genuinely funny once you see it. It's trained to continue text. So you type a question at it, and a really good text-continuation engine responds with… more questions, because that's what a list of questions looks like. It isn't being unhelpful on purpose. Helping was never in the objective. [REVEAL] So, reinforcement learning from human feedback. Have people rank the model's answers from best to worst, train a second model to predict those rankings, then tune the first model to score well against it. You've stopped teaching it facts and started teaching it what a good answer looks like to a human being. [REVEAL] And the numbers here are genuinely shocking. A 1.3 billion parameter model tuned this way was preferred by human raters over the raw 175 billion one. A hundred times smaller, and people liked it more. Behaviour beat scale. [Point at the stack.] So look at what ChatGPT actually is. There's the transformer at the bottom, which is 2017. Then GPT-3, which is 2020, two years old and already public. Then RLHF on top of that. And then the part nobody in the field thought mattered at all: a text box, free, with no signup friction, which got to a million users in five days. The lesson a lot of engineers took a while to swallow is that the last layer was the product. [FACT GUARD — ChatGPT is the packaging of a two-year-old model. GPT-3 shipped in 2020. Don't merge them.]

Hallucination & RAG · 2023

There were never any facts in there.

Fluent and true are separate.
Making things up is default.
Put facts in front.

Blueprint: two overlapping circles labelled FLUENT and TRUE, the overlap labelled BOTH

Structural Hallucination: Models optimize for plausible next tokens. Plausible text is sometimes false.
No Internal Database: Fluency and truth are separate axes.
Bridge to Next Slide: Grounding models with RAG (Retrieval-Augmented Generation).

Additional Context

[PAYOFF — pays off 131-word2vec: 60% on exact match, confidently wrong about the details. Same behaviour, thirteen years earlier, before it had a name.] So why does it make things up? And the answer is more uncomfortable than "there's a bug in it." It produces plausible next words. That's the whole mechanism. Fluent and true are two completely different axes [point at the circles] and it is only ever optimising one of them. There is no fact database in there to check against — it's a very large set of numbers about which words tend to follow which other words. [REVEAL] Which means making things up is exactly the same operation as getting it right. There isn't a separate step that malfunctions. It's producing plausible text, and plausible text is sometimes false. When a lawyer files a brief with invented case law, the model was not broken at that moment. It was working precisely as built. And you've seen this behaviour already tonight, by the way. Word2vec's table in 2013: right about the shape of the world, confidently wrong about the details, sixty percent on exact match. Same thing, thirteen years earlier, before it had a name. [REVEAL] So that tells you the shape of any fix. You cannot argue it into being right, and you cannot train the wrongness out of it, because the wrongness is the mechanism. What you can do is put the actual facts in front of it before it answers. That's the next slide, and it's my favourite callback of the night.

RAG · retrieval-augmented generation · 2020

You have already seen this slide.

Turn documents into vectors.
Measure documents instead of words.
Embeddings made similar a number.

1Turn every document into a vector.
2Turn the question into a vector.
3Hand it the documents nearest to the question.

And the funny part

A great deal of production retrieval is still BM25 — keyword counting built on Spärck Jones, 1972. Fifty-year-old maths, doing the fetching, inside your 2026 AI. Most serious systems run both.

The thing

Fetch the real documents, then answer from those.

It unlocks

It can answer about your documents, with no retraining.

RAG Mechanics: Turn documents into vectors, embed question, retrieve nearest, prompt model to answer from context.
Vector Search Callback: Reuses 2013 vector space similarity on full documents.
Bridge to Next Slide: 2024 Reasoning models.

Additional Context

[PAYOFF — pays off 122-vectors, 133-embedding and 134-similarity, and 081-tfidf via BM25. This is the biggest callback in the deck and it got skipped on 2026-07-31. Say “you have already seen this slide”.] [PACE — slow down here. This is a payoff slide and much of the room has English as a second language. Let the recognition land before you explain anything.] OPEN: “You have already seen this slide.” [Pause. Let them look.] [Point at the three steps.] Documents → vectors. Question → vector. Fetch the nearest. Answer from those, not from memory. THE CALLBACK: this is 2013. cat–kitten 0.6, cat–Tuesday 7.6. Same measurement, pointed at documents instead of words. Your handbook is a few thousand points; your question is one more point. [REVEAL] Older still: Schank, 1982, case-based reasoning. Fetch the nearest past case. Stalled because a human had to hand-specify “similar”. Embeddings made similar a number. [REVEAL callout — the promised joke. Land it.] You'd assume this is all vectors. A lot of it is still BM25, keyword counting weighted by rarity. Spärck Jones, 1972. Fifty-four-year-old maths doing the fetching inside a 2026 AI. Serious systems run both. [REVEAL] The pairing people always confuse: RAG changes what it KNOWS. Fine-tuning changes how it BEHAVES. Doesn't know your products? Fine-tuning won't fix it. Won't stop being chatty? RAG won't fix it.

Reasoning models · 2024–25

Let it think for longer.

Spent more words first.
Two dials: size and time.
More effort on hard problems.

Blueprint: a plot with LONGER on one axis and BIGGER on the other, points climbing toward BETTER

Two dials now, rather than one.

The thing

It writes out its working first, often for thousands of words you never see.

It unlocks

You can turn the effort up for a hard question and down for an easy one.

2024 Reasoning Models (o1 / CoT): Spending extra tokens thinking step-by-step before answering.
Two Dials: How big you train it, and how long it thinks at test-time.
Trade-off: Increased latency & token cost for complex math/coding tasks.
Bridge to Next Slide: 2025 Tool Use & MCP.

Additional Context

This one starts with a genuinely silly discovery. In 2022 people found that if you add "think step by step" to your prompt, models get substantially better at hard problems. Same model, same weights, nothing retrained. It just writes out its working before it commits. [REVEAL] So the obvious move is to stop relying on the user to ask nicely and train the model to do it by default, and that's what a reasoning model is. It works the problem through privately, often for thousands of words you never get to see, checks itself, and then answers. And that hands the field a second dial. For a decade the only knob was how big you train it, and now there's another one: how long it thinks at the moment you ask. [REVEAL] It isn't free. Thinking costs money and it costs seconds, and for "what's the capital of France" it's a complete waste. But it means that for the first time a model can put more effort into a harder question, which is what you'd expect from anything you were going to call reasoning.

Tool use & MCP · 2024–25

Language in, actions out. Sound familiar?

Give it tools to act.
MCP: USB-C for AI.
This sandbox holds reality.

Blueprint: a model connected through a single MCP connector out to SEARCH, CODE and DATA

One plug, many sockets.

The thing

Model Context Protocol: one agreed way for a model to call any outside tool.

It unlocks

Write the tool once and every model can use it, with no integration per model.

2025 Tool Use & MCP: Giving models tools (search, database, code execution).
Model Context Protocol (Anthropic 2024): Standardized USB-C like protocol for AI tools.
SHRDLU Callback: Same 1970 loop (Language in -> Action -> Result), but now operating in the real world.
Bridge to Next Slide: 2026 Autonomous Agents.

Additional Context

[PAYOFF — pays off 071-shrdlu. Identical loop, 55 years apart. Flint landed this one unprompted on 2026-07-31; it's the best callback in the talk.] A model that can only emit text is trapped. It can tell you what it would do but it can't actually do it. So give it tools: search, a database, your codebase, and let it decide when to reach for one. It acts, the result comes back, it keeps going. [REVEAL] MCP is the boring important part. Before it, hooking a tool up to a model meant a custom integration for every combination, n times m, forever. MCP standardises the connector so any tool can talk to any model. It's USB-C for AI, and like USB-C the whole value is just that everyone agreed. [REVEAL] Now look at that loop and tell me where you've seen it before. Instruction in English, actions taken in an environment, result reported back, explain yourself afterwards. That's SHRDLU, that's 1970, and we put it on screen forty minutes ago. Fifty-five years, and the difference isn't the loop, because the loop is identical. The difference is that SHRDLU's sandbox held seven blocks all typed in by hand by a graduate student, and this one holds your filesystem, the internet and your calendar. We didn't invent a new idea, we finally got a sandbox big enough to be worth acting in. [FACT GUARD — MCP is Anthropic, November 2024. Say “MCP” or “the Model Context Protocol”, not “the MCP protocol”.]

The agentic loop · 2026

Run the loop without asking permission each time.

Same model in a loop.
No human in every turn.
What that buys is duration.

Blueprint: a loop of PLAN, ACT, OBSERVE, REFLECT with TOOLS attached at ACT

The same four steps, over and over again, unattended.

The thing

The same model, called over and over, choosing its own next step each time.

It unlocks

You stop asking a question and start handing over a task.

2026 Autonomous Agents: Plan -> Act -> Observe -> Reflect loop running unattended.
Buys Duration: Working for 20 minutes on multi-step tasks (read codebase, edit, test, fix).
Bridge to Next Slide: Responsible AI & Human-in-the-Loop.

Additional Context

Here's the whole idea, and it's a good deal less mystical than the word "agent" makes it sound. Plan, act, look at what happened, adjust, go round again. An agent isn't a smarter model, it's the same model we've had all along, in a loop, holding tools, without a human approving every step. [REVEAL] And what that buys you is time. Not one answer to one question, but twenty minutes of work: read the codebase, make the change, run the tests, notice the test failed, fix it, run them again, with nobody watching each turn. And that's the walk. Sixty years from a program that swapped your pronouns around to something that picks up a task and gets on with it. [FACT GUARD — agents are the LATEST breakthrough, not the biggest. The transformer was the biggest; you said both on 2026-07-31 and they clashed.]

1966 · 2026

Sixty years, same conversation.

her Men are all alike.

ELIZA IN WHAT WAY

her They're always bugging us.

ELIZA CAN YOU THINK OF A SPECIFIC EXAMPLE

you Refactor the auth module.

agent Reading 14 files…

agent Tests failing on line 92. Fixing.

agent Done. 3 files changed.

Both felt like talking.
We stopped keeping score.
So what actually happened here?

1966 ELIZA vs 2026 Agents: Sixty years, same conversation interface.
Turing Bar Passed: Turing's 30% / 5 min benchmark stopped being the interesting question.
Standing on Shoulders: Modern AI built on 113 years of cumulative math, statistics, & computing.
Bridge to Next Slide: Final thank you & Q&A.

Additional Context

[PAYOFF — pays off 032-imitation-game. Turing's 30% over five minutes is planted there specifically to be cashed here. It did not get said on 2026-07-31.] On the left, 1966. On the right, this morning. [REVEAL right] Same shape. You type, it answers, sixty years apart. [REVEAL] And one of them is find-and-replace in a trench coat, because we took it apart and there's nothing in there. The other one read fourteen files, ran your tests, found the failure and fixed it. But here's the uncomfortable bit, and it's the reason I started with ELIZA: both of them feel like talking to something. That feeling was never evidence. [REVEAL] Now remember the number I asked you to hold onto. Turing's bar was to fool a judge thirty percent of the time across five minutes of conversation, and he guessed a machine might manage it by the year 2000. Nobody argues about that number any more, and it isn't because somebody proved it got passed. It just stopped being the interesting question, and we went past it and stopped keeping score. [Bridge — do NOT close here, two slides to go. This is the hand-off.] So the obvious question is what actually happened in between those two boxes. And it isn't the story you usually get told. [FACT GUARD — do NOT say “it passed the Turing test”. Nobody certified that. The line is “we went past it and stopped keeping score”.]

1913 → 2026 · 113 years

Thank you.

We taught rocks to talk.

Flint Barrow Fractional CTO · Flintspark Labs flintsparklabs.com

CONNECT ON LINKEDIN

Bonus

This talk was built with the thing it's about.

Research

Primary sources, cross-checked, and where they disagreed the slide says so instead of guessing.

Imagery

Every blueprint generated to a written spec, in one house style.

We TaughtRocks to Talk

What is Artificial Intelligence?

Language used to be a small corner of AI, and now it's the part everybody means.

A hundred and thirteen years, in twenty-three stops.

Six words, and where each one turns up.

He counted 20,000 letters of a novel by hand.

The states can be anything, not just letters.

N words in a row, counted.

He asked the question, then quietly changed it.

A judge, two hidden players and a screen.

You can spot spam without reading a word.

Declared nearly solved, then defunded.

He wrote it to show how little it took.

Two kinds of rule, and neither one reads.

Language in, actions out, all inside a sandbox.

A machine that walks through your sentence.

What kind of word is this?

Twenty-three years to read a word.

The commonest words tell you almost nothing.

Ask what's most likely, not what it means.

Statistical translation scales with parallel corpora.

The same counting, pointed at feelings.

Give every word a list of numbers.

A list of numbers is a place.

Meaning, as a direction you can travel.

Analogies as vector arithmetic.

Almost nobody used it for analogies.

Reading word by word: the recurrent loop.

The memory bottleneck.

Let it look back at everything.

Every word asks a question, and the others answer.

Everything the Transformer needed already existed.

“Attention Is All You Need”

From input text to probabilities.

Eight words, and seven are older than the paper.

One word doing two jobs.

Bigger stopped being boring.

Change the behaviour without touching the model.

Don't learn a new matrix. Learn two thin ones.

Somebody had to teach it manners.

There were never any facts in there.

You have already seen this slide.

Let it think for longer.

Language in, actions out. Sound familiar?

Run the loop without asking permission each time.

Sixty years, same conversation.

Thank you.

This talk was built with the thing it's about.

We Taught
Rocks to Talk