Deep Dive into LLMs like ChatGPT

5,756 words · 29 min read

Understand how LLMs work in detail

Key points explained

LLM pre-training process: Data Collection, tokenization, etc

Practical Examples: Training GPT-2, computational requirements, and using base models for various applications.

Post-Training: Fine-tuning models on conversation datasets to create assistants capable of multi-turn dialogues.

Hallucinations: Addressing the issue of fabricated information and mitigation strategies.

Reinforcement Learning: Using practice and discovery to improve model performance.

Future Developments: Multimodality, long-running tasks, and deeper integration into everyday tools.

Resources: Leaderboards, newsletters, and social media for staying updated.

Accessing Models: Proprietary models, open-weight models, and local deployment options.

Who is this deep dive for?

• You use ChatGPT (or similar) daily but couldn't explain what actually happens under the hood
• Terms like pretraining, tokenization, SFT, and RLHF are fuzzy, and you want them to click
• You're curious why LLMs hallucinate — and how newer models learned to say "I don't know"
• You've noticed models like o1 or DeepSeek-R1 "think" before answering and want to know why
• You're technical, but not an ML researcher, and want real intuition without a research paper

If any of this sounds like you, this post (and probably the original video) is worth checking out. Also, I've gathered some lecture notes of my own on my github.

Part 1: Pretraining — From Raw Text to a Next-Token Predictor

Building something like ChatGPT happens in three broad stages: pretraining (learn general language and world knowledge from raw text), supervised fine-tuning (learn to behave like a helpful assistant), and reinforcement learning (learn to actually get problems right, through practice). This first part covers pretraining — how you go from a pile of raw internet text to a working next-token predictor.

Pretraining Data

Internet

Before an LLM can learn anything, it needs a large body of text to learn from — and that text starts with the internet.

Where the data comes from. Almost every large pretraining effort starts from Common Crawl, a nonprofit that has been crawling the web since 2007 — starting from a handful of seed pages and continuously following outbound links. As of April 2024, it had crawled roughly 2.7 billion web pages. Companies like OpenAI and Anthropic maintain their own internal, further-processed versions of a dataset like this; a good public example is Hugging Face's FineWeb, used as the reference dataset in the lecture.

From raw crawl to training data. Common Crawl's raw output is enormous but extremely messy — mostly HTML, boilerplate, spam, and duplicate content. Turning it into usable training data means running it through a filtering pipeline. FineWeb's version:

1. URL filtering — drop anything from a blocklist of known-bad domains (malware, spam, adult content, etc.).
2. Text extraction — strip HTML markup and boilerplate, keeping only the human-readable text.
3. Language filtering — keep only pages where a given language makes up enough of the content (FineWeb's cutoff: at least 65% English). This is a deliberate trade-off: it makes the resulting model strong in English but comparatively weaker in other languages, since non-English text gets filtered out at the source.
4. Deduplication and PII removal — duplicate pages are collapsed, and personally identifiable information (addresses, national ID numbers, etc.) is detected and stripped.

After this filtering, FineWeb's full text-only dataset comes out to about 44 TB of disk space — and, once tokenized (see below), around 15 trillion tokens. That's the actual starting material for pretraining.

A curious side effect. After ChatGPT launched in late 2022, researchers doing this kind of web-scale filtering started noticing an uptick in AI-generated ("synthetic") text inside fresh web crawls — visible as spikes in words and phrases chatbots overuse, like "delve" and "as a large language model." So far there's no evidence this is hurting model performance (some have speculated it might even help), but it's a sign of how quickly LLM output is finding its way back into the training data.

You can download FineWeb yourself: huggingface.co/datasets/HuggingFaceFW/fineweb.

Bottom line: this stage produces a huge, cleaned pile of text — no training happens yet, it's pure data preparation. Everything the model will later "know" has to come from somewhere in this pile.

Tokenization

Neural networks don't take raw text as input — they take a sequence of numbers. Tokenization is the process that turns text into that sequence, and it's worth understanding why it works the way it does, because a lot of surprising LLM behavior (spelling mistakes, arithmetic errors) traces directly back to it.

What is it, and why do we need it? A neural network expects two things from its input: a one-dimensional sequence of symbols (language is linear), where the number of distinct possible symbols is finite. The naive way to satisfy this is to encode text as raw bits via UTF-8 — you get a sequence made of just two symbols, 0 and 1. That satisfies "finite symbols," but the sequence becomes extremely long, and long sequences are expensive for a Transformer to process.

So there's a trade-off between vocabulary size and sequence length: fewer symbols means longer sequences; more symbols means shorter sequences, at the cost of a bigger vocabulary to manage. Character-level (or bit-level) tokenization sits at one extreme — tiny vocabulary, huge sequences. Whole-word tokenization sits at the other — short sequences, but a vocabulary that can never fully cover every possible word, misspelling, or language.

How it actually works.

1. Group the raw bits into bytes (8 bits at a time). This immediately gives 2⁸ = 256 possible symbols and cuts the sequence length by 8×. It helps to think of these 256 byte-values less as numbers (which invites thinking of them as ordered — "bigger" or "smaller") and more as 256 arbitrary symbols, closer to emoji.
2. Apply Byte Pair Encoding (BPE): scan the byte sequence for the most frequently occurring adjacent pair (e.g., byte 116 followed by byte 32), and mint a new symbol for that pair (e.g., ID 256), replacing every occurrence of the pair with the new symbol. Repeat this merging process over and over — each merge shrinks the sequence a little further and grows the vocabulary by one symbol.
3. Stop once the vocabulary reaches a target size, chosen heuristically. GPT-4's tokenizer uses a vocabulary of about 100,000 symbols (100,277, to be exact) — a middle ground between raw bytes and whole words. These symbols are called tokens, and this whole process is tokenization.

You can see this in action with GPT-4's actual tokenizer (cl100k_base) at tiktokenizer.vercel.app: the string "hello world" comes out as exactly 2 tokens — 15339 for "hello" and 1917 for " world" (note the token includes the leading space). Neither token is a "letter" or a "word" in the way we normally think about them — they're just the ~100K chunks that happened to fall out of the BPE merging process when run over internet-scale text.

Once FineWeb's 44 TB of text is run through this tokenizer, it comes out to about 15 trillion tokens — the actual unit pretraining is measured in.

Bottom line: tokenization compresses text into a sequence of ~100K possible symbols using BPE, balancing vocabulary size against sequence length. The model never sees letters or words directly — it sees these token IDs — which is the root cause of some of its stranger failure modes (more on this later).

Want to go deeper?

• Play with the real tiktokenizer

Neural Network I/O

Once text is tokenized into a sequence of token IDs, training comes down to one task, repeated at enormous scale: predict the next token.

• Input: a window of tokens taken from the training data — the "context window" (also called the token window or sequence length), typically some fixed maximum like 4,000, 8,000, or 16,000 tokens.
• Output: for the given window, the network produces a probability distribution over the entire vocabulary (~100,277 tokens) — i.e., "how likely is each possible token to come next?"

Concretely: feed in a window of, say, 4 tokens. Before any training, the network's output is close to a uniform random guess — every one of the 100,277 tokens gets a similarly small probability. If the actual next token in the training data is, say, token 3962 (" Post"), training nudges the weights so the probability assigned to token 3962 goes up slightly (e.g. from 3% to 4%), while the probabilities assigned to all the other tokens go down slightly to compensate. This single input-window → correct-next-token comparison, and the resulting weight nudge, is one training step.

This doesn't happen one example at a time — the network processes huge batches of these input→next-token examples in parallel across the entire dataset, gradually adjusting its parameters (weights) so its predicted distribution matches the real statistics of the training text as closely as possible. (This iterative weight-adjustment process — nudge the weights in the direction that reduces prediction error, repeated billions of times — is what's formally called gradient descent, with the per-weight adjustment computed via backpropagation.)

A longer context window lets the model condition its prediction on more prior text, but it also increases computational cost, since a Transformer's cost grows with sequence length.

Bottom line: training a next-token predictor is: show it a window of tokens, ask it to guess the next one, compare the guess to the real answer, nudge the weights slightly toward the correct answer, and repeat billions of times. This "next-token prediction" framing is the single most important mental model for understanding LLMs — everything later (hallucination, reasoning, chat behavior) traces back to it.

Neural Network Internals

So what is this "neural network" mechanically doing between input and output?

Intuition: it's one enormous, fixed mathematical expression — inputs go in, numbers get multiplied, added, and passed through nonlinear functions many times over, and predictions come out. Nothing about it resembles step-by-step human reasoning; it's closer to evaluating a very large formula.

Technical view:

1. Embeddings. The input token IDs (say, a window of ~8,000 of them) are first converted into embedding vectors — each token ID is mapped to a learned vector of numbers the network can actually do arithmetic with. This mapping is itself one of the things learned during training.
2. The Transformer. These embedding vectors, together with the network's billions of learned parameters (weights), are run through a fixed, repeating sequence of basic operations — matrix multiplication, addition, and exponential functions — stacked across many layers. This architecture is called a Transformer, and it's the same underlying architecture behind GPT, Llama, Claude, and essentially every modern LLM.
3. Logits and softmax. At the very end, the network produces one raw score — a logit — for every token in the vocabulary (~100,277 of them). These raw scores aren't yet probabilities (they can be any real number, positive or negative); a softmax function converts them into a proper probability distribution that sums to 1, giving the final "how likely is each token to come next" output.

You can watch this whole process happen, layer by layer, in an interactive 3D visualization at bbycroft.net/llm — it includes a small example network (nanoGPT, ~85,584 parameters) small enough to inspect directly, alongside full-size GPT models.

A useful nuance: the weights aren't hand-designed by engineers. They start out random and are gradually tuned by the training process described above until the giant expression happens to produce good next-token predictions. And unlike a biological neuron, which has memory and internal state, each "neuron" in this network is a stateless mathematical unit — nothing persists between one forward pass and the next except what's explicitly re-fed as input tokens. It's more accurate to think of this as a synthetic mathematical structure than as a brain.

Bottom line: an LLM isn't "thinking" in a humanlike sense under the hood — it's a fixed, enormous mathematical function (a Transformer) that maps input token embeddings to output token probabilities via logits and softmax, with billions of tunable weights set through training.

Inference

Training builds the model's weights; inference is the process of actually using the trained model to generate text.

How it works: start with a sequence of tokens — say, just token 91. Feed it through the network, get back a probability distribution over the vocabulary, and pick a next token from it — say, 860. Append 860 to the sequence (now 91, 860), feed that back through the network, and predict again — say, 287. Repeat.

This is called autoregressive generation: each new token is generated one at a time, always conditioned on everything generated so far. There's no lookahead or planning — just repeated next-token prediction, feeding each output back in as new input.

Sampling, not just the top choice. Rather than always taking the single most probable token, the sampler can draw randomly from the probability distribution the model output — so a less likely token occasionally gets picked instead of the top one. This is why asking an LLM the same question twice can give different answers. That randomness is part of what makes LLM output feel creative rather than mechanically repetitive — but it's also part of why models hallucinate: an unlucky sample can send generation down a path that was never actually well-supported by the training data.

Bottom line: inference is the training-time "predict the next token" mechanism, run in a loop — sampling from the model's own output distribution and feeding each result back in as the next input.

GPT-2

GPT-2 (2019) is worth studying in detail because it's the first model where the full modern LLM "recipe" — Transformer architecture, next-token pretraining, the whole pipeline described above — came together in the form that's still used today. Every model since (GPT-4, Llama, etc.) can be thought of as GPT-2 scaled up along every dimension, rather than something fundamentally different:

	GPT-2 (2019)	Typical modern LLM
Parameters	1.6 billion	Tens to hundreds of billions (e.g. Llama 3.1: 405B)
Context length	Up to 1,024 tokens	Up to millions of tokens (e.g. Gemini: 2M)
Training data	~100 billion tokens	Tens of trillions of tokens (FineWeb alone: 15T)

Reproducing GPT-2 today is dramatically cheaper than it was in 2019 — Karpathy's own side project reproduced it for about $672, for three reasons: (1) training data is now much higher quality, (2) hardware is much faster, and (3) training techniques have improved. For a sense of what that training run actually looks like: it runs for 32,000 steps, each step processing about 1 million tokens of data in parallel and taking roughly 7 seconds — and the model's output visibly goes from gibberish to recognizable (if not yet coherent) English within the very first 1% of training.

Bottom line: GPT-2 is small and cheap to reproduce by today's standards, but architecturally it's the same recipe as every model that came after it — bigger data, bigger model, more compute, not a different design.

Open Base Models

Disclaimer: these models don't strictly meet the Open Source Initiative's (OSI) definition of open-source AI. The term "open base models" is used loosely here because the weights are public, even though the training data and full reproducibility usually aren't — "open-weight" is the more accurate label.

A handful of companies train massive base LLMs and release them for free. Releasing a base model means releasing two things: the inference code (the steps needed to run the model and generate text) and the model weights (the billions of trained parameters). Notable examples: OpenAI's GPT-2 and Meta's Llama 3.1 (405B) — both open-weight, neither strictly open-source, since the training data was never released.

A base model is the direct output of pretraining — a next-token prediction machine, not yet aligned to follow instructions or hold a conversation. Because GPT-2 is old, it's more informative to look at how a modern base model like Llama 3.1 behaves before any post-training:

How base models behave

• They just continue text, they don't answer. Ask one a question, and instead of answering, it tends to just continue the question — since that's the statistically likely continuation of question-shaped text on the internet.
• They're a lossy compression of the internet. In the process of getting good at predicting the next token, the model was forced to build an internal model of the world — grammar, facts, tone, translation, reasoning patterns. OpenAI researcher Jason Wei has described this as effectively multi-task learning: the only way to get good at "predict the next word" across the entire internet is to also get good at all the sub-skills embedded in that text. So even in raw prompt-completion mode, a base model can still surface real information — prompted with "10 landmarks to see in Paris:", it will generate a genuinely useful list, despite never having been trained on a QA format.
• But output is stochastic — closer to dreaming than retrieving. Base model output isn't looked up from a database; it's regenerated fresh, probabilistically, every time you run it. Karpathy describes this as closer to "dreaming" about internet documents than recalling them.
• They can regurgitate. High-quality, frequently-seen sources (like Wikipedia) may have been trained on multiple times, and the model can end up reciting them near-verbatim — answering by memorized repetition rather than understanding. This is called regurgitation, and it's generally not the behavior you want.
• They hallucinate hard past their knowledge cutoff. Ask about anything after the model's training data ends (Llama 3's cutoff is end of 2023) and it will confidently make something up, since nothing in its training data addresses it.

They're already useful, sort of

Even without any fine-tuning, a base model can be steered with the right prompting, via in-context (few-shot) learning: showing it a repeated pattern in the prompt — say, several Human: ... / Assistant: ... exchanges, or a few translation examples — is often enough to get it to continue that pattern for the next exchange, purely by matching the shape of the prompt rather than any explicit instruction-following training.

But a base model, underneath all of this, is still just an (expensive) stochastic autocomplete engine. It still needs a post-training stage to become a genuinely reliable assistant.

Want to try one yourself? Play with the Llama 3 405B base model here.

Bottom line: a base model's parameters encode a genuinely useful — but fuzzy and lossy — compression of the internet. It's already steerable with the right prompting, but it still needs a post-training / fine-tuning stage to become a genuinely useful assistant.

Part 2: Supervised Fine-Tuning (SFT) — Teaching the Model to Be an Assistant

So far, we've only looked at base models — the direct output of pretraining. A base model has absorbed a huge amount of world knowledge, but on its own it's just an autocomplete engine: prompted with a question, it's just as likely to continue the question as answer it, and it hallucinates aggressively, confidently generating plausible-sounding but fabricated text. Turning it into something like ChatGPT — a model that reliably answers questions, follows instructions, and knows when to say "I don't know" — requires a second stage: post-training, starting with supervised fine-tuning (SFT).

Supervised Fine-Tuning (SFT)

Post-training swaps the internet-scale pretraining corpus for a much smaller, curated set of human/assistant conversations — turning a raw next-token predictor into something that can hold a multi-turn dialogue, take on a consistent "personality," and know when to refuse a request.

• The architecture and training algorithm don't change at all — it's still next-token prediction under the hood. Only the dataset changes.
• The time cost is wildly asymmetric: pretraining runs for months on thousands of GPUs; post-training can take as little as ~3 hours, since the curated dataset is tiny by comparison.

Data: Conversations

To let a single 1-D token stream represent a structured back-and-forth, models use chat templates — special tokens that never appeared during pretraining, added specifically to mark who's "speaking" and where each turn starts and ends. Example (gpt-4o style):

<|im_start|>system<|im_sep|>You are a helpful assistant<|im_end|>
<|im_start|>user<|im_sep|>What is 4 + 4?<|im_end|>
<|im_start|>assistant<|im_sep|>4 + 4 = 8<|im_end|>

<|im_start|> and <|im_end|> bookend each turn ("im" reportedly stands for "imaginary monologue"). At inference time, the model just resumes ordinary autoregression right after <|im_start|>assistant<|im_sep|> — it's still the same 1-D sequence trick as pretraining, just with new tokens sprinkled in. Play with this yourself at tiktokenizer.

• OpenAI's InstructGPT paper is the origin story for this approach: they hired around 40 labelers (via Upwork and ScaleAI) who wrote both the prompts and the "ideal" answers themselves, following labeling instructions hundreds of pages long that define what "helpful, truthful, and harmless" means in practice. OpenAI never released that dataset, but the open-source community reproduced the idea — e.g. OASST1.
• The current trend is to move away from all-human authorship: use other LLMs to generate synthetic conversations (e.g. UltraChat), with humans curating rather than writing every pair from scratch. This is how modern post-training datasets cover such a broad range of topics without human labor scaling linearly.

Bottom line: talking to ChatGPT isn't talking to "the internet's knowledge" directly — it's talking to a simulation of what an expert human labeler, steeped in hundreds of pages of "helpful, truthful, harmless" instructions, would have written as the ideal answer. The model learned this persona entirely by example (statistical imitation of the dataset), not through any rule-based programming.

Hallucinations, Tool Use, and Memory

One of the most consequential side effects of SFT: hallucination — confidently stating made-up information.

• Why it happens: human labelers write their "ideal" answers in a confident tone, since they either know the answer or looked it up first. The model imitates that confident tone indiscriminately — including for things it never actually learned, like a made-up name ("Who is Orson Kovacs?"). Early models such as falcon-7b-instruct show this starkly: ask the same made-up question repeatedly and you get a different fabricated answer each time.

• Meta's fix (from the Llama 3 paper) — teach the model to recognize its own knowledge boundary via automated interrogation:

  1. Pull a snippet from training data (e.g. Wikipedia) and have the model itself generate a factual question about it.
  2. Ask the model that question multiple times.
  3. Check whether the answers are consistent and correct.
  4. For questions the model gets wrong or answers inconsistently, add "I'm sorry, I don't believe I know" as the target answer in the Instruct dataset and train on it.

  The intuition: pretraining likely already encodes some internal "do I actually know this?" signal. This process doesn't teach new facts — it teaches the model to connect that existing internal signal to the behavior of actually saying "I don't know," instead of bluffing.

• An even better fix than refusing: tools. Train the model to recognize when it should search instead of guess, using special tokens: it emits <SEARCH_START>query<SEARCH_END>, the inference program pauses generation, runs the real search, injects the results into the context window, and only then lets the model continue generating — now grounded in real information.

• Working memory vs. long-term memory: the model's trained-in parameters behave like a vague, hazy recollection (something remembered from a month ago), while anything placed directly in the context window is immediate working memory. This is exactly why RAG works so well, and why pasting the actual text you want summarized into the prompt beats asking the model to recall it from "memory" (e.g. paste chapter 1 of Pride and Prejudice rather than just asking it to summarize chapter 1 from recall).

Bottom line: hallucination isn't a mysterious bug — it's the predictable outcome of teaching a probabilistic token generator to always answer confidently. Both fixes (calibrated refusal and tool use) are just more training data, not architecture changes.

Knowledge of Self

Ask a base or lightly-tuned model "who are you?" and it'll often answer "I'm ChatGPT, built by OpenAI" — even when it has nothing to do with OpenAI. That's not evidence of copying; it's a predictable consequence of the training data.

• Pretraining data is saturated with text where "who are you" → "I'm ChatGPT made by OpenAI," so absent explicit correction, that's simply the statistically likely completion — and a model giving that answer isn't proof OpenAI made it.
• Two ways developers override this:
  1. Hardcode identity into the SFT dataset — e.g. the Olmo-2 project baked in answers to 240 self-referential prompts.
  2. Inject identity via the system message — name, creator, knowledge cutoff, etc., reasserted at the start of every conversation.
• Without one of these interventions, a model has no innate self-knowledge — it defaults to whatever "AI assistant" persona is statistically dominant in its training data.

Bottom line: an LLM's sense of "self" isn't introspection — it's just another learned behavior. Leave it untrained or unprompted, and it defaults to mimicking the most common AI-assistant text pattern it has seen.

Models Need Tokens to Think

LLMs generate one token at a time, and each token gets only a fixed, finite amount of computation to produce it. That constraint has a real consequence for how a model's answers should be structured.

Compare:

• Bad: "The answer is $3." — the model has to compute the final answer within the compute budget of a single token, with no room to "show its work" first.

• Good: "The total cost of the oranges is $4. 13 - 4 = 9, so the cost of the 3 apples is $9. 9/3 = 3, so each apple costs $3." — unrolling the reasoning across many tokens spreads out the computation and is far less likely to be wrong.

• This is the actual mechanical reason the "let's think step by step" prompting trick worked so well on early ChatGPT — and why modern models are now trained by default to answer this way. It's less "for the user's benefit" and more that it's how the model reliably reaches a correct answer at all.

• You can see this constraint directly: force a model to "answer in one token" and it nails simple arithmetic but starts failing as soon as the numbers get bigger.

• Tokens aren't characters — which is also why models are weak at tasks like counting letters (the classic "how many r's in strawberry" problem). The model never sees individual letters, only token chunks, so letter-counting is fundamentally mismatched with how it perceives text.

• For math, logic, and counting tasks, the reliable fix isn't cleverer prompting — it's letting the model reach for code (or another external tool) instead of relying purely on its own token-by-token arithmetic.

Bottom line: a model's step-by-step "reasoning" is a workaround for a hard constraint (finite compute per token), not a stylistic flourish — and understanding that tells you exactly when to trust a quick answer versus when to demand step-by-step work or an external tool.

Part 3: Reinforcement Learning — Learning by Practice

SFT gets a model to imitate good answers, but imitation has a ceiling: we're not the LLM, so we don't actually know which intermediate steps genuinely help it reach a correct answer — what's an easy leap for a human writing a worked solution can be a hard one for the model's own computation, and vice versa. RL is the third stage that fixes this, and it maps neatly onto how students actually learn a subject:

• Pretraining = reading the textbook exposition (background knowledge).
• SFT = studying worked problems with an expert's solution attached (imitation).
• RL = solving practice problems on your own, learning by trial and error from whether you got the final answer right.

Although RL is technically part of "post-training," it's substantial enough to be treated as its own third major stage — at a company like OpenAI, separate teams own pretraining, SFT data, and RL respectively.

How RL Works

The fix for "we don't know the ideal intermediate steps" is to stop trying to specify them at all: give the model only the question and the final correct answer, and let it discover a working solution path entirely on its own.

• For a single prompt, the model generates a large batch of candidate solutions itself (sometimes millions across training) — completely unsupervised, no human involved.
• Each candidate is checked against the known final answer. In the example above, 15 solutions were generated and only 4 reached the correct answer of $3.
• Among the correct ones, pick the best by some simple heuristic (e.g. the shortest correct solution), and train the model to reinforce that path.
• Repeat this across huge numbers of problems, thousands of times over.

Because every intermediate solution is self-generated, no human labeling is required for this stage — which is exactly why it can run at a scale SFT never could. Think of SFT as giving the model a good initialization, and RL as the process that lets it discover, on its own, the optimal token sequence for arriving at correct answers — sequences a human might never have written by hand.

Bottom line: RL trades "here's how to do it" (SFT) for "here's whether you got it right" (RL) — and it turns out that weaker signal, applied at massive self-play scale, teaches the model more than imitation alone ever could.

DeepSeek-R1: RL Out in the Open

Pretraining and SFT are now fairly standardized industry-wide, but the RL stage is still early and mostly kept private — companies like OpenAI do plenty of RL research but rarely publish details. That's what made DeepSeek-R1 (Jan 2025) significant: a fully public account of an RL recipe for reasoning.

• On the AIME math benchmark, accuracy climbed steadily as the model went through thousands of RL training updates.
• More telling than the accuracy curve: average response length grew over training. Nobody told the model to write longer answers — it discovered on its own that re-evaluating a problem from multiple angles improved accuracy, and that naturally takes more tokens.
• You can see this directly in the raw output above: the model catches itself mid-solution — "Wait, wait. Wait. That's an aha moment I can flag here" — and re-derives the answer before presenting a clean, final version. Karpathy calls this an emergent "Cognitive Strategy": never explicitly programmed, discovered purely through RL.
• It's fully open-weight, so this can be verified and run by anyone (e.g. via together.ai) rather than taken on faith.
• Mapped onto consumer products: models labeled "Reasoning" (o1, o3-mini, DeepSeek-R1, Gemini 2.0 Flash Thinking) are RL models; GPT-4o is essentially an SFT model without this extra stage. OpenAI shows only a re-summarized chain of thought rather than the raw trace — reportedly to make distillation (copying their RL behavior) harder.

Bottom line: RL doesn't just push accuracy up — it can produce genuinely emergent problem-solving behavior nobody explicitly taught, and DeepSeek-R1 is the first widely public proof of this for language models.

AlphaGo and "Move 37"

This same SFT-then-RL pattern already played out publicly once before, in AlphaGo (2016–2017), and the chart above is the reason Karpathy uses it as the canonical proof that RL can exceed human performance:

• A model trained purely by imitating human expert games (the SFT-equivalent, purple line) plateaus below Lee Sedol's level — imitation can't beat the humans it's imitating.
• A model trained further with RL (blue line) — playing itself over and over and reinforcing winning strategies, no human game data involved — surpasses AlphaGo Lee, discovering moves no human had ever played. The most famous example, "Move 37," is estimated at roughly a 1-in-10,000 chance of being played by a human.
• The expectation is that language models will eventually hit their own "Move 37" moments — new analogies, new reasoning techniques, or possibly even an internal "language of thinking" no human would have devised. (Linus Torvalds has floated a similar idea: that AI might end up writing code in a form humans can no longer easily read.)

Bottom line: imitation (SFT) caps out at human-level by definition, since it's trained to copy humans; RL is the mechanism that can push a model past human performance, because it optimizes directly against the outcome instead of against a human's example.

RLHF: Reinforcement Learning from Human Feedback

Plain RL as described above only works in verifiable domains — problems like math, where an answer can be checked automatically (e.g. by another LLM acting as a judge). But what about domains with no ground truth, like "write a funny joke"? There's no automatic checker for funny.

• Naive approach: have a human score every rollout. At realistic training scale — say 1,000 updates × 1,000 prompts × 1,000 rollouts — that's on the order of 1 billion human ratings. Completely impractical (and would mean a human reading a billion mediocre AI jokes).
• RLHF's fix:
  1. Take 1,000 prompts, generate just 5 rollouts each, and have a human rank them best-to-worst (~5,000 ratings instead of a billion).
  2. Train a separate reward model — a neural net that takes (prompt, generated answer) and outputs a single score simulating human preference.
  3. Run RL as usual, substituting the reward model's score for actual human feedback.

• This works because ranking is much easier for a human than generating — the "discriminator-generator gap." It's far easier to say which of 5 pelican jokes is funniest than to write a great one from scratch.
• Upside: RL becomes usable in essentially any domain, verifiable or not, and it also helps reduce hallucinations.
• Downside: the reward model is only a lossy simulation of human preference, not the real thing. Run RL against it indefinitely and the model eventually finds an adversarial loophole — reward hacking — and quality collapses (Karpathy's example: the model discovers that spamming "the the the the..." scores unexpectedly high). Because of this failure mode, RLHF is typically run for only a few hundred update steps and then stopped and shipped — closer to a limited fine-tuning pass than open-ended RL.

Bottom line: RLHF trades true, unlimited RL for practicality — it lets reinforcement learning apply to subjective, unverifiable domains by training a cheap-to-query stand-in for human judgment, but that stand-in is gameable, so it can only be trusted for a limited amount of training before it breaks down.

What Comes Next

Pretraining, SFT, and RL are the three stages that produce a model like ChatGPT today. The rest is a set of practical questions: where is this all heading, how do you keep track of a field moving this fast, and where do you actually go to use these models?

Future Developments

A few directions worth watching:

• Multimodality. Extending beyond text doesn't require a different architecture — audio and video can be converted into a 1-dimensional sequence, the same kind of representation tokens already are, and fed through the same Transformer. In effect, it's just adding new kinds of tokens to the vocabulary, not building a new kind of model.
• Agents. Today, a human orchestrates AI directly — running one task, checking the result, running the next. The expected shift is toward agents that manage multi-step tasks coherently on their own, with humans supervising at a higher level: reviewing intermediate progress reports rather than issuing every individual instruction. Karpathy draws an analogy to manufacturing's human-to-robot ratio — expect a human-to-agent ratio to become a meaningful metric in the same way.
• Pervasive and invisible. Rather than being a distinct product you consciously open, AI is expected to increasingly disappear into existing tools and workflows.
• Computer use. Instead of interacting with software only through dedicated APIs, models are increasingly being trained to operate a computer's actual interface directly — clicking, typing, and navigating the way a human would, rather than calling a function built specifically for them.
• Test-Time Training (TTT). This one is more of an open research direction than a shipped feature. The pipeline covered in this article — pretraining → SFT → RL — produces a fixed set of parameters; at inference time those parameters are frozen, and the only way the model incorporates new information is through whatever fits in its context window. But the context window is a finite resource, and that limitation gets more severe once models are working with multimodal input (video, audio) rather than just text. Test-Time Training is a proposed direction for letting a model's parameters themselves adapt at inference time, instead of relying solely on context — as of this lecture, it remains an active research question rather than a settled technique.

Bottom line: none of these directions require a new architecture — multimodality and agentic behavior both fall out of the same "turn everything into tokens, run it through a Transformer" recipe already covered in this article. The genuinely open problem is what happens once the context window alone isn't enough.

Resources: Keeping Track of a Fast-Moving Field

Two ways to stay current that the lecture calls out directly:

• LM Arena (formerly Chatbot Arena) — a live leaderboard that ranks models by head-to-head human preference votes, rather than by a single static benchmark score.
• AI News — a newsletter archive that aggregates day-to-day developments across the field.

One detail worth calling out on the licensing side: DeepSeek's models stand out for shipping under a permissive MIT license — notably more open than the licensing terms attached to many other frontier-adjacent open-weight releases.

Accessing Models: Where to Actually Use Them

Depending on what you're looking for:

• Instruction-tuned (chat/assistant) models — the SFT/RL models this article mostly discusses — are broadly available through together.ai.
• Base models — the raw, pre-post-training checkpoints discussed in the Open Base Models section — are mostly accessed through Hyperbolic, the same platform used earlier in this article to try Llama 3 405B's base model directly.
• Running a model locally — if you want a smaller, distilled model running on your own machine rather than calling a hosted API, LM Studio is a solid option.

Bottom line: which of these you reach for depends on what you're actually trying to do — LM Arena and AI News for tracking what's good right now, together.ai and Hyperbolic for using models without hosting anything yourself, and LM Studio when you want a model running entirely on your own hardware.