Era II · 2018–2020
The Scaling-Laws Era
By C.W. Jameson · Published 19 May 2026 · Last reviewed 19 May 2026
Between 2018 and 2020 the field stopped tinkering with architectures and started buying compute. Kaplan, McCandlish, and a small team at OpenAI published the scaling-laws paper in January 2020 that gave the trend a name. GPT-3, released that summer, validated it. The lesson was that capability rose smoothly and predictably with parameters, training data, and compute.
GPT-2
Released in stages through 2019 — 124M, 355M, 774M, 1.5B parameters. OpenAI initially withheld the 1.5B weights citing misuse risk. The risk did not materialise; the weights were released in November 2019. The capability shocked the field. The headline demo was the unicorn-in-the-Andes story.
Scaling laws
Kaplan et al., 'Scaling Laws for Neural Language Models', arXiv 2001.08361, published 23 January 2020. The paper described power-law relationships between loss and compute, model size, and data. The practical implication was that the right way to improve capability was to spend more, not to invent.
GPT-3
175 billion parameters, released June 2020. The API was invitation-only for the first year. The few-shot-prompting demonstration in the technical report changed what most people thought a language model was.
Signature models of the era
- GPT-2 (1.5B)
- GPT-3 (175B)
- T5
- Megatron-LM
Technical shifts
- Few-shot prompting replaces task-specific fine-tuning for many use cases
- Compute budgets become the primary lever
- The first $1M+ training runs
Market shifts
- OpenAI's API launches; commercial LLMs become a product category
- Anthropic founded in 2021 by Dario Amodei and team departing OpenAI
Authentication — is the document from this era?
| Tell | Meaning |
|---|---|
| Reference to 'few-shot prompting' as a novelty | Document dates to 2020 or shortly after. |
Primary sources
- [1] Kaplan et al.: Scaling Laws — 2020-01-23
- [2] Brown et al.: GPT-3 — 2020-05-28
From the Almanac shop
The AI Eras — Pocket Field Guide
Ten eras of AI on a single foldable. The Almanac in your pocket.
$19 — Coming soon