Talkie-1930: Researchers Release a 13B Model With No Knowledge of the Internet or World War II

Talkie-1930 is an unusual experiment that tries to put a language model back into the intellectual context of the early 20th century. It is an open 13-billion-parameter model trained only on English-language texts published before December 31, 1930, so it knows nothing about the internet, smartphones, or World War II as a historical fact. Instead of yet another chatbot that talks about everything, the researchers got a clean testbed for studying how a model reasons, predicts and generalizes when its worldview is locked to a single point in history.

The project was presented by a team of Nick Levine, David Duvenaud and Alec Radford. The base version, talkie-1930-13b-base, was trained on 260 billion tokens from books, newspapers, magazines, scientific papers, patents and court records. A separate conversational checkpoint, talkie-1930-13b-it, is fine-tuned for dialogue. Both versions are published with open weights under the Apache 2.0 license.

The authors also built a "modern twin" with the same architecture and the same compute budget, but trained on the FineWeb corpus. This lets them compare nearly identical systems trained on different kinds of data, rather than abstract models from different eras.

Talkie-1930's main value lies not in its retro style but in its research cleanliness. Modern LLMs almost inevitably suffer from contamination, when test tasks, benchmark fragments or similar data have already leaked into training. A vintage model has far less of this problem by definition: if a benchmark describes events or technologies after 1930, Talkie could not have seen them in advance. That makes it a convenient tool for testing how far a model can actually go beyond its corpus.

For example, the authors tested whether it can write Python code from a few in-context examples, even though neither Python nor digital computers appear in its training data. The results are weak so far, but the fact that some answers are correct suggests the model can borrow the structure of a solution, not just copy memorized patterns.

The team also uses Talkie-1930 as a tool for temporal and historical assessment. In one experiment, the model measured the "surprise" of short descriptions of real events from The New York Times archive. Past the 1930 cutoff, the stories become noticeably less predictable for it, especially events of the 1950s and 1960s. This offers a careful way to study how models "see" the future from the past, and how their forecasting ability changes over long time distances.

Another interesting question is what shapes a model's personality. Almost all modern LLMs descend in one way or another from web data; Talkie breaks that lineage and makes it possible to separate the properties of the language model itself from the quirks of the internet as a training environment.

Technically, the project turned out to be much harder than simple date filtering. The most dangerous risk is temporal leakage: misdated documents, modern editorial introductions to old books, or later footnotes can quietly smuggle knowledge of the future into the corpus. The authors built a document-level anachronism classifier, but admit it is not perfect: early versions of the model knew about Franklin Roosevelt's presidency and the New Deal reforms, and the 13B checkpoint retains some information about World War II, the UN and the postwar arrangement of Germany.

Recognition quality is an equally painful problem. Since no digital publishing pipeline existed in 1930, the whole corpus had to be assembled through OCR. In controlled tests, standard OCR delivered only about 30% of the training effectiveness of human transcription of the same texts; simple cleanup with regular expressions raised that to roughly 70%, but a large gap remained.

To keep the conversational version from picking up modern habits, post-training also had to be built from scratch. Instead of typical instruction datasets, the team extracted instruction-response pairs from historical reference works: etiquette manuals, letter-writing guides, cookbooks, dictionaries, encyclopedias, and collections of fables and poems. The model was then improved with online DPO, using a modern LLM as the judge; by the team's internal evaluation, instruction following rose from 2.0 to 3.4 out of 5.

Next, the authors plan to grow the corpus to more than 1 trillion tokens, extend it beyond English, and release a GPT-3-level vintage model as early as summer 2026.

Bottom line: Talkie-1930 matters not as a nostalgic chatbot but as a laboratory for testing big questions about AI: what a model truly understands, what it has merely memorized, how far it can generalize without hints from the future, and how much the web has shaped the character of modern LLMs. If the project manages to reduce leakage and OCR noise, researchers will gain one of the cleanest tools for studying the limits of generalization in language models.