Wieszcz-XIX: A 3.1-Billion-Word Polish Corpus From Before 1918 Trains Models From Scratch

Historical Polish lacks machine-readable data. For 1800–1918, there are only about a million words of annotated text, and the rest sits in scans with OCR of varying quality. The authors of Wieszcz-XIX built a corpus from those scans: 6.75 billion tokens, or about 3.1 billion words, in 294 369 documents. Most of the documents are issues of periodicals. The corpus is more than three orders of magnitude larger than the annotated one. The team trained a family of models from 47M to 349M parameters on it from scratch, then tested how well the models "don't know" what happened after 1918.

Corpus: dirty OCR, but defect-aware

The sources are Wolne Lektury and the Internet Archive. The pipeline filters the texts, removes duplicates, checks them for leakage from after 1918, and splits the data into samples at the document level. The authors do not hide the defects. They measure them:

• 6.75 billion tokens, about 3.1 billion words, 294 369 documents;
• the share of post-war text remaining in the training corpus is 0.04–0.38% of bytes, and the remainder was found in a transcribed source;
• on a manually corrected sample, the character error rate is 0.68% where the text is legible;
• 45% of the sampled fragments cannot be corrected;
• near-identical duplicates are removed.

The published corpus matches the training corpus document for document, so the results can be verified on the same data.

Models without a future

All the models are decoder-only and trained from scratch, without fine-tuning a modern base model. The authors compared them with two modern Polish base models, one of which is noticeably larger. The 349M model shows a "crossover" with them, and so does the 107M model when compared with a model of similar size. Post-1918 vocabulary costs the historical models about 3.1 bits per byte more than period vocabulary. The modern models show no such gap. Period words cost the historical models less than they cost the modern ones.

There is also a qualitative effect. Given a text from the period, the model preserves its spelling, while modern models do so only partially. This matters for working with sources: the model does not "modernize" the spelling as it continues the text.

Another result concerns resources. Adding parameters gives roughly twice the gain of a second pass over the same data. With 3.1 billion words, that is an argument for scaling the model rather than repeating epochs.

A warning and practical use

The authors explicitly warn that the models reproduce the prejudices of the era, including antisemitic statements. The data has not been cleaned of this and must not be presented as a neutral source.

For researchers, the practical value lies in three things. The corpus, code and weights are open, so they can serve as a base for historical research, such as spelling normalization, search across press archives, or improving the OCR itself. The models are small, up to 349M parameters, and run on modest hardware. The method of measuring "temporal confinement" by the bits-per-byte difference between period vocabulary and later vocabulary can be transferred to other languages and periods where digitized periodicals exist. The limitation is obvious: almost half of the checked fragments cannot be corrected by hand, so quality depends on the source scans, and anyone using the models for publications needs to verify the output against the sources.