habrGPT: training a 0.5B LLM from scratch on Habr articles — Karpathy’s nanochat and an fp8 vs bf16 comparison

A developer on Habr has published an experiment: training a language model called habrGPT with 500 million parameters from scratch on a corpus of articles from the platform — using a home GPU instead of a professional cluster. The work is based on Andrey Karpathy's nanochat, with the comparison centered on fp8 and bf16 training formats.

What is nanochat and why adapt it at home

nanochat is an open-source project by Andrey Karpathy, former AI Director at Tesla and co-founder of OpenAI. Its core thesis: "train your own ChatGPT for $100". In the original pipeline, training is designed for a cluster of eight NVIDIA H100 GPUs with 80 GB of video memory each — professional-grade hardware inaccessible to most enthusiast developers.

• habrGPT model: 0.5 billion parameters (0.5B), trained from scratch — without fine-tuning an existing model
• Corpus: articles from Habr — the largest Russian-language IT community
• Experiment hardware: a home GPU, significantly weaker than 8×H100 80 GB
• Formats compared: fp8 versus bf16
• Minimal goal: achieve coherent text generation

The author tests how realistic the "$100 and your own ChatGPT" slogan is outside professional datacenters and how much quality degrades when adapted for home hardware.

Is the Habr corpus enough for training a language model

Habr is one of the largest Russian-language technical platforms with tens of thousands of articles on programming, DevOps, machine learning, and information security. Using it as a training corpus means obtaining a model with a pronounced "Habr style": technically competent, saturated with code, and specialized terminology.

The main question of the experiment is not style, but data volume. Language models trained from scratch require a massive amount of text even to master basic grammar. Habr articles are a thematically specific and high-quality corpus, but significantly smaller in size than web corpora typically used to train LLMs. The authors set a modest bar: check whether there is enough material for the model to "at least string a few words together."

Fp8 versus bf16: what's the difference for home training

Fp8 and bf16 are floating-point number storage formats that directly affect computational accuracy and GPU memory consumption during neural network training.

Bf16 (bfloat16) has become the standard for most modern LLM training: it is sufficiently accurate and natively supported by NVIDIA GPUs starting with the Ampere architecture. Fp8 is a more aggressive format that consumes half the video memory. This allows fitting a larger model on the same home GPU or accelerating epoch passes — at the cost of potential loss of gradient precision.

For home training, fp8 is especially attractive: it reduces video

memory requirements precisely where it is most scarce.

Comparing fp8 and bf16 on the Russian-language Habr corpus will show how aggressive precision reduction affects the coherence and grammar of generated text — and whether the saved memory is worth the loss in quality.

What this means

The habrGPT experiment demonstrates: training a small language model from scratch has ceased to be an exclusive privilege of cloud clusters. Karpathy's nanochat combined with fp8 format makes this task feasible for a single developer with a consumer GPU — albeit at the cost of compromises in computational accuracy and training data volume. The barrier to entry for practical LLM research continues to lower.