How to Train a Text Neural Network at Home: habrGPT and an fp8 vs bf16 Comparison

On Habr, a developer described training habrGPT, a text language model with 500 million parameters, from scratch on the platform's article corpus, using a home GPU. The base is Andrej Karpathy's nanochat, and the comparison centers on two training formats: fp8 and bf16.

What nanochat is and why adapt it for home use

nanochat is an open-source project by Andrej Karpathy, former AI director at Tesla and co-founder of OpenAI. Its core pitch: "train your own ChatGPT for $100." The original pipeline is designed for a cluster of eight NVIDIA H100 GPUs with 80 GB of video memory each, professional-grade hardware that most enthusiast developers cannot access.

• The habrGPT model: 0.5 billion parameters (0.5B), trained from scratch, not fine-tuned from an existing model
• Corpus: Habr articles; Habr is the largest Russian-language IT community
• Experiment hardware: a home GPU, significantly weaker than 8×H100 80 GB
• Formats compared: fp8 vs bf16
• Minimum goal: coherent text generation

The author tests how realistic the "$100 and your own ChatGPT" slogan is outside professional data centers, and how much quality degrades when the setup is adapted to home hardware.

Is the Habr corpus enough to train a language model?

Habr is one of the largest Russian-language technical platforms, with tens of thousands of articles on programming, DevOps, machine learning and information security. Using it as a training corpus yields a model with a distinct "Habr" style: technically literate, heavy on code and specialized terminology.

The main question of the experiment is not style but data volume. Language models trained from scratch need a huge amount of text even to learn basic grammar. Habr articles are a topically narrow, high-quality corpus, but considerably smaller than the web corpora typically used to train LLMs. The authors set a modest bar: check whether there is enough material for the model to "at least string a couple of words together."

Fp8 vs bf16: what the difference means for home training

Fp8 and bf16 are floating-point number formats. They directly determine computational precision and video memory use when training neural networks.

Bf16 (bfloat16) has become the standard for most modern LLM training: it is accurate enough and natively supported by NVIDIA GPUs starting with the Ampere architecture. Fp8 is a more aggressive format that uses half as much video memory. On the same home GPU, that allows a larger model to fit or epochs to run faster, at the cost of potentially losing gradient precision.

For home training, fp8 is especially attractive: it reduces video

memory requirements exactly where memory is scarcest.

Comparing fp8 and bf16 on the Russian-language Habr corpus will show how aggressive precision reduction affects the coherence and grammar of generated text, and whether the saved memory is worth the loss in quality.

What this means

The habrGPT experiment shows that training a small language model from scratch is no longer the exclusive privilege of cloud clusters. Karpathy's nanochat combined with fp8 puts the task within reach of a single developer with a consumer GPU, though at the cost of compromises in computational precision and training data volume. The barrier to practical LLM research keeps falling.

Can you train a text neural network without the cloud?

Yes. The article describes training habrGPT on a home GPU instead of a professional cloud cluster.

What is nanochat?

An open-source project by Andrej Karpathy (former AI director at Tesla and co-founder of OpenAI) for training language models on a budget of about $100.