Free Intensive Nov 21–22Free 2-day hands-on Vibecoding Intensive on Nov 21–22 at 16:00 MSK (Sat–Sun) · Live Cohort 3 starts Nov 23Join Free →Cohort 3 (Nov 23)
ReleaseCode / SDK · ActionableHabr: MTS AI→ original

MWS AI Open-Sources OpenAutoNLU: A Library That Picks One of Three Training Methods for Your Chatbot

To get a chatbot to understand what a customer wants, developers have to manually choose a training method, clean up the labeling, and teach the bot to reject off-topic questions. MWS AI has open-sourced OpenAutoNLU, a library that selects one of three methods based on the number of examples per class, ranging from few-shot to full fine-tuning. If there is no test set, GPT-4o-mini generates one, and the gap from a real test set did not exceed 5%. For a prototype bot with little labeled data, this is a fast start.

AI-processed from Habr: MTS AI; edited by Hamidun News
MWS AI Open-Sources OpenAutoNLU: A Library That Picks One of Three Training Methods for Your Chatbot
Source: Habr: MTS AI. Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceHabr: MTS AI ↗
Editor-in-Chief
Little dataAncSetFit. Medium: SetFit. Lots of data: full fine-tuning.
Key metric #2Off-topic queries are rejected by a threshold tuned on synthetic data.
Key metric #3Export to ONNX, with automatic selection of CUDA, CoreML, or CPU.

Picture a support bot at a bank. It has to work out what the customer wants: block a card, check a limit, or change a plan. This task is called NLU, or natural language understanding. To train such a bot, a developer picks a method, tunes the settings, cleans up messy labeling, and then discovers that the bot confidently answers questions about the weather, which it was never prepared for. Such off-topic queries are called OOD (out of scope) in the literature. The MWS AI team, together with researchers from ITMO and MBZUAI, has open-sourced OpenAutoNLU, a library designed to remove this manual grind from text classification and named entity recognition (NER).

What problem the library solves

The authors, including project lead Grigory Arshinov, point to three weak spots in existing AutoML tools. They either cannot train NLU models out of the box, cannot build good detectors for off-topic queries, or are too complex and require expert knowledge.

The grind looks like this. First, you need to know suitable hyperparameters in advance or search through all combinations and presets, which takes a long time and a lot of compute. Second, labeling in real projects is rarely clean: it contains errors, ambiguous examples, duplicates, and class imbalance. Most AutoML solutions do not check data quality within the pipeline. Third, in production the model receives inputs it never saw during training. Usually such examples are either ignored or require labeling in advance, which is not always possible.

OpenAutoNLU is a Python library with a simple API. The user provides data and starts training, while mode selection, data analysis, balancing, and inference preparation run automatically. The library is already part of the MWS AI Agents Platform toolkit.

How it works, by example

The core idea: the library looks at the class distribution and the minimum number of examples per class, then selects a strategy on its own. If data is very scarce, AncSetFit kicks in: a few-shot method with anchor examples, designed for 2–5 examples per class. With a moderate amount of data, SetFit runs: a sentence transformer with a simple classifier on top. When there are enough examples, the entire transformer is fine-tuned, optionally with parameter search via Optuna. By default, that means ten iterations, stopping after five steps without improvement in the target metric, macro-F1.

Suppose your intent dataset has two examples for some classes and hundreds for others. The system first boosts the small classes through augmentation with Augmentex and, if needed, through an LLM. For few-shot modes, it does the opposite and trims the large classes: to 80 examples for SetFit and to 5 for AncSetFit, so the imbalance does not break training. After that, the library re-evaluates the data and may switch modes.

A separate layer handles label diagnostics. It shows which examples should be relabeled: where the prediction diverges from the label, where the model is uncertain, and which examples contribute only noise. For NER, predictions from several Monte Carlo dropout passes are aggregated and treated as the opinions of different annotators.

For off-topic queries, there are three methods: Marginal Mahalanobis Distance, Centroid Mahalanobis Distance, and MaxProb. The first performs better when the smallest class has at least 81 examples. The library tunes the rejection threshold on synthetic out-of-scope examples, so manual labeling of those is not required, although you can add your own examples.

If there is no test set, an LLM generates one from the training examples. The experiments used GPT-4o-mini: on sets of up to 80 examples per class, the gap between the synthetic and real test sets did not exceed 5%. Any OpenAI-compatible endpoint will work, including a local model.

The output is a package containing an ONNX checkpoint, a tokenizer, a label mapping, and meta.json. The inference manager detects CUDA, CoreML, or CPU on its own and picks a batch size so it does not run out of memory.

Validation and where to be careful

The authors compared the library with AutoIntent, AutoGluon, LightAutoML, and H2O on four intent datasets: banking77, massive, hwu64, and snips. Runs used small samples (5–10 examples per class), medium samples (81–100), and full datasets, with results averaged over three runs. They measured F1-macro, F1-in-scope, and F1-OOD. Tests ran on a machine with an NVIDIA H100 with 80 GB.

The caveats are obvious. These are the authors' own measurements on standard intent benchmarks, and how the library performs on your noisy dataset has to be verified. Reading BIO-format labels for NER is not yet supported directly and is promised for the next release.

The practical takeaway is simple. If you are building a bot, a request router, or entity extraction and have little labeled data, OpenAutoNLU lets you quickly assemble a working baseline with label diagnostics and protection against off-topic queries. A demo is temporarily available at openautonlu.dev, and you can build it yourself by following the instructions in the repository.

Story

Key Questions Answered

What problem the library solves?

The authors, including project lead Grigory Arshinov, point to three weak spots in existing AutoML tools. They either cannot train NLU models out of the box, cannot build good detectors for off-topic queries, or are too complex and require expert knowledge. The grind looks like this. First, you need to know suitable hyperparameters in advance or search through all combinations and presets, which takes a long time and a lot of compute.

How it works, by example?

The core idea: the library looks at the class distribution and the minimum number of examples per class, then selects a strategy on its own. If data is very scarce, AncSetFit kicks in: a few-shot method with anchor examples, designed for 2–5 examples per class. With a moderate amount of data, SetFit runs: a sentence transformer with a simple classifier on top. When there are enough examples, the entire transformer is fine-tuned, optionally with parameter search via Optuna.

What are the key details on Validation and where to be careful?

The authors compared the library with AutoIntent, AutoGluon, LightAutoML, and H2O on four intent datasets: banking77, massive, hwu64, and snips. Runs used small samples (5–10 examples per class), medium samples (81–100), and full datasets, with results averaged over three runs. They measured F1-macro, F1-in-scope, and F1-OOD. Tests ran on a machine with an NVIDIA H100 with 80 GB. The caveats are obvious.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…