MWS AI Open-Sources OpenAutoNLU: A Library That Picks One of Three Training Methods for Your Chatbot
To get a chatbot to understand what a customer wants, developers have to manually choose a training method, clean up the labeling, and teach the bot to reject off-topic questions. MWS AI has open-sourced OpenAutoNLU, a library that selects one of three methods based on the number of examples per class, ranging from few-shot to full fine-tuning. If there is no test set, GPT-4o-mini generates one, and the gap from a real test set did not exceed 5%. For a prototype bot with little labeled data, this is a fast start.
AI-processed from Habr: MTS AI; edited by Hamidun News
Picture a support bot at a bank. It has to work out what the customer wants: block a card, check a limit, or change a plan. This task is called NLU, or natural language understanding. To train such a bot, a developer picks a method, tunes the settings, cleans up messy labeling, and then discovers that the bot confidently answers questions about the weather, which it was never prepared for. Such off-topic queries are called OOD (out of scope) in the literature. The MWS AI team, together with researchers from ITMO and MBZUAI, has open-sourced OpenAutoNLU, a library designed to remove this manual grind from text classification and named entity recognition (NER).
What problem the library solves
The authors, including project lead Grigory Arshinov, point to three weak spots in existing AutoML tools. They either cannot train NLU models out of the box, cannot build good detectors for off-topic queries, or are too complex and require expert knowledge.
The grind looks like this. First, you need to know suitable hyperparameters in advance or search through all combinations and presets, which takes a long time and a lot of compute. Second, labeling in real projects is rarely clean: it contains errors, ambiguous examples, duplicates, and class imbalance. Most AutoML solutions do not check data quality within the pipeline. Third, in production the model receives inputs it never saw during training. Usually such examples are either ignored or require labeling in advance, which is not always possible.
OpenAutoNLU is a Python library with a simple API. The user provides data and starts training, while mode selection, data analysis, balancing, and inference preparation run automatically. The library is already part of the MWS AI Agents Platform toolkit.
How it works, by example
The core idea: the library looks at the class distribution and the minimum number of examples per class, then selects a strategy on its own. If data is very scarce, AncSetFit kicks in: a few-shot method with anchor examples, designed for 2–5 examples per class. With a moderate amount of data, SetFit runs: a sentence transformer with a simple classifier on top. When there are enough examples, the entire transformer is fine-tuned, optionally with parameter search via Optuna. By default, that means ten iterations, stopping after five steps without improvement in the target metric, macro-F1.
Suppose your intent dataset has two examples for some classes and hundreds for others. The system first boosts the small classes through augmentation with Augmentex and, if needed, through an LLM. For few-shot modes, it does the opposite and trims the large classes: to 80 examples for SetFit and to 5 for AncSetFit, so the imbalance does not break training. After that, the library re-evaluates the data and may switch modes.
A separate layer handles label diagnostics. It shows which examples should be relabeled: where the prediction diverges from the label, where the model is uncertain, and which examples contribute only noise. For NER, predictions from several Monte Carlo dropout passes are aggregated and treated as the opinions of different annotators.
For off-topic queries, there are three methods: Marginal Mahalanobis Distance, Centroid Mahalanobis Distance, and MaxProb. The first performs better when the smallest class has at least 81 examples. The library tunes the rejection threshold on synthetic out-of-scope examples, so manual labeling of those is not required, although you can add your own examples.
If there is no test set, an LLM generates one from the training examples. The experiments used GPT-4o-mini: on sets of up to 80 examples per class, the gap between the synthetic and real test sets did not exceed 5%. Any OpenAI-compatible endpoint will work, including a local model.
The output is a package containing an ONNX checkpoint, a tokenizer, a label mapping, and meta.json. The inference manager detects CUDA, CoreML, or CPU on its own and picks a batch size so it does not run out of memory.
Validation and where to be careful
The authors compared the library with AutoIntent, AutoGluon, LightAutoML, and H2O on four intent datasets: banking77, massive, hwu64, and snips. Runs used small samples (5–10 examples per class), medium samples (81–100), and full datasets, with results averaged over three runs. They measured F1-macro, F1-in-scope, and F1-OOD. Tests ran on a machine with an NVIDIA H100 with 80 GB.
The caveats are obvious. These are the authors' own measurements on standard intent benchmarks, and how the library performs on your noisy dataset has to be verified. Reading BIO-format labels for NER is not yet supported directly and is promised for the next release.
The practical takeaway is simple. If you are building a bot, a request router, or entity extraction and have little labeled data, OpenAutoNLU lets you quickly assemble a working baseline with label diagnostics and protection against off-topic queries. A demo is temporarily available at openautonlu.dev, and you can build it yourself by following the instructions in the repository.
- Nov 21, 2026FDE job postings grew 4.6x in a year: what a forward deployed engineer actually does at the client's site
- Nov 21, 2026A swimming team captain handed ChatGPT three scheduled tasks and cut her routine work by a multiple
- Nov 21, 2026OpenAI GPT-6 Cyber Found About 35 Flaws in iPhone, Mac and Other Apple Devices
- Nov 21, 202622 Claude Agents Spent Six Weeks on Black Hole Problems — and Solved Two
Key Questions Answered
What problem the library solves?
The authors, including project lead Grigory Arshinov, point to three weak spots in existing AutoML tools. They either cannot train NLU models out of the box, cannot build good detectors for off-topic queries, or are too complex and require expert knowledge. The grind looks like this. First, you need to know suitable hyperparameters in advance or search through all combinations and presets, which takes a long time and a lot of compute.
How it works, by example?
The core idea: the library looks at the class distribution and the minimum number of examples per class, then selects a strategy on its own. If data is very scarce, AncSetFit kicks in: a few-shot method with anchor examples, designed for 2–5 examples per class. With a moderate amount of data, SetFit runs: a sentence transformer with a simple classifier on top. When there are enough examples, the entire transformer is fine-tuned, optionally with parameter search via Optuna.
What are the key details on Validation and where to be careful?
The authors compared the library with AutoIntent, AutoGluon, LightAutoML, and H2O on four intent datasets: banking77, massive, hwu64, and snips. Runs used small samples (5–10 examples per class), medium samples (81–100), and full datasets, with results averaged over three runs. They measured F1-macro, F1-in-scope, and F1-OOD. Tests ran on a machine with an NVIDIA H100 with 80 GB. The caveats are obvious.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.