Where an NLI Model Helps in a Meeting Assistant: Three Fittings, Two Rejections

Charoiteai, developer of the local meeting assistant Charoite, shared where an NLI model works and where it gets in the way. He tried it in five places — two survived.

In July he ran the same meeting transcript through the auto-summarization model four times and got 39 different statements, 32 of which appeared in only one run. He wanted a tool that would flag where the runs contradict each other. He picked mDeBERTa-v3-base-xnli-multilingual-nli-2mil7: multilingual, trained on 2.7 million pairs, 1.1 GB in ONNX format, loads in a second and a half.

Spot One: Repeats in the Live Feed — Works Daily

In a long meeting, the summarization model only sees 800 characters of fresh text, so it repeats what's already been said in different words. NLI catches duplicates through bidirectional logic: it's a duplicate when A follows from B and B follows from A at the same time. A clarification (one-directional only — "budget approved" follows from "budget approved through year-end," but not vice versa) stays in the feed. A contradiction doesn't pass in either direction.

The scheme is simple: a new statement is compared against all 16 preceding ones using difflib with a threshold of 0.3 (deliberately low so nothing gets filtered too early). Pairs that pass go into the model in both directions. If both probabilities are ≥ 0.8, the statement is dropped as a duplicate; otherwise it stays. The threshold is 0.8 rather than the default 0.75, because a repeat is annoying, but a lost new fact is worse. Each call takes tens of milliseconds, and most pairs never reach the model thanks to the prefilter.

This is the only spot where NLI has worked without issues for three months straight. Not perfectly: at a September 29 meeting, the same topic came back five times in 8 minutes without adding anything new. The mechanism is already noted — it's the next task.

Spot Two: Contradictions in Minutes — Rejected

He tested it on the archive: 14 meetings, 51 minute items, 1,192 pairs for analysis. Of the 76 pairs that passed the prefilter, the model flagged 30 as contradictions. Almost all were false. The reason is length: the average minute item is 138 characters, 69% longer than 120. A phrase like "It is necessary to initiate direct contact with the responsible person to clarify the general principles of connection, bypassing intermediate approval stages" is actually three ideas in one paragraph. Any two such paragraphs will have a common word and something that looks like conflict.

The solution is two orders of magnitude cheaper: short statements (under 60 characters) with a named object, along with numbers and dates, are checked directly against the transcript via search. In the same run, this kind of check caught one fabrication out of ten statements: a release scheduled for "08.17," when August 17 was never mentioned, neither as a number nor in words.

Spot Three: Nightly Core Revision — Disabled

Cores are topics that live longer than one meeting. Each core has an essence, a chronicle, people, and decisions. The problem: the same topic gets called different things. An example from the architecture document in the repository: configuring API access and obtaining a token are one piece of work, but two cores with two incomplete chronicles.

Revision worked in two steps: bge-m3 gathered candidates (cosine similarity from 0.55), then NLI judged their essence in both directions with four outcomes: duplicate (both directions above 0.8 — merge candidate); possible duplicate (0.72–0.8 — flagged); nesting (one direction above 0.85, the other below 0.5 — a narrow topic within a broad one); boundary (both above 0.45 — adjacent topics). A core with three nestings was considered a hub and left untouched.

In August it looked good: 284 live cores, 40,186 pairs, 1,806 passed the prefilter, revision completed in 6 minutes. By October, there were over 1,000 cores. The September 27 revision hit the nightly ceiling after 3.5 hours and left 824 cores for subsequent nights. Dossiers (topic summaries) failed to compile at all on eight of twelve nights.

He looked at ten random high-confidence pairs the revision was about to merge. Zero were real duplicates. One was debatable. The confidence of the false pairs: 0.93–1.00. The explanation: not every core has its own essence. If it doesn't, the revision falls back on a boilerplate status like "Discussion of requirements and features." Two different work topics with the same boilerplate — and NLI says: these phrases follow from each other. Yes, they do, because it's the exact same phrase. The model performed correctly; this is a data error, not a model error. Possible-duplicate flags piled up to 239 in the graph, zero reviewed: nobody opened the service note with the flags.

On October 5, he disabled it with a single line in the config. Auto-merging was never turned on, but it's now clear it would have glued together different work topics — a prompt during a meeting would have shown the status of someone else's task. The merge backup lives for a week and nobody looks at it — in practice, it's irreversible.

Spot Four: Gate Before the Lightning Bolt — Didn't Help

The ⚡ button (quick answer) triggers on a question just asked during the call. A structural filter finds a candidate in the feed, and a local model writes an answer in seconds. But sometimes the candidate isn't actually a question: someone asked again, thought out loud, or cut off mid-sentence. The model answers "I don't see a question," and time is wasted.

He wanted a cheaper judge ahead of the model, without fine-tuning: two affirmative hypotheses (negated hypotheses work worse for the model): "The speaker is asking a question and expects an answer" and "The speaker is asking again, thinking aloud, or cutting off a phrase." A cascade: confident — decide on its own; unsure — hand off to the model.

In shadow mode: the judge ran alongside and decided nothing, only logging. Four meetings, 117 utterances. The model answered 115, declined 2 — that's 1.7%, the ceiling of what the gate could save even if it were perfect.

But NLI was far from perfect. Accuracy against the benchmark: 55.6%, almost a coin flip. Poor calibration — confident noticeably more often than correct. Median call time: 99 ms. He checked thresholds:

| Threshold | Filtered out | Empty ones removed | Questions lost | |-----------|---------------|---------------------|-----------------| | 0.60 | 31.6% | 1 of 2 | 31.3% | | 0.70 | 17.9% | 1 of 2 | 17.4% | | 0.80 | 9.4% | 0 | 9.6% | | 0.90 | 0 | 0 | 0 |

At any threshold that filters out anything at all, it loses dozens of times more real questions than empty ones it removes. He had written the closure rule a week before the measurement: if empty ones are under 15%, close the topic. It came out to 1.7%. On October 6, he turned off the shadow test.

What Broke

The most unpleasant failure was a silent one. First time: the models sit in the data root, but the code looked for the NLI model next to itself, in the code root. In the developer's own repository, both roots happen to coincide, so the defect was invisible. On an installation where data lives separately, the model wasn't at that path. The rule in place was: no model on disk — deduplication off. No error, no warning, the feed just started repeating again. It was noticed not from the feed, but during a review. It happened twice. Now the path is computed at call time.

The second time was in core revision, also twice in a row. The judge would load but couldn't judge, and the revision counted the run as completed: it found no pairs, moved the flag forward, and fresh cores dropped out of review entirely. Zero for the revision meant "similar" instead of "nothing to judge." Now a judge failure is an exception, not a zero.