Moonshot releases Kimi Vendor Verifier: 6 tests for checking K2.6 providers

Alongside the Kimi K2.6 model, Moonshot AI open-sourced Kimi Vendor Verifier (KVV), a test suite that checks how accurately third-party providers reproduce the model's behavior. A full run takes about 15 hours on two servers with eight NVIDIA H20 GPUs each. Reference results for the official Kimi API are 91.0 on OCRBench, 98.4 on AIME2025 (avg@32) and 78.8 on MMMU Pro Vision. Anyone can now compare their own deployment against these numbers.

Where the problem came from

After the release of K2 Thinking, Moonshot began receiving regular complaints about odd benchmark results. Analysis showed that a significant share of the cases traced back to incorrect decoding parameters. The first line of defense went in at the API level: in Thinking mode, Temperature=1.0 and TopP=0.95 are enforced, and the API also checks that the content of the model's "reasoning" is correctly passed back in the request.

Then subtler anomalies appeared. In one LiveBenchmark test, a third-party API scored noticeably lower than the official one. Checking different infrastructure providers showed that the gap appears everywhere. Moonshot's blog draws this conclusion: the more freely available the weights and the more varied the deployment channels, the harder quality is to control. If users cannot tell a defect in the model itself from an engineering error in its implementation, trust in open models collapses.

Six checks

The suite is built so that each test catches a specific class of failure:

• Pre-Verification checks that the API actually enforces constraints on temperature, top_p and other parameters. Nothing moves forward until this stage passes.
• OCRBench serves as a five-minute smoke test of the multimodal pipeline.
• MMMU Pro checks image preprocessing across a range of visual inputs.
• AIME2025 produces long generations. These expose KV-cache bugs and quantization-related degradation that short tests miss.
• K2VV ToolCall measures the consistency of tool-call triggering (F1) and JSON Schema accuracy. In agents, tool errors compound, so it pays to catch them early.
• SWE-Bench tests full agentic coding. It is not part of the open-source release because it depends on a sandbox.

Separately, Moonshot is working with the vLLM, SGLang and KTransformers communities to fix root causes in the inference engines rather than just record symptoms. The company also gives providers early access to models before release so they can validate their stacks. A public vendor leaderboard is promised as well. The idea is that transparency will push providers to keep a close eye on accuracy.

What verification costs and what to do in practice

Moonshot ran the entire process on two servers with 8 NVIDIA H20 GPUs each; sequential execution took about 15 hours. The scripts are optimized for long-running inference: they offer streaming output, automatic retries and resumption from a checkpoint. That matters because an AIME2025 run with a limit of about 98 thousand tokens can easily be interrupted by a network failure.

The reference settings are listed in the table of official results: Temperature 1.0, TopP 0.95, and token limits of 16 384 for OCRBench, 98 304 for AIME2025 and 65 536 for MMMU Pro Vision.

For teams that get K2.6 from a third-party provider, the procedure is simple. Run Pre-Verification first, then the OCRBench smoke test, and only after that start the expensive long tests. If results diverge from the reference 91.0, 98.4 and 78.8, look first at decoding parameters, quantization and long-context handling. Conclusions that the model itself is weak should come last.

For providers, KVV offers a ready-made pre-launch checklist. Moonshot acknowledges, however, that vendor coverage is still incomplete, and it is looking for lighter-weight agentic tests. Without them, SWE-Bench remains the strictest check, but one that cannot be reproduced independently.