NVIDIA Releases SkillEvaluator: 500+ Rules Catch Errors in AI Agent Skills
NVIDIA has released SkillEvaluator, an open-source tool for checking AI agent "skills": the folders of instructions used with Claude Code or Codex. More than 500 rules look for formatting errors, leaked secrets, and dangerous scripts. A quick check that needs no API key returns a score out of 100, with 70 as the passing threshold. Some failures are worse: a description of 1,028 characters won't load in Copilot CLI, where the limit is 1,024. For that reason, teams should run skills through CI the same way they run code through a linter.
AI-processed from GitHub Trending AI; edited by Hamidun NewsA "skill" for an AI agent is an ordinary folder of instructions and supporting files that gets attached to Claude Code, Codex, or Copilot CLI. For example, a skill might say "how to format reports using our template." These folders are usually checked by eye, and errors only surface when the agent starts "behaving strangely." NVIDIA has published SkillEvaluator, an open-source tool on GitHub that aims to close that gap. It includes more than 500 static rules: ready-made automated checks for formatting, versioning, licensing, personal data, suspicious Unicode characters, and scripts. It also measures whether a skill actually helps the agent. The project appeared on GitHub Trending and gained 133 stars in a day.
Three Levels of Checking
The tool works like a ladder, and each level can be run on its own.
The first level answers whether a skill is safe and correctly formatted. This is where the 500+ rules and detection patterns apply. External scanners are integrated: Gitleaks (for finding leaked secrets), Bandit (for Python vulnerabilities), and optional Semgrep. For the first level there is a mode that does not use a language model: skillevaluator tier1 ./my-skill --no-llm.
The second level looks for duplication and overlap. If a directory contains several skills, the tool compares them and shows where instructions duplicate each other. This requires a catalog file. If the comparison cannot be completed, the report says so plainly.
The third level is the most demanding: live agent evaluation. The tool runs the agent in a sandbox (Docker by default) on tasks with and without the skill, then compares the results. This shows whether a skill delivers a real improvement rather than just looking tidy. It requires access to a model, a supported agent, and a set of tasks. The task set can be generated automatically or prepared by hand, and then reviewed manually.
Checking Whether an Agent Loads the Skill
The most practical feature is called harness-check. A skill can be well written, yet a specific agent may simply never see it. In the documentation's example, the command skillevaluator harness-check ./my-skill produces a report like this. Codex truncates the description at 1,021 characters. Claude Code does not scan the .agents/skills folder. Copilot CLI does not load the skill because the description takes 1,028 UTF-16 code units against a limit of 1,024, which is 4 over the allowed size. As a result, 2 of 6 agent profiles failed to load the skill. This is a typical mistake: the text looks normal, but the agent doesn't know the skill exists.
How to Run It
Installation is simple: pip install skillevaluator or uv tool install, and Python 3.12–3.14 is required. The quick check skillevaluator quality-check ./my-skill needs no key, scanners, GPU, or Docker. It returns a score out of 100 with a breakdown of findings, and the default passing threshold is 70. This covers only static quality: safety and real effect must be verified separately.
For all three levels, NVIDIA offers a free NVIDIA Build key, but only to members of the NVIDIA Developer Program, and with usage limits. The command skillevaluator validate ./my-skill with the nv_build provider runs everything at once. Before that, it's worth running skillevaluator doctor: it checks whether the environment is ready, but it does not guarantee that the model's output will pass.
The tool can be built into a repository. A setup wizard for GitHub and GitLab opens a pull request with a ready-made workflow, and the checks run on changed skills and leave reports and comments in the PR.
Limitations
NVIDIA states plainly that the project is community-supported, with no SLA and no enterprise support. Scores may differ between releases, and installation pulls in third-party open-source components whose licenses should be reviewed in advance. The third level depends on the quality of the test tasks: an automatically generated set covers little.
The practical takeaway for teams that maintain internal libraries of agent skills: start with the free quality-check and harness-check in CI. It is cheap and catches the most frustrating errors. Heavy evaluation with agent runs is worth enabling only for skills that a significant part of the work depends on.
- Nov 21, 2026Novo Nordisk Taught AI Agents to Answer from Manufacturing Procedures: Minutes Instead of Hours and Days
- Nov 21, 2026Alibaba Open-Sources AI Code Reviewer That Tens of Thousands of Developers Used for Two Years
- Nov 21, 2026Anthropic Paid $3.4 Billion for 1.7 GW of Gas Generators
- Nov 21, 2026OpenAI Names 4 Reasons GPT-6 Astra Hides Its Reasoning from Monitors
Key Questions Answered
What are the key details on Three Levels of Checking?
The tool works like a ladder, and each level can be run on its own. The first level answers whether a skill is safe and correctly formatted. This is where the 500+ rules and detection patterns apply. External scanners are integrated: Gitleaks (for finding leaked secrets), Bandit (for Python vulnerabilities), and optional Semgrep. For the first level there is a mode that does not use a language model: skillevaluator tier1 ./my-skill --no-llm. The second level looks for duplication and overlap.
What are the key details on Checking Whether an Agent Loads the Skill?
The most practical feature is called harness-check. A skill can be well written, yet a specific agent may simply never see it. In the documentation's example, the command skillevaluator harness-check ./my-skill produces a report like this. Codex truncates the description at 1,021 characters. Claude Code does not scan the .agents/skills folder. Copilot CLI does not load the skill because the description takes 1,028 UTF-16 code units against a limit of 1,024, which is 4 over the allowed size.
How to Run It?
Installation is simple: pip install skillevaluator or uv tool install, and Python 3.12–3.14 is required. The quick check skillevaluator quality-check ./my-skill needs no key, scanners, GPU, or Docker. It returns a score out of 100 with a breakdown of findings, and the default passing threshold is 70. This covers only static quality: safety and real effect must be verified separately.
Build & Ship Production AI Agents Without Boilerplate Code
Start your free 72-hour trial at Hamidun Academy (academy.hamidun.com) — interactive lessons on Claude Code & Cursor, live Tech Radar, and production agent templates.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.