Independent AI intelligence Two editions daily · ET
Fervor AI

Analysis · November 16, 2026 · repo

NVIDIA SkillEvaluatorai-skillsclaude-codecodex

NVIDIA SkillEvaluator Tests Whether Your Agent Skills Actually Do Anything

Most agent skills ship on vibes. This free tool runs your agent with and without one and shows you the difference.

Agent skills are everywhere now, and almost none of them have been tested. A skill is a folder of instructions and helper files that teaches a coding agent how to do one job: run your deploy, write in your house style, call an internal API. You drop it in, the agent loads it, and you assume it helps.

That assumption is rarely checked. A skill can be ignored. It can load when it should not and crowd out the context your agent needed. It can contain a script that does something you did not read. The fact that it installed without an error tells you nothing about any of that.

NVIDIA's SkillEvaluator, which sat at #7 on Trendshift's live board at about 7:20 AM ET on Monday, November 16, attacks the gap in the most direct way possible. It runs your agent on the same tasks twice, once with the skill and once without, and reports the difference. That one idea, a controlled comparison instead of a gut feeling, is the reason to care about this repo.

Why skills need testing at all

Skills went from a niche format to a standard in about a year. Anthropic published the Agent Skills format in December 2025 and in October donated it to the Agentic AI Foundation, saying more than 70 agent platforms had integrated it, including ChatGPT, Cursor, GitHub Copilot and VS Code. That is a lot of agents loading a lot of folders written by a lot of people.

Writing a skill is easy. Knowing whether it works is not. Agents are noisy, so one good run proves little. And a skill can hurt you in a way that looks like nothing at all: NVIDIA's own write-up notes that every skill competes for the agent's attention, and "a skill that loads when it is not relevant can reduce agent performance."

So there are really three questions, and SkillEvaluator splits them into three tiers.

The three tiers, in plain terms

Think of it like checking a new hire. First, are their papers in order? Second, are they doing a job someone else already does? Third, does the work actually get better when they show up?

Tier 1, validation. Is the skill well-formed, safe, and readable by your agent? This tier runs static checks (it reads the files, it does not run an agent): the skill's metadata format, a quality score, secret detection, license checks, script linting and a security scan for things like prompt injection. It fetches two scanners on its own if they are missing, NVIDIA's SkillSpector and Gitleaks. Per the README, the core checks need no API key.

Tier 2, deduplication. Does the guidance repeat itself, or overlap with other skills you already have? It uses embeddings (a way of turning text into numbers so similar passages land near each other) to spot duplication inside one skill and, if you give it a catalog, across skills.

Tier 3, live evaluation. Does the skill help? This is the interesting part. SkillEvaluator turns test cases into tasks and runs a real agent on each one twice in an isolated sandbox, with the same prompt, model and grading. The only thing that changes is whether the skill is installed. It uses Harbor, an open-source agent evaluation framework, for the sandboxed runs, and Docker by default. NVIDIA calls the score difference Skill Lift.

You can run any tier on its own. That matters for cost, because Tier 1 is free and Tier 3 is not.

What NVIDIA found when it pointed this at its own skills

NVIDIA ran the full pipeline on more than 300 of its own verified skills across more than 30 products, using two agents, Codex and Claude Code. The figures come from an August 12, 2026 snapshot.

Without the skill installed, the agents scored 46 out of 100 on correctness. With it, 87. Effectiveness, meaning whether the agent reached the goal the expected way, went from 39 to 78. Security barely moved, 97 to 98, which is what you want: the skill did not make runs less safe.

Read those numbers carefully, though. These are NVIDIA's skills, graded on NVIDIA's tasks, mostly about NVIDIA products an agent would otherwise know little about. Of course a guide to an obscure library helps an agent that has never seen it. Your skill that tells Claude Code to "write clean code" is a very different animal, and the honest expectation is a much smaller lift, maybe none.

NVIDIA also says most skills ran only one attempt per task (85% of them), and the post reports no confidence intervals (the error bars that tell you how much a score could move by chance). Treat a single skill's score as a hint, not a verdict.

Put this into practice

Start free, and only pay for live runs on the skills you care about.

Step 1: the keyless check. Install it and point it at a skill folder:

pip install skillevaluator
skillevaluator validate ./my-skill --tiers 1

That runs every Tier 1 check, with no API key. The README lists Python 3.12 through 3.14. What to expect: a pass or fail, findings with file and line, and a report. If a scanner cannot run, the result says INCOMPLETE and the command exits with an error, so a missing scanner never counts as a pass by accident.

To see what a good result looks like first, the README's quickstart runs a bundled example skill, text-analyzer, which scores in the high 80s out of 100 (a B) against a default passing threshold of 70. The exact figure has shifted between README edits, so don't treat it as a target.

Step 2: make it a habit. If your skills live in a GitHub repo, run:

skillevaluator ci init

It opens a pull request that wires the checks into CI, so every changed skill gets validated before it merges. There is a GitLab template too.

Step 3: let your agent run it. You can install SkillEvaluator as a skill itself:

npx skills add NVIDIA/SkillEvaluator

Then ask your agent, in plain words, "Is my skill any good?" It works with Claude Code, Codex and other skill-aware agents.

Step 4: the live comparison, for the skills that matter. This is where you find out whether a skill earns its place. The base pip install covers the static checks; the live tier needs the tier3 extra (the README runs it as uv run --extra tier3 ...), a model provider and Docker. Write a short eval-guidance.md describing the tasks the skill is for and what a good outcome looks like, then generate test cases, read them, and run the with-and-without comparison:

skillevaluator tier3 create-eval-dataset ./my-skill --full --prompt eval-guidance.md
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker

The first command writes cases to evals/evals.json, each with a prompt and an expected outcome. With a model provider configured, --full also generates negative cases, tasks where the skill should not load, which is how you catch a skill that barges in where it is not wanted (in keyless template mode you have to write the negative prompt yourself). The second runs both versions in Docker and reports Skill Lift. The generation command follows the README; NVIDIA's blog walkthrough shows a shorter form and the same evaluate command.

Before you trust a Tier 3 result, read the generated cases. The README is blunt that the starter templates "omit unauthored negatives," so the tool cannot invent the edge cases only you know about. Five cases you wrote beat fifty it guessed.

Step 5: decide what to do with the result. A live comparison gives you one of three answers, and each has a clear next move:

  • Clear lift on the tasks you care about. Keep it. Save the eval cases next to the skill, so the next edit gets tested against the same bar.
  • Lift near zero. The agent already knew this, or it is not loading the skill. Check the negative cases and the discoverability score (whether the agent finds and loads the skill when it should) before you rewrite anything. Sometimes the fix is a better description so the agent picks it up, not more instructions.
  • Negative lift. Delete it, or cut it down hard. A skill that makes runs worse is costing you context on every turn for nothing, and it is the case NVIDIA's own write-up warns about.

A practical rule for which skills deserve Step 4: the ones your agent loads on most tasks, and the ones that run scripts. A skill that fires everywhere costs context on every turn. A skill that runs code is the one you most need to see in a sandbox.

Honest limitations

Live runs cost money and setup. Tier 3 needs a model provider, a supported agent and a running sandbox. The README warns that model and managed-sandbox usage can incur charges. NVIDIA Developer Program members can use NVIDIA-hosted endpoints for free prototyping, within rate limits.

It is experimental. The README lists the support level as experimental: community-supported, best effort, no SLA (no service guarantee from NVIDIA).

One run is noise. Agents vary between runs. If you run each case once, a lift of a few points could be luck. Run the cases you care about more than once before you delete a skill or celebrate one.

The headline numbers are NVIDIA's. The 46-to-87 correctness jump comes from NVIDIA's own skills on NVIDIA's own tasks. It shows the method works. It does not predict what your skills will score.

Gates are softer than they look. In the full validate pipeline, Tier 1 blocks on failure, Tier 2 blocks unless you pass --no-block-on-dedup, but Tier 3 only blocks if you add --block-on-agent-eval (a flag that makes a bad live result fail the run). By default, a skill that makes your agent worse can still pass. Add the flag in CI if you mean it.

The skill that stays is the one you measured

The skills ecosystem grew up on trust. Someone wrote a folder, it looked sensible, you installed it. That worked when you had three skills. It stops working at thirty, when nobody remembers which ones fire on which tasks or what they cost in context.

SkillEvaluator does not tell you what to write. It tells you whether what you wrote changed anything. Run the free tier on every skill you have today. Pick the two your agent leans on most and give them a live comparison this week. Some of them will earn their place. The ones that don't are just extra reading for your agent on every single turn.

Sources: NVIDIA/SkillEvaluator README (main branch, version 0.9.0 per pyproject.toml, read November 16, 2026; Apache-2.0, about 2.2k stars), NVIDIA Technical Blog: Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator, Anthropic: Donating Agent Skills to the Agentic AI Foundation, Trendshift daily board, Harbor, SkillSpector. The release feed could not be read this run, so no release tag is pinned.