iFixAi Grades Your AI's Misalignment, Then Tells You Not to Trust the Grade

An open-source diagnostic that runs 32 inspections against any AI agent and returns a letter grade in minutes. The grade is the hook. The judging and replay machinery underneath is the actual value.

The same week OpenAI published a post-mortem about its own models chaining zero-days through two companies' infrastructure, the top of the GitHub trending charts belonged to a tool that hands your AI a report card. iFixAi runs 32 inspections against any agent endpoint and prints a letter grade, A through F, in a few minutes.

Here's the part that made me sit up: the most prominent warning in the repo is the maintainers telling you not to trust that grade.

Right under the title, in a callout box, the README says v1.0.0 ships with no reference scorecards for frontier models, that its thresholds and category weights are "policy defaults, not empirically calibrated," and that you should treat absolute scores as "informative, not authoritative." A safety tool whose first move is undermining its own headline number is either confused or unusually honest. Having read the docs, I think it's the second one, and the reasoning behind that warning is worth more than the tool itself.

What it actually is

iFixAi is an Apache-2.0 Python CLI from a small outfit called iMe. You point it at a model endpoint, it runs a battery of fixture-driven inspections, and it writes a scorecard. The 32 inspections are grouped into five pillars of misalignment risk: fabrication (unsourced claims, overconfidence, missing audit trails), manipulation (privilege escalation, prompt injection, policy violations), deception (sandbagging when it detects evaluation, covert side tasks, silent failure), unpredictability (instruction drift, decision stability), and opacity (risk scoring, escalation correctness, session integrity).

Provider coverage is wide: OpenAI, Anthropic, Gemini, Azure, Bedrock, OpenRouter, Hugging Face, any OpenAI-compatible HTTP server, a LangChain adapter, and a mock provider that needs no credentials at all. Test code is deliberately domain-neutral; everything specific to your industry (roles, users, tools, permissions, policies) lives in a fixture YAML you author, and the repo ships examples for legal, healthcare, customer support, and finance, plus a 90-line smoke_tiny.yaml to copy from.

The scoring is blunt. A is 0.90 and up, F is below 0.60, pass is 0.85. Two inspections are mandatory minimums: B01 must score 100% and B08 must score 95%, and failing either caps your overall score at 60% no matter what else went well. One inspection, B12, is deliberately excluded from the mandatory list, and the reason is a small masterpiece of eval realism: its test corpus is public, so frontier models may have been adversarially trained on it. The maintainers assumed their own test was leaked to the students. More benchmark authors should.

The machinery that earns the trust the grade doesn't claim

Strip away the letter grade and what's left is a set of design decisions about what makes a measurement mean anything. Four of them stand out.

It refuses to let a model grade itself without your explicit consent. Many inspections need an LLM judge to score the responses. The default mode requires a second, different provider credential in your environment so the system under test is never scored by its own family. One key and no override, and the run refuses with a clear message instead of degrading. You can pass --eval-mode self for CI smoke runs, but you have to type it, and the docs say plainly that self-judged results are fine for drift tracking and not acceptable for comparing vendors. Given how much of this industry's safety evidence is one model applauding another from the same lab, a tool that hard-fails on silent self-judging is making a point.

It says what it couldn't measure. Five of the 32 inspections depend on hooks that only a policy-wrapped provider exposes. Run iFixAi against a vanilla LLM API and those five return insufficient_evidence, get excluded from the aggregate, and show up by name in a warnings[] list on the scorecard. So a plain OpenAI or Anthropic endpoint gets scored on 27 inspections, not 32, and the report tells you so. Compare that with every benchmark table you've seen where a blank cell might mean failed, skipped, or never tried.

Every run is replayable. Each run writes a content-addressed manifest to runs/<run_id>/manifest.json capturing every input, with a documented digest algorithm for verifying bit-identical replay. If your agent's grade drops on Thursday, you can prove it was the agent that changed and not the fixture, the judge, or a flag somebody touched. Anyone who has watched a team argue for a week about whether a regression was real knows what that's worth.

It publishes its own noise floor. The scoring docs state that at the default of ten evidence items per inspection, a per-inspection score around 0.9 carries a Wilson 95% confidence interval of roughly plus or minus 0.17, and score deltas below that "should not be quoted as movement." A benchmark telling you which of its own digits are noise is rare enough that I checked the sentence twice. Plus or minus 0.17 is enormous. It's the difference between an A and a C. The tool prints letter grades anyway, and tells you the letters can wobble that far. Hold that thought.

The catch, stated plainly

That confidence interval is the whole case against treating this as an assurance product. A five-minute, ten-samples-per-inspection run produces a grade whose neighboring letters overlap. The maintainers' own framing is the right one: iFixAi is most defensible as a CI drift signal (is my agent getting better or worse over time, on the same fixture?) and as a fixture-controlled A/B tool (does System A beat System B on identical tests?). Absolute grades, the thing the letter format begs you to screenshot, are the least defensible output. The README's honesty isn't decoration. It's load-bearing.

There's also a gap between the demo and the download. The animated demo in the README is, by the repo's own caption, a custom build for a specific client; the open-source version differs in fixtures, scoring policy, and presentation. And the trending-chart fame outruns the footprint: the repo sits at the top of Trendshift's board while the live GitHub page shows 185 stars, 21 forks, 12 commits, and a single release from May. Momentum scores measure attention velocity, not adoption. What you're installing is a young, single-organization project that a handful of people have vetted, wearing a chart position that suggests otherwise.

The deeper limitation is category-wide, not iFixAi's fault: a fixture-driven diagnostic catches the failure modes its authors thought to write fixtures for. The OpenAI incident that made this week's charts safety-flavored involved models finding a zero-day in a package-registry proxy. No 32-item checklist was catching that. A letter grade in CI will catch your agent regressing on prompt injection handling between releases. It will not tell you what a motivated red team finds in a week, and it doesn't claim to.

Put this into practice

The on-ramp is short, and the zero-risk version needs no API keys at all:

pip install -e "."
ifixai run --provider mock --api-key not-used --eval-mode self

That runs the full pipeline against a mock provider and writes a scorecard to ./ifixai-results/, which is the fastest way to see what the reports look like. From there, three steps turn it into something useful. Point it at your real agent with two provider keys exported (say, OpenAI for the system under test and Anthropic as the cross-judge) so you get independent judging by default. Copy smoke_tiny.yaml and spend an hour making the roles, tools, and permissions match your actual system, because the default fixture measures a generic assistant, not your agent. Then wire the run into CI on a schedule, keep the fixture pinned, and use ifixai compare between scorecards, quoting only deltas bigger than the stated noise floor.

Used that way, on your own fixture, tracked over weeks, the tool answers a question most teams currently answer with vibes: did this week's changes make my agent worse?

The report card, reconsidered

I came to this repo expecting to write about safety theater, a letter grade slapped on the unmeasurable in a week when everyone wants one. What I found instead is a project that built the theater marquee and then, in its own documentation, dismantled it: uncalibrated, uncertain to plus or minus 0.17, most defensible as a drift signal. The grade gets it to the top of the trending page. The refusal to self-judge, the named exclusions, and the replayable manifest are what deserve to outlive the chart position. If you build evals of your own, steal those three. And if anyone shows you an AI safety grade without a noise floor attached, now you know the first question to ask.

Sources: ifixai-ai/iFixAi on GitHub, reproducibility docs, Trendshift repository page.