The Agent Cache That Hit 5 Percent, and the Release Note That Said So
A one-maintainer agent toolkit shipped a release called "measured, and worse than claimed." The measurement method is worth more than the package.
Most release notes are a list of things that got better. On September 20 at 16:00 UTC, a repository called hermes-jev-skills shipped v0.18.0 under a heading almost nobody writes voluntarily: measured, and worse than claimed. Underneath it are three features the maintainer had already shipped, tested properly for the first time, and found wanting. One of them had been running in production at a 5% hit rate.
The package itself is niche. It routes an agent's small structured decisions, model routing, memory filtering, compaction, skill selection, mailbox triage, to a fast decision model instead of the main LLM. It needs a third-party paid API key to do anything. The interesting artifact is not the package. It is the method that produced the 5%, because that method would find the same class of bug in software you have shipped.
The bug is a mismatch between your key space and your input distribution
Version 0.14.0 added a plan cache. The one expensive call in this toolkit whose answer is a pure function of its input is plan generation: roughly one second and $0.0006 per call, repeated every time someone issues the same command. Caching it is obviously correct. The cache keyed on the command text, byte for byte.
Here is the thing the key assumed and the system does not provide: these commands arrive by dictation.
A dictation engine does not emit the same string twice. It capitalises the first word sometimes. It capitalises an app name sometimes. It adds a full stop at the end. It occasionally leaves a doubled space. Six identical spoken intentions produce six different byte sequences, and an exact-match key treats all six as unrelated.
The maintainer drove the planner with an injected transport over repeats of six spoken commands, re-transcribed the ways a real dictation engine varies them. From the project's own write-up:
It hit once. 5%.
And then, in the same paragraph, the detail that makes the number worse rather than better: the one hit was leading and trailing whitespace, which strip() had already handled since the first version. The cache, as shipped, contributed nothing at all. Every other repeat of something the person had already said was a second model call for a plan sitting on disk.
What replaced it, and what deliberately still misses
The fix is not "normalise everything." That would be a different bug.
There are now two keys in one file. Both carry the front application, the running apps the command names, the model, the endpoint hostname, and a hash of the prompt, the step schema, the vocabulary, the step limit and the never-send rule version. They differ only in how they carry the command text:
- the exact key holds it byte for byte, spacing included
- the loose key holds one utterance rather than one transcription of it: case-folded, inner whitespace collapsed, trailing engine punctuation removed
The constraint doing the real work is the write rule. A plan is written to the loose key only when no field of it copies the command's exact characters. Exactly two fields can: the text of a type_text step, because that gets typed character for character, and the path component of an address, because /Docs and /docs are different pages. Everything else a step holds, an app name, an on-screen target, a menu path, a key name, a count, is matched case-insensitively by whatever receives it anyway.
So a plan that dictates a note stays on the exact key, where "write: Buy milk" and "write: buy milk" remain the two different commands they are. A read from the loose key re-checks the same condition before serving, on the stated grounds that the file is not the program's to trust.
Two near-misses are left in on purpose, and the reasoning is the most useful paragraph in the document. An inserted comma (Open Safari, and go to example.com) and a politeness wrapper (Please open Safari) both still miss, because stripping punctuation anywhere but sentence-end would merge ex.ample.com with example.com, and dropping words is a rewrite whose failure mode is the safety filter losing the word the person actually said. Refusing a normalisation that would improve your hit rate, because the failure it enables is worse than the cache miss it prevents, is a judgment call most teams never write down.
The second bug is better than the first
Inside the same rewrite is a smaller finding I have thought about more than the headline.
The key includes "the apps the command names," determined by looking each running app's name up in the command string. That lookup did not originally collapse whitespace. So Switch to Visual Studio Code, dictated with a doubled space inside the name, does not contain "visual studio code". The app went unnamed. An app that goes unnamed is an app whose running-or-not state has silently dropped out of the cache key.
Measured consequence: that command produced the same key whether the app was running or closed, and the plan stored for the machine where it was open, which said click it in the dock, was served to a machine where it was not running at all.
The project's own framing is the right one. Word-level rewrites it does not attempt are one thing. Losing a correctness field to the exact input variance the loose key was built to absorb is another. A normalisation gap in one part of a key can delete a different part of the key entirely, and the symptom is a wrong answer rather than a miss.
The repository disagrees with itself about its own numbers
I want to be straight about something I found while reading, because it cuts against the piece's own hero.
The v0.18.0 changelog says the measurement ran over 22 repeats and that the normalised key reaches 64%. The file docs/response-caches.md says 19 repeats and, for what it explicitly calls "the same 19 repeats," 68%. Both round the original hit rate to 5%, and 1/19 and 1/22 both do. The docs page states that both figures are asserted by tests sitting next to the corpus in tests/test_plan.py, so you can re-measure rather than take anyone's word for it.
A project whose entire release thesis is we measured and published what we found states that measurement twice, inconsistently, in the same repository on the same day. That is not a scandal. It is the ordinary thing that happens when a number gets written in two places and one of them is updated. It is also a reminder that "asserted by tests" tells you a test passed, not that two prose descriptions of that test agree.
Take the 5%. Treat the improvement figure as "into the sixties" until the repository picks one.
Put this into practice
The transferable thing here is a test-design habit, not a package. Four steps, none of which need this repo.
1. Write down what your cache key assumes, in one sentence. For an exact-match key the sentence is "identical intent produces identical bytes." Say it out loud about your actual input source. If that source is speech, OCR, a rich-text editor, a mobile keyboard with autocorrect, an LLM, or a human retyping from memory, the sentence is false.
2. Build a variance corpus, not a test case. The move that produced the 5% was re-transcribing the same six intentions the way the real engine varies them, then committing that corpus as a named fixture. Do the same with whatever your input actually is: capture fifty real inputs, group them by intent, and keep the variants. A fixture you can re-measure is worth more than a number in a changelog, which is exactly what this project found out the hard way.
3. Measure the shipped feature in shadow before you trust it. This toolkit defaults its cache to shadow, where the model is always called and nothing is reused, and every result records miss, shadow_agree (the stored plan equalled the fresh one) or shadow_differ (it did not). Read a batch of those before turning anything on. Mostly miss means your inputs do not repeat and the cache buys nothing. A shadow_differ means the model gave two different answers for the same input in the same context, and with the cache on you would have received the older one. Their line is the one to keep: a cache can only ever be as right as the first answer it kept.
4. Check that expiry removes. The seven-day TTL in this project refused to serve stale entries and left them on disk. An entry only left the file as a side effect of storing a new one in the same run, so five of six run types, including a cache hit, an outage, and a run whose plan looked sensitive, left a ten-day-old dictated note sitting there. If your TTL is implemented as a read-time predicate, you have a retention policy that does not retain what it says and does not delete what it says either. Go read yours.
Honest limitations
This is one maintainer's package and I would not put it on your critical path today.
It depends entirely on a third-party paid decision-model API, so there is no self-hosted story. Two verification passes I ran this afternoon returned contradictory star counts for the repository, so I am not quoting one. The 0.13.0 changelog entry discloses that a real person's name and real customer correspondence were in this public repository, pulled from a live system during debugging, removed from the tree but present in git history prior to commit 8ac177c. That is exactly the kind of disclosure I want maintainers to make, and it is also a thing that happened.
On the cache itself, two limits are stated by the project and worth repeating. Plans are stored with the dictated text of type_text steps in plain text, at mode 0600, for seven days, and "looks sensitive" is a check for things that look like secrets, not things that are private. And concurrent writers lose entries: four processes writing 120 plans left 33 to 35, because the file swap is atomic rather than merged. The project argues that costs exactly one model call per lost entry and that a lock would cost every caller a way to hang, which I think is the right call for a single-user desktop tool and the wrong call for a shared service.
Finally, everything above about hit rates is measured on one person's six commands with one dictation engine. The project says so itself: if your commands mostly carry dictated text, expect something much nearer the 5%.
What to do with this
You have a cache somewhere that was never measured after it shipped. Most of us do. Go find out what it hits against the input your users actually send rather than the input your tests send, and write the number down where someone else can argue with it.
The release title is the part worth stealing.
Sources: kerpopule/hermes-jev-skills · docs/response-caches.md · CHANGELOG.md, v0.18.0. Licence verified as MIT, "Copyright (c) 2026 Steve Darlow", from the LICENSE file; release tag and date from the repository's releases feed.
Medium metadata
Title: The Agent Cache That Hit 5 Percent, and the Release Note That Said So Subtitle: A one-maintainer agent toolkit shipped a release called "measured, and worse than claimed." The measurement method is worth more than the package. Tags: Software Engineering, AI Agents, Caching, Testing, Open Source Canonical: publish on fervorai.dev first, then import to Medium from that URL.