CLI-Anything Gives Agents Real Software, and Hands You a Generated Harness to Maintain

A Hong Kong lab's answer to agents fumbling around in Blender and LibreOffice is not better vision. It is a generated command-line interface that mirrors the app's own data model, and the tradeoff sits in who maintains it.

Most attempts to get an AI agent to operate professional software start from the assumption that the agent needs to see better. Screenshots, accessibility trees, pixel coordinates, a model that can find the layers panel. CLI-Anything starts from the opposite assumption: the agent is fine, the software just never shipped a text interface, so build it one.

The repo's own summary of the problem is blunt. AI agents are good at reasoning and bad at using real professional software, and the existing options are fragile UI automation, thin APIs, or reimplementations that miss most of the functionality. Its answer is a generator. You point a plugin at a codebase and it produces a working command-line harness for that application, with structured output and session state, then publishes it so agents can install it like any other package.

At scan on August 16, the GitHub page served 43.5k stars and 4.1k forks against 844 commits, under Apache-2.0, from HKUDS, the Data Intelligence Lab at the University of Hong Kong.

What the generator actually does

The pipeline runs in seven phases, and the interesting work happens in the first two.

Phase one analyzes the source and maps GUI actions onto the application's real APIs. Phase two designs command groups, a state model, and output formats from that map. Only then does it implement, using Click, with a REPL mode, JSON output, and undo and redo. Phases four through six plan tests, write them, and record results in a TEST.md. Phase seven writes setup.py and installs the command onto PATH.

The design decision worth stealing is in that ordering. The harness is derived from the application's internal model rather than from its menus. A GUI is a projection of a data model onto a screen, and automating the projection is what makes RPA brittle. Automating the model is what makes a CLI stable across releases. There is also a phase 6.5 that generates a SKILL.md from the CLI's own Click decorators, setup.py, and README, so an agent reads a description of the tool that was extracted from the tool rather than written about it.

Distribution is the other half. pip install cli-anything-hub gets you a package manager, cli-hub install <name> pulls a harness, and there is a meta-skill whose entire job is letting an agent browse the registry and install the CLI it needs without a human picking. The catalog is wide: Blender, GIMP, Inkscape, Krita, Audacity, Kdenlive, Shotcut, OBS Studio, LibreOffice, FreeCAD, QGIS, Obsidian, Zotero, n8n, RenderDoc. The changelog reads like a stream of merged community harnesses, several per week, each arriving with its own test suite.

The README claims 2,280-plus tests with real software validation across 18 major applications. That figure is the maintainers' own, it sits next to a registry that clearly lists more than 18 applications, and other sections of the same README report 2,461 and 2,464 instead. Treat the low number as a floor on the well-tested core rather than a description of the whole catalog.

The position: this is a token argument, not a capability argument

The framing that gets attention is "agents can finally use Blender." The framing that matters to anyone paying an inference bill is different.

A screenshot loop spends tokens on pixels to recover information the application already has in structured form, then spends more tokens recovering from a misread. A command that returns JSON spends tokens on the answer. Same task, different order of magnitude, and the reliability difference compounds over a long session because every screenshot-driven step can fail independently.

The repo makes a second claim in the same direction that I think is underrated: once a GUI application has a CLI, you can synthesize agent tasks, evaluators, and benchmarks entirely in code. Evaluating a GUI agent normally requires a human to look at a screen or a scoring model to judge one. Evaluating a CLI agent requires diff. If you are trying to measure whether your agent got better at video editing this month, that difference is the whole project.

That is the real shift here. Making software agent-native gets pitched as a capability win. It is closer to a measurement win, and measurement is what lets everything else improve.

Putting this into practice

The friendly first step is not generating anything.

Install the hub and try an existing harness. pip install cli-anything-hub, then cli-hub install something you already know well. Pick the application whose output you can judge at a glance, because your first job is calibrating how good the generated commands actually are. Note that harnesses wrapping desktop software need the upstream application installed too, so a Blender harness expects Blender.

Give your agent the meta-skill only after that. Autonomous discovery and install is genuinely useful and it is also an agent installing packages based on a registry lookup. Watch it work by hand a few times before you let it do that unsupervised.

Generate against something small and yours. The plugin installs into Claude Code through /plugin marketplace add HKUDS/CLI-Anything and /plugin install cli-anything, and there is a Pi Coding Agent extension in .pi-extension/. Run the generator on an internal tool or a small open-source app rather than on Blender, because your first generated harness is a code review exercise, and you want a diff you can actually read.

Read the generated tests before you read the generated code. Phase four and five produce the test plan and the suite. If the tests only exercise happy paths on synthetic data, the harness is a prototype no matter how long the command list is.

Use refine incrementally. /cli-anything:refine ./yourapp "batch processing and filters" does gap analysis against the software's full capability surface and adds what is missing. Each run is described as incremental and non-destructive, which makes narrow, repeated passes a better strategy than one giant generation.

Treat the harness as your code from the moment it exists. This is the part people skip. A generated CLI that ships to your team is a dependency your team maintains, and nobody at HKUDS is on call for the one you made on a Tuesday.

Where this breaks

The honest limitations here are specific, and the project's own history supplies most of them.

Generated harnesses have shipped real vulnerabilities. The changelog records a GIMP Script-Fu path injection fix in March and a security hardening pass on the browser CLI adding URL validation and DOM sanitization. That is not a criticism of the maintainers, who found and fixed both. It is the predictable shape of the risk: you are running machine-written code that shells out to real applications with real file paths, and the failure mode is command and path injection. Any harness you generate deserves the same review you would give a contractor's pull request, and probably a sandbox.

Coverage is uneven by construction. A harness is only as good as the analysis phase's read of that specific codebase, so two applications of similar complexity can produce very different quality. The refine command exists because of this, which tells you the first pass is rarely complete.

There is a dependency you might not want. The generator needs the source, and the pitch for closed software is to find an open-source alternative and generate against that instead. That is a fine answer for a hobby workflow and a bad one if your team's actual output lives in a proprietary format.

The registry is a supply chain. Community harnesses land fast, an agent can install them autonomously through the meta-skill, and each one is code that runs local applications on your machine. The speed that makes the catalog useful is the same speed that makes it worth pinning versions and reading what you install.

And the numbers move fast enough to make a point. The repo page served 43.5k stars and 4.1k forks on one load during this write-up and 47.5k stars, 4.4k forks, and 873 commits on a check a few hours later. Both are real readings. For a project merging several harnesses a week, any figure in this sentence is already wrong.

What to do with this

The question CLI-Anything raises is bigger than whether you want an agent driving Blender. It is whether the software you depend on will ever get a machine-readable surface, and if the vendor is not going to build one, whether you are willing to generate it yourself and own the result.

For a lot of internal tools, the answer should be yes. The generator is the cheapest path from "no API" to "an agent can use this," and a stateful CLI with JSON output is a good artifact regardless of whether an agent ever touches it. Your humans get a scriptable interface out of the same work.

Just be clear-eyed about the trade. You are not removing the fragility that made UI automation miserable. You are moving it off the screen and into a harness you now maintain, where at least it fails loudly, gets version-controlled, and can be tested. That is a better place for it to live. It is still living with you.

Sources: HKUDS/CLI-Anything on GitHub; CLI-Anything plugin directory; CLI-Hub meta-skill; CLI-Hub registry.