rea-video Tells Your Coding Agent How Any Video Was Made, Shot by Shot

It sorts every shot into filmed, AI-generated or code-rendered, then writes a remake plan. Its own tests show exactly where that sorting breaks.

Short videos have stopped being one kind of thing. A 20-second product clip can now be three different productions stitched together: a real person filmed on a phone, an AI-generated scene in the middle, and animated text built in code. From the outside they blur into one smooth edit, and that makes "how would I make something like this?" a surprisingly hard question. A new open-source tool now answers it shot by shot, and its own test results show exactly when that answer can be trusted.

It matters because each kind of shot is made in a completely different way. A filmed shot needs a camera, a light and a set. An AI shot needs a prompt and a video model. A code-rendered shot needs a programmer and a tool like Remotion (a library for making videos with React code). Guess wrong about which one you are looking at, and your remake plan is wrong from the first step.

rea-video, an open-source tool that was #14 on Trendshift's GitHub momentum board on November 4, attacks that question directly. You hand it a video through Claude Code or Codex, and it splits the video into shots, labels how each one was made, and writes a plan for remaking it. The interesting part is not the remake. It is that the tool publishes where its labels go wrong, and the failure pattern is worth knowing even if you never install it.

What it does, in plain terms

The project's own one-line pitch: give it a talking-head clip, an AI video, a 3D explainer or a mixed edit, and "it tells you shot by shot how the visuals, styled text, editing and audio were made, then gives you a plan to make one yourself." (README)

It works in four steps:

Split. Cut the video at every shot change and separate the audio into voice, music and sound effects.
Gather evidence. Measure the frames, look at the images and listen to the sound. The measurements include frame cadence (the rhythm at which frames change, which differs between cameras and generators), sensor noise (the faint grain every real camera sensor leaves), whether the whole image moves as one flat plane, and any AI-generation tag hidden in the file's metadata. The visual check looks at hands, text and physics. The audio check pulls out tempo, key, and whether a voice is singing or speaking.
Decide. Hard evidence, like an AI tag or a telltale frame rhythm, outranks "looks like." When signals conflict, the shot gets flagged for a closer look.
Plan the remake. Group the shots according to what the target video model can actually do, such as its maximum clip length and how many reference images it accepts.

The output is a one-line verdict ("41% AI video + 59% code UI, 8 s of real screen recording, library music at 122 BPM" is the README's example), a remake plan split into what to film, what to generate and what to code, and a full report.html with frames, evidence and per-shot notes. Subtitles export as SRT or ASS files.

A few example verdicts from the README show what this looks like in practice. A 70-second product promo came back as 83% code-driven motion graphics and 12% screen recording, so the plan was one Remotion project and no video model at all. An AI music video's song was correctly flagged as AI-generated. And a live-action product clip that mixed in AI footage was caught by an AI tool's watermark and garbled text on a gift box.

The part worth reading: where it gets the labels wrong

Most tools in this space publish one big accuracy number. rea-video publishes several, and the differences between them are the lesson.

Measured as the share of video time where the production method was labeled correctly, with only the free offline tools and no AI vision model, the README reports 98.3% on its tuning set of 15 videos. That is the set the rules were designed on, so it is the least trustworthy number. On two held-out sets of 9 and 7 videos, it reports 99.6% and 98.9%. The project's own results file is unusually candid about those too: after the first holdout run it added a frame-rate rule based on two mistakes there, so it says that set is no longer fully independent. Only the 7-video set ran for the first time after the rules were finalized. (eval/RESULTS.md)

Then the numbers drop. On 10 videos downloaded from WeChat articles, which the platform had recompressed, the offline tools scored 88.6%. Recompression wipes out the faint sensor grain that separates a real camera from a generator, so one of the strongest clues simply disappears. The README says adding a vision model (Gemini) brought that set back to 99.6%, but the detailed results file, recomputed under its newer weighting, shows 88.8% for Gemini alone and 100% only after agent visual review. It also labels this set a development set, because rules were added after looking at these videos. The README's live-action unboxing case shows the same effect up close: after the platform re-encoded the file, the offline tools called 35% of an entirely real video AI, and agent review corrected it to 97% live-action.

That is the insight in one line: a provenance label (a call about where a shot came from) is only as good as the evidence that survives the upload. Original files carry fingerprints. Social video has been sanded down. Any tool, or any person, judging "is this AI?" from a compressed download is working with half the clues.

There is a second, subtler finding in the docs. The command-line guide and the results file record a test where an agent that could not view images (DeepSeek's dsh agent running DeepSeek V4.1 Flash) reviewed the tool's calls on 11 videos with known answers. The agent overturned the tool four times: twice correctly, twice wrongly, and both wrong overrides came from judging pixel statistics rather than actually looking at the images. The fix the author shipped was a rule in the agent's instructions: an agent that cannot see images may not overturn the tool's production-method call. (docs/cli.md, eval/RESULTS.md)

"Have the agent double-check" only helps if the agent can actually see.

Put this into practice

What you need: ffmpeg, Python 3.10 or newer, and Claude Code, Codex or another agent that can run commands. The offline mode is free. A vision model is optional and billed by whichever provider you use.

The agent-first install. The README's quick start is a single sentence you give your agent:

Install https://github.com/JuneYaooo/rea-video, then break down this video: /path/to/video.mp4

The agent installs the command-line tool, reads the workflow from the repo's skill file and runs it. Expect the reports and detailed docs to be in Chinese; the README says your agent reads them and answers in your language, and six example reports in the repo are translated into English if you want to see the output before installing.

The manual route, if you would rather see each step. From a clone of the repo, the command-line guide shows:

pip install -e ".[ocr]"
rea-video decompose ref.mp4 -o out/ref          # offline evidence only
rea-video decompose ref.mp4 -o out/ref --vlm    # add a vision model
rea-video review out/ref                        # per-shot visual review tasks
rea-video apply-review out/ref --by <agent>     # merge the agent's answers into the report

The first run downloads a voice-separation model of about 67 MB into ~/.cache/rea-video/models, and the guide says a 30-second video takes about a minute without a vision model. The --vlm flag, which adds a vision-language model, reads any API that accepts OpenAI's request format from REA_VLM_BASE_URL, REA_VLM_API_KEY and REA_VLM_MODEL.

If a label is wrong, you can correct it yourself, for example rea-video correct out/ref --shots 3 --label still_motion --note "this is stop-motion".

How to get trustworthy results:

Use the original file whenever you can. If you are studying your own past videos or a client's, ask for the export, not the platform download. That is where the offline tools are strongest.
For anything downloaded from a social platform, add a visual check. Turn on a vision model (the README names Gemini or Doubao; you give the agent the endpoint and key), and run the review step in an agent that can actually view images. In the detailed results on recompressed video, that agent review step did most of the work.
Read the evidence as well as the verdict. The report lists why each shot got its label. A shot labeled AI because of a metadata tag is a different kind of claim from one labeled AI because the hands looked odd.
Treat the remake plan as a draft. Code shots become a suggested Remotion project, AI shots get reverse-engineered prompts, and filmed shots get a shooting plan with camera positions and lighting. Automatic generation of the AI shots runs through a companion project, and the README says it does a free dry run before it calls any paid model.

Honest limitations

The accuracy figures are the project's own, on small sets: 15, 9, 7 and 10 videos. The results file is open about which sets were used to tune rules, which is more than most tools offer, but the repo points to no independent rerun. Its README and its detailed results file also disagree on the Gemini-only score for the WeChat set, so trust the detailed file.

Compressed video is the weak spot, and it is where most real-world inputs live. The offline tools fell to 88.6% on recompressed WeChat videos, and on two Douyin videos (a live unboxing and an AI plus live-action mix) the Chinese README reports 46% offline, 64% with a vision model, and 97.5% only after agent review. It also mislabels footage of real phone screens as 3D or AI.

The tool is young. It is version 0.1.0 in its package file, has no GitHub releases, and had a few hundred stars on November 4 (counts differed by source). The docs are mostly in Chinese.

The remake demos lean on the author's own setup: a self-hosted MiniMax H3 video model with gpt-image-2 keyframes. Your results will depend on the models you actually have.

And the obvious one, which the README states plainly: this is for learning a video's structure, pacing and technique, then remaking it with material you have rights to. Do not reuse other people's footage, music, characters or likeness, and follow your platform's AI-labeling rules when you publish.

Where this leaves you

If you make short videos, rea-video is worth an afternoon on your own back catalog, where you have original files and know the right answers. That is the fastest way to learn whether its labels match your eye.

If you only care about the bigger question of spotting AI video, the project's own numbers are the takeaway. Fingerprints live in original files. Once a video has been recompressed, every detector, human or software, needs to actually look.

Sources: rea-video README (English), reviewed at commit d904b1e, Nov 4, 2026; eval/RESULTS.md; docs/cli.md; Chinese README; LICENSE (Apache-2.0); Trendshift, read about 15:10 ET Nov 4, 2026.