Watch Skill

Give every AI agent eyes for video — and a way to check its own work.

Install · Documentation · Examples · Comparison · Roadmap

Watch Skill turns videos, live streams, meetings, and screen recordings into a searchable, timestamped index. An agent can ask what happened, get an answer that cites the exact moment behind it, and ask again tomorrow without processing the video a second time.

When the video is the agent's own browser or desktop session, THE LOOP closes the circle: record the work, critique it against plain-language criteria, and show before and after. That critique is advisory — a model describing pictures. To decide whether the work actually succeeded, attach a verification contract: deterministic checks, frozen before the run, that hold the verdict.

uvx --from "watch-skill[standard]" watch-skill setup

_{THE LOOP catching a $NaN total that an end-state screenshot misses, then showing the fix.}

What it does

Available as Claude Code skills, 39 MCP tools, a CLI, a REST API, and native adapters for LangChain/LangGraph, CrewAI, the OpenAI Agents SDK, LlamaIndex, and AutoGen.

Four things it will not do, because each one is a way of being confidently wrong:

• Answer from a video that changed. Identity follows the bytes, not the path. Overwrite demo.mp4 and the next question returns stale, not yesterday's frames.
• Upload a frame you did not agree to send. A configured API key is not consent. watch-skill plan prints every network action before a run makes one.
• Call an absent judgement a pass. No frames, no OCR, an unreachable model, a timed-out check — all inconclusive. Only a required deterministic check produces a pass.
• Claim a capability it has not checked. watch-skill capture-capabilities says what this machine can actually record, and whether each answer was machine-tested or merely probed.

Install

Two pieces, and you want both. The engine does the work; the skills teach your agent when to reach for it.

# 1. the engine — installs, wires up every AI agent on the machine, backs up each config
uvx --from "watch-skill[standard]" watch-skill setup

# 2. the skills — into Claude Code, Codex, Cursor, Copilot, Gemini CLI, and 20+ more
npx skills add oxbshw/watch-skill -g

Watch Skill ships on PyPI, not npm. The second command runs Vercel's skills CLI, which reads the ten SKILL.md files out of this repository and installs them into whichever agents you have — there is no watch-skill npm package to install, and the engine is Python either way.

Neither needs a clone, and the engine command works the same on macOS, Linux, and Windows — CI runs it on all three on every push.

Prefer a permanent install to uvx fetching on demand?

pipx install "watch-skill[standard]"     # or: pip install "watch-skill[standard]"
watch-skill setup

/plugin marketplace add oxbshw/watch-skill
/plugin install watch-skill@watch-skill
/watch-skill:setup-watch-skill

docker run --rm -i -v watch-skill-data:/data ghcr.io/oxbshw/watch-skill serve

curl -fsSL https://raw.githubusercontent.com/oxbshw/watch-skill/main/scripts/install.sh | sh

powershell -ExecutionPolicy Bypass -c "irm https://raw.githubusercontent.com/oxbshw/watch-skill/main/scripts/install.ps1 | iex"

{ "mcpServers": { "watch-skill": {
    "command": "uvx",
    "args": ["--from", "watch-skill[standard]", "watch-skill", "serve"] } } }

standard is frames, retrieval, and MCP — about 200 MB. watch-skill[all] adds OCR, local Whisper, REST, and the browser THE LOOP drives. watch-skill doctor names anything missing and prints the one command that installs it, so starting small is safe.

Coming from claude-video? Your /watch commands and flags work unchanged — see the migration guide.

First run

watch-skill watch "https://youtu.be/..." "Summarize the important moments."

That prints a report and an id. Everything after it is a lookup against the index, not a second download:

watch-skill ask <video_id> "when does the demo first fail?"
watch-skill search "pricing decision"        # across every video you've watched
watch-skill library ask "what did the team decide about auth?"

Useful flags on watch:

And the rest of the surface:

watch-skill serve                            # MCP over stdio — what agents connect to
watch-skill api                              # REST, port 8748
watch-skill doctor                           # check and repair the setup
watch-skill viewer <video_id> --out r.html   # one self-contained page to share
watch-skill loop viewer <loop_id>            # a run's iterations, compared
watch-skill bench providers                  # compare every provider you have a key for

Transcription, OCR, and search run locally and need no API key. Visual question answering uses whichever provider you already pay for — Anthropic, OpenAI, Gemini, OpenRouter, Groq, Together, Fireworks, DeepSeek, xAI, Mistral, MiniMax, Moonshot, Z.ai, or Qwen — or nothing at all with a local Ollama model. Anything else that speaks the OpenAI format (vLLM, LM Studio, llama.cpp, LiteLLM, Azure OpenAI, a company gateway) works through the custom provider:

watch-skill setup-vision --provider groq            # or any of the above
watch-skill setup-vision --provider custom \
  --base-url http://127.0.0.1:8000/v1               # your own server

See Getting started for manual installation and Configuration for provider and privacy settings.

Why use it

• Evidence instead of frame dumps. Scene detection and perceptual deduplication spend the frame budget on distinct moments. Answers include timestamps, confidence, and the evidence used to support them.
• Persistent video memory. Analyze once, ask again without downloading or transcribing the same video. Hybrid full-text and vector retrieval works within one video or across the entire library.
• Local-first processing. Original-language captions are preferred, local Whisper is the default fallback, and cloud speech-to-text is opt-in. An Ollama configuration keeps the complete pipeline on the machine.
• Flow verification. THE LOOP records an agent's browser, screen, or window and checks the result against plain-language criteria, producing a before/after comparison. The model's read of that recording is advisory; a verification contract turns it into a decision, and its evidence bundle is hash-bound so an edited result stops verifying.
• Corrections that persist. report_mistake stores a local lesson, applies it to related questions, and turns it into a replayable evaluation.
• Measured cost controls. Text-first answers, semantic caching, configurable token budgets, and explicit cheapest, quality_first, and offline_only policies keep the trade-offs visible.
• Multilingual retrieval. Script-aware OCR routing, Arabic normalization, CJK substring matching, and multilingual embeddings support questions across languages.

Sixteen providers is a menu, not an answer, so there is a benchmark that settles it on your own keys: watch-skill bench providers reads the same committed frames with every provider you have configured and prints char-hit rate, latency, and cost from each one's reported tokens — see method and results.

The repository includes reproducible cost and perception benchmarks. Product claims in this README link to the relevant implementation notes or testable example rather than relying on unqualified marketing numbers.

Works with your agent

The setup command detects supported clients and updates their configuration with a backup. Manual guides are available for every entry below.

Claude Code	Claude Desktop	Cursor	Codex CLI
Cline	Windsurf	Gemini CLI	VS Code
GitHub Copilot CLI	Kimi Code	Qwen Code	OpenCode
Goose	OpenHands	Kilo Code	Qodo
Agent Zero	OpenClaw	Pi	Hermes
Zed	Roo Code	Continue	JetBrains
Amp	Aider

Native tools are also available for LangChain/LangGraph, CrewAI, OpenAI Agents SDK, LlamaIndex, and AutoGen; any other framework can use REST or MCP.

Why both skills and MCP

MCP gives an agent 39 tools. Skills give it the judgement about when to use them — that a screen recording in the conversation is worth watching, that a follow-up question should hit the index instead of re-processing, that a UI change deserves a verification pass. An agent with only the tools waits to be told; an agent with the skills reaches for them.

That is why npx skills add oxbshw/watch-skill -g is step two of the install and not an optional extra. Pick individual ones with --skill <name>, or list them first:

npx skills add oxbshw/watch-skill --list

The full compatibility matrix separates machine-tested, machine-configured, and documentation-verified integrations. If your agent is missing, the adapter template provides a short contribution path.

Browser Runtime

Watch Skill has one browser subsystem with two modes. Observer mode watches someone else work and verifies the result. Operator mode does the work and holds itself to the same standard.

from watch_skill.operate import (
    Action, ActionKind, BrowserRuntime, Expectation, SideEffect, Target,
)

runtime = BrowserRuntime(source)          # an already-running browser session
result = runtime.run_task("save the display name", [
    Action(kind=ActionKind.CLICK, intent="save",
           target=Target(role="button", name="Save"),
           side_effect=SideEffect.REVERSIBLE,
           expect=Expectation(text_present="Saved", network_ok=True)),
])

result.verified          # False — the page said Saved, PATCH /api/save returned 500
result.receipts[-1].reason

The rule the runtime is built around: dispatching an action is not the same as proving its effect. Playwright returning from click() proves a click was delivered and nothing more, so every action carries an expectation written down beforehand, and the verdict is the comparison. An action with no expectation is UNVERIFIED, never SUCCEEDED.

That is what catches the failure mode nothing else does — a page that renders success over a request that failed. network_ok correlates the requests made during the step, so "Saved" over a 500 is a failure with the request named in the receipt.

Other properties worth knowing:

• Targets resolve deterministically first — accessible role and name, then label, test id, placeholder, selector, text. Vision is last because it is the most expensive signal and the least stable across a redeploy.
• Ambiguity is refused, not guessed. Two buttons named "Delete account" is not a case where the first one is probably right.
• Retries respect side effects. Clicking "Next" again is fine; clicking "Buy" again is not. Recovery never repeats an action that may have taken.
• Every step produces a receipt — how the target was found and with what confidence, what changed, which requests ran, the verdict, and any recovery.

Run the benchmark against the bundled local fixture site:

python -m watch_skill.operate.benchmark --out build/benchmark

It scores false-success rate — tasks where the runtime claimed the goal was met and the server disagrees — because task success rate on its own counts a confident wrong answer as a win. Ground truth comes from the fixture server's own state, not from anything the browser reported.

On that nine-task fixture benchmark every ground-truth verdict was classified correctly and no false-success verdict was produced. Nine tasks on one synthetic site is a regression gate rather than a capability claim: it does not cover real websites, authentication, or shadow DOM, and no other tool was measured under the same method. Full method and results and the design.

Common workflows

Build a searchable video library

watch-skill batch ./recordings --limit 50
watch-skill library overview
watch-skill library ask "What did the team decide about authentication?"

library ask synthesizes evidence across videos and retains per-video timestamp provenance. The library example demonstrates a question whose answer is distributed across four clips.

Verify an agent's browser work

watch-skill loop start "browser:http://127.0.0.1:3000" \
  "Checkout completes and the total is always a valid currency amount"

The loop captures the full interaction, critiques failures, and records the before/after comparison once the agent applies a fix — the run shown at the top of this page. Example 14 walks through that transient $NaN bug.

The critique is one model's reading of the recording. To make success a decision rather than an opinion, attach deterministic checks:

watch-skill verify run checkout-contract.json --dir .

pass requires every required check to pass. A check that fails, times out, or never runs makes the run inconclusive — never a pass. See Verification.

Write up a video, with every line cited

watch-skill notes <video_id> --write notes.md

Chapters, what was said, what was on screen, and the frames to prove it — assembled from the index rather than generated, so every statement carries the timestamp it came from and can be checked against that second in the source. No model runs in the path, so the same index always produces the same document.

Export an offline report

watch-skill viewer <video_id> --out video-report.html

The generated page contains its frames, transcript, OCR, cached answers, and cited evidence. It has no external runtime dependencies and can be opened without a server.

Benchmarks

Measured on stated hardware, or not stated at all. Every result is generated by a command anyone can rerun, from fixtures whose ground truth is committed beside them.

The video-backend benchmark is the newest and the strictest. It grades a provider on whether Watch Skill could store what it returns as a citation: are the frames the ones that were asked for, do the timestamps mean what they appear to mean, does a transcript land where the speech is, and does a written analysis say anything traceable to the source.

It runs against two kinds of input. A generated fixture carries ground truth exact to the millisecond — hard cuts on frame boundaries, a 25-frame ladder where every frame has its own identity, speech placed at offsets the generator chose. Real footage has no authored truth, so the benchmark derives it from the file: decode the window around each probe and locate the returned frame inside it.

The first subject is Adversal MCP 0.1.4, tested at the vendor's request days after release. It is a preview of an early version, and it is written as one — including the parts that did not work, and the axes where the two systems came out exactly equal because both call the same underlying tool.

The chart is drawn from the run's own raw JSON every time the report is generated, so it cannot drift from the numbers beside it. Two of the seven axes are exact ties, and they are labelled as such: both systems transcribed the fixture's script without a single word error, and both return the same frame for a requested time because both shell out to the same ffmpeg seek. The gap on the remaining axes is largely one reproducible bug, which the report documents with a reproduction rather than a verdict.

Examples

The examples progress from a first watch to agent integration, cross-video memory, and self-verification.

See the example catalog for prerequisites, expected output, and a recommended path through all 20 examples.

Architecture

All interfaces call the same Python core. Skills and agent adapters decide when to use Watch Skill; acquisition, perception, transcription, indexing, answering, and verification remain in src/watch_skill.

flowchart LR
    A["Agents and frameworks"] --> S["Skills · MCP · CLI · REST"]
    S --> AC["Acquire"]
    AC --> P["Scenes · OCR · transcript"]
    P --> I[("Persistent index")]
    I --> Q["Answers · extraction · library"]
    I --> L["Lessons and evaluations"]
    V["Browser · screen · stream capture"] --> C["Loop critic"]
    C --> I

Read Architecture for the data model, provider boundaries, and extension points.

The Workspace

This repository holds both halves of the product. The Python engine above is the root; workspace/ is the agent workspace built on DeepSeek Harness — the Watch DSH plugins, the Memory service, and the Web and Desktop applications.

The two are independent to install. pip install watch-skill reads pyproject.toml at the root and never descends into workspace/, and the workspace needs Node and pnpm but no Python.

cd workspace && node scripts/bootstrap.mjs

Start at workspace/docs/setup.md; the workspace has its own README, ADRs and gate suite.

Documentation

Development

git clone https://github.com/oxbshw/watch-skill
cd watch-skill
uv sync --extra all
uv run pytest
uv run ruff check .

See CONTRIBUTING.md for test tiers, documentation standards, and the agent-adapter checklist. Security and privacy reports are covered by SECURITY.md.

Listed on

Independent directories that index Watch Skill. They are maintained by their operators, so the details there can lag a release.

• Agent Skills Hub
• Neuralbox