Learning From a 58-Second Video

Most of what people know how to do is in short videos, not documents. Turning a reel into a procedure an agent can execute, and then checking each step by doing it, is a small pipeline with a large payoff.

Ask someone how they set up a 3D printer, tune a guitar pedal or configure a router and they will send you a one-minute video. Documents are rare; reels are everywhere. An agent that only learns from text misses most of what its owner actually knows. The pipeline below turns a short video into something the agent can act on, and it fits on one page.

Five steps

Transcribe in pieces. Audio is split into chunks and transcribed with a local speech model, keeping timestamps. Chunks matter: a long file transcribed at once loses alignment, and the timestamps are what let the agent say "the flow test starts at 0:34".

Digest. A small model writes a short summary of what the video is about and who it is for. This is what gets embedded for search; the transcript itself is too long and too noisy to retrieve well.

Extract the procedure. The transcript is turned into numbered steps with the preconditions each needs: "printer unlocked", "SD card mounted". Steps the video skips but implies are added and marked as inferred.

Verify by doing. This is the step that makes the result trustworthy. The agent attempts each step with its tools where it safely can, marks it verified or blocked, and records what blocked it. In the example on our product page, four of six steps verified and two were blocked on a locked printer. The playbook says so, rather than pretending.

Save the playbook and a lesson. The procedure, its verification state and the source timestamps go into the knowledge base. If something went wrong, one lesson is written in the imperative: "check the SD card is mounted before loading the flow test".

Why verification changes the economics

Without verification, a transcribed procedure is a guess with good formatting. With it, the second time the owner asks for the same job, the agent runs the verified steps without asking and only stops at the ones that were blocked. That is the difference between a search engine over videos and an assistant that has watched them.

Numbers from the test fixtures

The pipeline is tested on two short fixtures: a one-minute audio clip and a 58-second reel. Transcription, chunking, digest, extraction and the verify-by-doing pass all run in the test suite on a laptop CPU, in under two minutes. The whole module is about 400 lines.

Resources