Automate a Prompt-to-Render Video Loop With Agents

The experiment was not another clip—it was the loop

Patrick combined a Sora video test, an ElevenLabs voice change, and the OpenAI Agents SDK to automate a narrow sequence: write the prompt, render, inspect the result, describe the delta, and re-prompt. The goal was to reduce repetitive human handling while preserving final creative approval.

Start with a state machine

Define explicit states: brief approved, prompt ready, render queued, render complete, quality review, revise, accepted, rejected, and stopped. Every transition should have an owner, allowed action, evidence, retry count, and spending limit.

The minimal pipeline

A human approves the brief, reference assets, and maximum budget.
The prompt agent compiles one standalone generation prompt.
The render adapter submits the job and stores provider identifiers.
A sampler extracts representative frames and basic metadata.
The reviewer scores continuity, composition, artifacts, safety, and brief fit.
The revision agent changes only the highest-impact prompt section.
The loop stops after approval, rejection, budget exhaustion, or retry limit.

Make prompt deltas legible

Store the full prompt and a compact change note for every attempt. A useful delta says what changed, why the evidence justified it, what should improve, and what must remain fixed.

Treat voice as a separate asset

Obtain permission for any changed or cloned voice. Store the approved script and voice settings, disclose synthetic audio where appropriate, and review pronunciation, pacing, identity risk, and commercial rights independently from the video.

Control cost and failure

Use low-cost stills or short renders first. Add idempotency keys so a timeout does not create duplicate paid jobs. Require human approval before higher-resolution output, publishing, or exceeding the original budget.

THE TOOL KIT

Software, links, and setup

OpenAI Agents SDK documentation.
Sora.
ElevenLabs.
FFmpeg — frame sampling and media inspection.
A database or durable job store for states, provider IDs, costs, and prompt versions.

COPY, ADAPT, RUN

Prompt pack

Pipeline designer

Design a prompt-to-render-to-QC video workflow as an explicit state machine. Include each state, allowed transitions, input schema, output schema, durable identifiers, retry limit, budget check, human approval gate, and recovery from timeouts or duplicate callbacks.

Quality reviewer

Review these sampled frames against the approved brief. Score continuity, subject identity, composition, physical coherence, artifacts, safety, and narrative fit. Return accept, revise, or reject. If revise, name one highest-impact prompt delta and all invariants that must remain unchanged.

Cost guard

Before submitting a render, calculate estimated calls and maximum spend from current retry count and resolution. Stop if the limit is uncertain or exceeded. Never retry a timed-out provider job until its durable job ID is checked.

THE SHORT VERSION

What to remember

Model the workflow as states and transitions, not a loose chain of prompts.
Store every prompt delta and the evidence behind it.
Use retry, idempotency, and spending guards around paid renders.
Keep final creative, voice-rights, and publishing approval human.

⌘

Build the smallest version that can teach you something.

Then test it, document it, and make the next version easier for someone else to reproduce.