Hybrid reels: live talking-head + AI B-roll under TTS

Problem

Fully generating a talking-head is possible today, but it is expensive, the face drifts frame to frame, and for a legally sensitive offer (real estate) it fails compliance outright: the client needs a recognizable person people trust, not an avatar that can later be sued for “promising yield from the mouth of a non-existent face.”

On the other hand, shooting every cutaway — a construction site, transport, a document close-up, a phone in hand — is also expensive: a field shoot for three seconds of atmosphere.

Hence the hybrid. The speech skeleton is delivered by the expert on camera — trust and face. Between paragraphs go AI bridges: generative faceless video plus synthesized speech. The job is to assemble this into a predictable pipeline where every step can be run by hand, costed in dollars, and fixed in an evening — not a “magic button” that sometimes ships a masterpiece and sometimes lays a 13-second line over a 4-second clip looping three times.

Up front: what this is not. It is not “animate a photo” and not lipsync. Faces in T2V are banned at the system-prompt and brief level. AI owns atmospheric B-roll only — where shooting is expensive and face accuracy is irrelevant because there is no face in frame.

Methodology

Five stages, strict order, deterministic joins. The stack is deliberately boring: plain Node ESM with no Python, a hand-rolled .env parser, fetch to HTTP APIs, local ffmpeg from npm packages.

Principle 1 — immutable skeleton and generated slots

The script splits into two non-mixing parts. The skeleton (source-scenario.md — fixed paragraphs the expert already spoke on camera; they must not be touched or baseline clips on disk stop matching the text. Between paragraphs — placeholders <CUT 1>, <CUT 2>, …

The model fills slots only. It does not rewrite the skeleton or “improve” the whole script. Early runs showed that “fill the slots” is gladly read by an LLM as “write a good script from scratch” — and it rewrites paragraphs that already have footage. The fix is not softer prompting but a contract: the model returns CUT blocks only; the script merges them with the skeleton. Deterministic merge gives idempotent re-runs, bit-identical baseline text, and a clean diff — only inserts change.

Each slot is described in the brief cuts-brief.md: funnel role (pain / method / trust / CTA amplify), speech intent (no finished wording — the LLM writes that), T2V shot type, and join with neighboring paragraphs. Global constraints — 9:16 vertical, 3–5 spoken seconds per insert, no prices, named complexes or developers, yield promises, or mortgage guarantees. That is compliance and a guard against hallucinated numbers at once.

Final concat order lives in timeline.json — a simple alternation of baseline files and CUT numbers. Speech and clips are tied by order, not by timecodes inside one file: any piece can be reshot without rebuilding timings.

Principle 2 — duration is measured, not guessed

Stage order is strict: speech first, then video. From each CUT block take the text before the separator (after it — the English video prompt). That text goes to TTS (ElevenLabs, eleven_multilingual_v2, premade voice). File cut-N.mp3 is saved — and its duration is taken not from the API response but from local ffprobe. The duration source is written into the manifest: "duration_source": "ffprobe".

That measured length — not a “about five seconds” guess from the brief — sets the T2V clip duration. The whole point: do not trust an estimate where you can measure. In result-2 four lines gave 6.0 / 4.6 / 6.3 / 5.1 seconds — no longer “all fives.”

Principle 3 — model limits as a product parameter, not an integration bug

Text-to-video — Seedance 1.5 Pro via OpenRouter Videos API: aspect_ratio 9:16, resolution 480p, generate_audio: false (audio already comes from TTS; native audio generation just inflates per-second cost). Lifecycle — POST job → poll every 5 s → download → write to the manifest.

Seedance accepts duration only in the 4–12 second range. A 3-second request returns HTTP 400. Hence the formula:

requested = max(4, min(10, ceil(speech_duration_sec)))

Lower bound 4 — API requirement. Upper bound 10 — hard cost guard (API formally allows up to 12, but every extra video second is money). ceil — so video is never shorter than speech. If speech still exceeds 10 seconds, video is ordered shorter and at mux the picture loops to the end of audio (-stream_loop -1). Loop is a safety net, not the happy path.

Principle 4 — mux and final concat

For each pair cut-N.mp4 + cut-N.mp3: ffprobe durations, if videoDur + 0.05 < audioDur — loop video (50 ms slack kills false loops from probe rounding), then ffmpeg lays speech over video, trims to speech length, encodes H.264 + AAC. Result — cut-N-av.mp4.

The final reel is assembled separately from timeline.json: phone baseline clips alternate with AI inserts. iPhone .MOV and Seedance .mp4 almost never match on codec/resolution/fps, so concat goes through re-encode onto a single canvas (scale + pad, 30 fps, yuv420p, AAC); silent clips get anullsrc so the audio track does not jump.

Principle 5 — manifests as a ledger

Every paid or measurable stage writes JSON next to the artifacts: cut-tts.json (durations, characters, quota, voice), cut-videos.json (jobId, cost per clip, ordered duration), cut-av.json (looped or not, final lengths). That lets you explain the OpenRouter/ElevenLabs bill clip by clip, rebuild only a part (--only 3), debug “why did video loop,” and not guess where Seedance duration came from. Media stays in .gitignore; git keeps scenarios, brief, timeline, and manifests. Reproducibility without gigabytes.

Artifact

An experimental pipeline grown from a one-off T2V script via OpenRouter and layered with stages. No public repository — client contour. Skeleton:

reels/
├── fill-cuts.mjs             # LLM fills <CUT N> slots; script merges with skeleton
├── generate-cut-tts.mjs      # ElevenLabs → cut-N.mp3 + cut-tts.json (length from ffprobe)
├── generate-cut-videos.mjs   # Seedance → cut-N.mp4 + cut-videos.json
├── mux-cut-av.mjs            # speech over video → cut-N-av.mp4
├── baseline/variant-*/       # source-scenario.md, cuts-brief.md, timeline.json, *.MOV
└── generations/result-*/     # scenario.md, cut-N.*, manifests, cuts-av.mp4

ffmpeg and ffprobe are static binaries from npm packages, wrapped in agent skills concat-videos and extract-video-frame. On Windows this is not a whim: system python from WindowsApps is a Store stub, and ffmpeg is usually missing from PATH.

Two real runs as proof:

A paradox worth stating: result-2 is more expensive on video but more honest visually — no triple scrubbing of a 4-second B-roll under a 13-second line. Ten cents of difference buy inserts that no longer look like a stuck GIF.

For whom and why

For people building multimodal pipelines on external APIs — text-to-video, TTS, LLM — who want a predictable cost per second of picture instead of a “magic button.” The practical interest is broader than reels: any pipeline where several paid models meet through files and manifests.

Three takeaways transfer almost verbatim to other multimodal builds. First: separate the immutable skeleton from generated slots — in data, in code, and in the model’s permissions. Second: multimodal clip duration is measured, not guessed — TTS → ffprobe → T2V duration is the only reliable order. Third: model limits (4–12 s, free vs library voices) are part of product design and belong in the schema, not caught at runtime.

Hybrid talking-head + B-roll is a pragmatic compromise: a human face where trust is needed, generation where atmosphere shoots are expensive. The pipeline stays experimental: no orchestrator service, queues, or UI. But every step is a readable .mjs of tens to hundreds of lines that a one-developer-plus-IDE-agent team can run by hand and fix in an evening.

Links

• Artifact: client experimental pipeline (no public repo); agent skills concat-videos and extract-video-frame for ffmpeg/ffprobe.
• External context: OpenRouter Videos API (Seedance 1.5 Pro), ElevenLabs TTS (eleven_multilingual_v2), ffmpeg-static / ffprobe-static.
• Blog: How we built a hybrid reels pipeline: a real expert plus AI B-roll

Keep reading

• Reproducible agent automation
• Eval as a release gate: catch agent drift before production
• How to Control What AI Agents Do
• All breakdowns in the series →
• Connect the expert as MCP →
• Case study: PersonaClick — Personalization and ML at Scale