The brief sounded deceptively simple: short vertical clips where a real-estate expert talks to camera, and between his points sit clean AI cutaways — construction sites, transit, a document in close-up, a phone in hand. The face is filmed by a human; the atmosphere is generated by a neural net. What started as a one-off generation script grew into a five-stage pipeline, and along the way we collected nearly every gotcha this kind of build has to offer.
Below is how it's wired, why there are exactly five stages, and the order in which they broke before we lined them up correctly. This is for people who stitch external models — text-to-video, TTS, LLM — into one pipeline and want a predictable cost per second of footage instead of a "magic button."
Why hybrid, and not "generate everything"
The temptation was obvious: synthesize the talking-head end to end and never send a human to a shoot. But a fully synthetic talking-head today is expensive, the face drifts frame to frame, and for real estate — an offer where words carry legal weight — an avatar won't do. The client needs a recognizable expert people trust, not a digital double who gets asked next month where he pulled "guaranteed returns" from.
Shooting every cutaway is expensive too: a site trip for three seconds of mood. Hence the hybrid. The expert speaks the scaffold himself — that's the face and the trust. Between his paragraphs go the AI bridges: faceless generated video plus synthesized speech.
One decision was firm from the start and stayed that way: we do not generate the face. The system prompt and the brief explicitly forbid faces in text-to-video. AI handles only atmospheric B-roll — where the shoot is expensive and face fidelity doesn't matter, because there's no face in frame.
Five stages and a strict order
The stack is deliberately boring: plain Node ESM, no Python, a hand-rolled .env parser, fetch against HTTP APIs, local ffmpeg from npm packages. No orchestrator, no queues, no UI — every step is a readable .mjs of tens to hundreds of lines you can run by hand.
The order: an LLM fills the scenario slots → TTS voices the cutaways → text-to-video renders the picture → mux lays speech over video → the final concat interleaves live footage with AI inserts. Here's what turned out to be non-obvious at each step.
Stage 1. The script: what the model writes, and what we never touch
The script splits into two parts that must not mix. There's the scaffold — the fixed paragraphs the expert has already spoken to camera. You can't touch them: shift a word and the baseline clip on disk stops matching the text. Between the paragraphs sit placeholders <CUT 1>, <CUT 2> — the slots for AI inserts.
The first thing we tripped over: we asked the model to "fill the slots," and it rewrote the entire script. All of it. Fluent, coherent — and out of sync with the footage already shot. Turns out "fill the slots" reads to an LLM as "write a good script from scratch," especially when it dislikes something about the one it's given.
Coaxing the model in the prompt — "don't touch the fixed paragraphs" — is pointless; it's under no obligation to obey prose in the prompt. It's the same lesson as a security perimeter that can't rest on a system prompt: a constraint holds when it's wired into code, not into a polite request. So we changed the contract: the model returns CUT blocks only, and the script does the merge with the scaffold. A deterministic merge gives idempotent re-runs, a byte-for-byte match on the baseline text, and a clean diff — only the inserts change. The raw model response is also saved separately, so you can rebuild the script without a fresh API call when you're editing merge logic rather than content.
Every slot is described in a brief: its role in the funnel (pain / method / trust / call-to-action boost), the intent of the line (no ready wording — the LLM writes that), the shot type, and how it joins the neighboring paragraphs. Global constraints — vertical 9:16, three to five seconds spoken, no prices, no named developments, no promises of returns or mortgage guarantees. That's both compliance and insurance against the model inventing a number.
This stage costs pennies: about $0.009 per run on a cheap chat model. Dust next to the video.
Stage 2. Speech first, and take the length from the file, not from your head
Here's the rule that governs the whole pipeline: speech first, video second. From each CUT block we take the line, send it to ElevenLabs (multilingual model, a premade voice), and get cut-N.mp3.
And here's the subtlety that later saved the whole edit. We take the clip duration not from the API response and not from a "roughly five seconds" estimate, but by measuring the finished mp3 with local ffprobe. The source of the length goes straight into the manifest: "duration_source": "ffprobe". Same instinct as treating a measurable result over a gut feel: there it's about gauging an agent with a harness rather than by eye, here it's about measuring a line by probing the file rather than guessing. In a real run, four lines came out at 6.0 / 4.6 / 6.3 / 5.1 seconds. Not "all five," as the brief promised.
That measured length is what drives the video duration downstream. Trust the estimate instead and you get drift on every other insert.
Stage 3. Text-to-video, and a Seedance that won't do three seconds
The picture is generated by Seedance 1.5 Pro via OpenRouter: vertical 9:16, 480p, audio generation off — the audio already exists from TTS, and the model's native audio just makes a second more expensive for nothing.
The first surprise here: order a three-second clip and you get HTTP 400. Seedance, it turns out, accepts a duration only in the 4–12 second range, and none of the marketing "short clips" copy says so. Which is where this formula came from — the one that doubles as the main cost-versus-quality lever:
requested = max(4, min(10, ceil(speech_duration_sec)))The lower four is the API requirement. The upper ten is a hard cost guard: the API formally allows up to 12 seconds, but every extra second of video costs money, and we decided to pay for no more than ten. ceil keeps the clip from ending before the speech does when the line spills just past a whole number of seconds.
Stage 4. Mux: speech over video, and the hypnotic loop
For each "video + speech" pair the script probes both durations and, if the video is shorter than the audio, loops the picture to the end of the line. A 50-millisecond margin in the condition kills false loops from probe rounding. Then ffmpeg lays the speech on top, trims to the length of the line, and encodes to H.264 + AAC.
And this is where a symptom from the early run surfaced — the one we nicknamed hypnotic: a four-second B-roll spinning two or three times under one long line, the insert looking like a stuck GIF. The cause was exactly that the video was requested at a fixed short length while the model wrote long speech — nobody enforced the brief's "three to five seconds."
The fix took three moves at once: TTS first with length from ffprobe; ceil(speech) into Seedance with a ten-second cap; and shortening the speech itself in the brief to one or two sentences. We kept the loop — but as a safety net for the rare case, not as the normal mode.
Comparing the two generations of the pipeline turned out instructive:
| Early run | After tightening | |
|---|---|---|
| Speech per slot | long paragraphs (up to ~13s) | 1–2 short sentences (~5–6s) |
| Video duration | fixed 4s | from TTS: 6 / 5 / 7 / 6s |
| Looping | all 4 clips | none |
| Video cost | ~$0.18 | ~$0.28 |
The paradox is right there: tightening made it more expensive. But more honest — no triple-spin of one clip under a long line. Those ten cents of difference buy back an insert that no longer hypnotizes the viewer.
Stage 5. Splicing phone footage with a neural net
The final reel is assembled from timeline.json — a plain list interleaving baseline clips from the phone with AI-insert numbers. The link between speech and clips rests on order, not on timecodes inside one file: any piece can be reshot without rebuilding the timing.
The last gotcha waited at the seam between two worlds. iPhone .MOV and Seedance .mp4 are almost always incompatible in codec, resolution, and frame rate — trying to splice them by stream-copy gives black frames or audio drift. So for the full reel we go straight to re-encode with a single canvas: scale + pad to a common size, 30 fps, yuv420p, AAC. Clips without an audio track get silence padded in, so the sound doesn't jump at the seams. One such reel — six live shots plus four inserts — came out around 81 seconds and 113 MB. About right for vertical phone HD.
Manifests as an accounting ledger
Every paid or measurable stage writes a JSON next to its artifacts: speech durations and quota spent, the jobId and cost of each clip, whether the video looped. It sounds like bureaucracy, but it pays off immediately. From these files you can explain the OpenRouter and ElevenLabs bill clip by clip, rebuild a single insert instead of the whole reel, debug "why did the video loop," and stop guessing where the Seedance length came from.
The media itself (mp3, mp4, the source .MOV) sits in .gitignore. What stays in the repo is scripts, briefs, timeline, and manifests — reproducibility without gigabytes in the history.
The gotchas that didn't make the main plot
A couple of stories worth their own paragraph, because each ate an evening.
A "free" voice that costs 402. We grabbed a library voice in ElevenLabs with an honest-looking "available to free users" tag — and got paid_plan_required. As it turns out, on the free tier the API only serves premade voices; shared/library ones need a paid plan even when the catalog hints otherwise. free_users_allowed: true in the catalog does not mean the right to synthesize with a free key. For a "native" Russian timbre you need a paid plan or a different provider.
ffprobe-static is not ffmpeg-static. One npm package exports a string path, the other an object { path }. Mix them up and spawn falls over with undefined instead of a path. The cure is one line everywhere you call a binary: typeof mod === "string" ? mod : mod?.path.
Windows with no ffmpeg and a fake Python. On the test machine the system python opened the app store (a WindowsApps stub), and ffmpeg wasn't in PATH at all. A Node-only pipeline plus static binaries pulled in by npm install into the skills closed the question without any "please install ffmpeg for us."
Where the money is
The economics of one run over four inserts, 480p, short lines:
| Stage | Order of magnitude |
|---|---|
| Slot fill (cheap chat model) | ~$0.01 |
| ElevenLabs TTS (free quota) | $0 (or cents on paid) |
| Seedance × 4 | ~$0.20–0.35 |
| Mux and concat | $0 (local CPU) |
| API total | ~$0.25–0.40 |
The expensive part here isn't the API — it's the human: filming the live scaffold, the edit review, the legal read of the offer. The pipeline cheapens exactly the insert iterations — regenerating one CUT is cheaper than reshooting the whole reel. Which is precisely why the ten-second cap per clip and the ban on the model's native audio aren't micro-optimizations but product decisions. The OpenRouter video-model collection, by the way, sorts by popularity, not by price — it's easy to pick an expensive model by not paying attention.
What carries over to other builds
Three takeaways transfer almost verbatim to any multimodal pipeline, not just reels.
First: separate the immutable scaffold from the generated slots — in the data, in the code, and in the model's permissions. Let an LLM rewrite everything and it will rewrite everything.
Second: the duration of a multimodal clip is measured, not guessed. TTS → ffprobe → video length is the only order that doesn't collapse into drift.
Third: model constraints are part of the product design, not an integration bug. The 4–12 second range, free versus library voices, the cost of native audio — you bake these into the schema up front, not catch them at runtime on an HTTP 400.