{"id":80,"date":"2026-07-28T12:34:44","date_gmt":"2026-07-28T12:34:44","guid":{"rendered":"https:\/\/www.dobryakov.net\/blog\/80\/"},"modified":"2026-07-28T12:34:44","modified_gmt":"2026-07-28T12:34:44","slug":"hybrid-reels-pipeline","status":"publish","type":"post","link":"https:\/\/www.dobryakov.net\/blog\/80\/","title":{"rendered":"How we built a hybrid reels pipeline: a real expert plus AI B-roll"},"content":{"rendered":"

The brief sounded deceptively simple: short vertical clips where a real-estate expert talks to camera, and between his points sit clean AI cutaways \u2014 construction sites, transit, a document in close-up, a phone in hand. The face is filmed by a human; the atmosphere is generated by a neural net. What started as a one-off generation script grew into a five-stage pipeline, and along the way we collected nearly every gotcha this kind of build has to offer.<\/p>\n

<\/p>\n

Below is how it's wired, why there are exactly five stages, and the order in which they broke before we lined them up correctly. This is for people who stitch external models \u2014 text-to-video, TTS, LLM \u2014 into one pipeline and want a predictable cost per second of footage instead of a "magic button."<\/p>\n

Why hybrid, and not "generate everything"<\/h2>\n

The temptation was obvious: synthesize the talking-head end to end and never send a human to a shoot. But a fully synthetic talking-head today is expensive, the face drifts frame to frame, and for real estate \u2014 an offer where words carry legal weight \u2014 an avatar won't do. The client needs a recognizable expert people trust, not a digital double who gets asked next month where he pulled "guaranteed returns" from.<\/p>\n

Shooting every cutaway is expensive too: a site trip for three seconds of mood. Hence the hybrid. The expert speaks the scaffold himself \u2014 that's the face and the trust. Between his paragraphs go the AI bridges: faceless generated video plus synthesized speech.<\/p>\n

One decision was firm from the start and stayed that way: we do not generate the face.<\/strong> The system prompt and the brief explicitly forbid faces in text-to-video. AI handles only atmospheric B-roll \u2014 where the shoot is expensive and face fidelity doesn't matter, because there's no face in frame.<\/p>\n

Five stages and a strict order<\/h2>\n

The stack is deliberately boring: plain Node ESM, no Python, a hand-rolled .env<\/code> parser, fetch<\/code> against HTTP APIs, local ffmpeg from npm packages. No orchestrator, no queues, no UI \u2014 every step is a readable .mjs<\/code> of tens to hundreds of lines you can run by hand.<\/p>\n

The order: an LLM fills the scenario slots \u2192 TTS voices the cutaways \u2192 text-to-video renders the picture \u2192 mux lays speech over video \u2192 the final concat interleaves live footage with AI inserts. Here's what turned out to be non-obvious at each step.<\/p>\n

Stage 1. The script: what the model writes, and what we never touch<\/h3>\n

The script splits into two parts that must not mix. There's the scaffold<\/strong> \u2014 the fixed paragraphs the expert has already spoken to camera. You can't touch them: shift a word and the baseline clip on disk stops matching the text. Between the paragraphs sit placeholders <CUT 1><\/code>, <CUT 2><\/code> \u2014 the slots for AI inserts.<\/p>\n

The first thing we tripped over: we asked the model to "fill the slots," and it rewrote the entire script. All of it. Fluent, coherent \u2014 and out of sync with the footage already shot. Turns out "fill the slots" reads to an LLM as "write a good script from scratch," especially when it dislikes something about the one it's given.<\/p>\n

Coaxing the model in the prompt \u2014 "don't touch the fixed paragraphs" \u2014 is pointless; it's under no obligation to obey prose in the prompt. It's the same lesson as a security perimeter that can't rest on a system prompt<\/a>: a constraint holds when it's wired into code, not into a polite request. So we changed the contract: the model returns CUT blocks only, and the script does the merge with the scaffold.<\/strong> A deterministic merge gives idempotent re-runs, a byte-for-byte match on the baseline text, and a clean diff \u2014 only the inserts change. The raw model response is also saved separately, so you can rebuild the script without a fresh API call when you're editing merge logic rather than content.<\/p>\n

Every slot is described in a brief: its role in the funnel (pain \/ method \/ trust \/ call-to-action boost), the intent of the line (no ready wording \u2014 the LLM writes that), the shot type, and how it joins the neighboring paragraphs. Global constraints \u2014 vertical 9:16, three to five seconds spoken, no prices, no named developments, no promises of returns or mortgage guarantees. That's both compliance and insurance against the model inventing a number.<\/p>\n

This stage costs pennies: about $0.009 per run on a cheap chat model. Dust next to the video.<\/p>\n

Stage 2. Speech first, and take the length from the file, not from your head<\/h3>\n

Here's the rule that governs the whole pipeline: speech first, video second.<\/strong> From each CUT block we take the line, send it to ElevenLabs (multilingual model, a premade voice), and get cut-N.mp3<\/code>.<\/p>\n

And here's the subtlety that later saved the whole edit. We take the clip duration not from the API response and not from a "roughly five seconds" estimate<\/strong>, but by measuring the finished mp3 with local ffprobe<\/code>. The source of the length goes straight into the manifest: "duration_source": "ffprobe"<\/code>. Same instinct as treating a measurable result over a gut feel<\/a>: there it's about gauging an agent with a harness rather than by eye, here it's about measuring a line by probing the file rather than guessing. In a real run, four lines came out at 6.0 \/ 4.6 \/ 6.3 \/ 5.1 seconds. Not "all five," as the brief promised.<\/p>\n

That measured length is what drives the video duration downstream. Trust the estimate instead and you get drift on every other insert.<\/p>\n

Stage 3. Text-to-video, and a Seedance that won't do three seconds<\/h3>\n

The picture is generated by Seedance 1.5 Pro via OpenRouter: vertical 9:16, 480p, audio generation off \u2014 the audio already exists from TTS, and the model's native audio just makes a second more expensive for nothing.<\/p>\n

The first surprise here: order a three-second clip and you get HTTP 400. Seedance, it turns out, accepts a duration only in the 4\u201312 second<\/strong> range, and none of the marketing "short clips" copy says so. Which is where this formula came from \u2014 the one that doubles as the main cost-versus-quality lever:<\/p>\n

requested = max(4, min(10, ceil(speech_duration_sec)))<\/code><\/pre>\n
The lower four is the API requirement. The upper ten is a hard cost guard<\/strong>: the API formally allows up to 12 seconds, but every extra second of video costs money, and we decided to pay for no more than ten. ceil<\/code> keeps the clip from ending before the speech does when the line spills just past a whole number of seconds.<\/p>\n
Stage 4. Mux: speech over video, and the hypnotic loop<\/h3>\n
For each "video + speech" pair the script probes both durations and, if the video is shorter than the audio, loops the picture to the end of the line. A 50-millisecond margin in the condition kills false loops from probe rounding. Then ffmpeg lays the speech on top, trims to the length of the line, and encodes to H.264 + AAC.<\/p>\n
And this is where a symptom from the early run surfaced \u2014 the one we nicknamed hypnotic: a four-second B-roll spinning two or three times under one long line, the insert looking like a stuck GIF. The cause was exactly that the video was requested at a fixed short length while the model wrote long speech \u2014 nobody enforced the brief's "three to five seconds."<\/p>\n
The fix took three moves at once: TTS first with length from ffprobe; ceil(speech)<\/code> into Seedance with a ten-second cap; and shortening the speech itself in the brief to one or two sentences. We kept the loop \u2014 but as a safety net for the rare case, not as the normal mode.<\/p>\n
Comparing the two generations of the pipeline turned out instructive:<\/p>\n\n\n\n\n\n\n\n\n
<\/th>\nEarly run<\/th>\nAfter tightening<\/th>\n<\/tr>\n<\/thead>\n
Speech per slot<\/td>\nlong paragraphs (up to ~13s)<\/td>\n1\u20132 short sentences (~5\u20136s)<\/td>\n<\/tr>\n
Video duration<\/td>\nfixed 4s<\/td>\nfrom TTS: 6 \/ 5 \/ 7 \/ 6s<\/td>\n<\/tr>\n
Looping<\/td>\nall 4 clips<\/td>\nnone<\/td>\n<\/tr>\n
Video cost<\/td>\n~$0.18<\/td>\n~$0.28<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
The paradox is right there: tightening made it more expensive. But more honest \u2014 no triple-spin of one clip under a long line. Those ten cents of difference buy back an insert that no longer hypnotizes the viewer.<\/p>\n
Stage 5. Splicing phone footage with a neural net<\/h3>\n
The final reel is assembled from timeline.json<\/code> \u2014 a plain list interleaving baseline clips from the phone with AI-insert numbers. The link between speech and clips rests on order<\/strong>, not on timecodes inside one file: any piece can be reshot without rebuilding the timing.<\/p>\n
The last gotcha waited at the seam between two worlds. iPhone .MOV<\/code> and Seedance .mp4<\/code> are almost always incompatible in codec, resolution, and frame rate \u2014 trying to splice them by stream-copy gives black frames or audio drift. So for the full reel we go straight to re-encode with a single canvas: scale + pad to a common size, 30 fps, yuv420p, AAC. Clips without an audio track get silence padded in, so the sound doesn't jump at the seams. One such reel \u2014 six live shots plus four inserts \u2014 came out around 81 seconds and 113 MB. About right for vertical phone HD.<\/p>\n
Manifests as an accounting ledger<\/h2>\n
Every paid or measurable stage writes a JSON next to its artifacts: speech durations and quota spent, the jobId and cost of each clip, whether the video looped. It sounds like bureaucracy, but it pays off immediately. From these files you can explain the OpenRouter and ElevenLabs bill clip by clip, rebuild a single insert instead of the whole reel, debug "why did the video loop," and stop guessing where the Seedance length came from.<\/p>\n
The media itself (mp3, mp4, the source .MOV<\/code>) sits in .gitignore<\/code>. What stays in the repo is scripts, briefs, timeline, and manifests \u2014 reproducibility without gigabytes in the history.<\/p>\n
The gotchas that didn't make the main plot<\/h2>\n
A couple of stories worth their own paragraph, because each ate an evening.<\/p>\n
A "free" voice that costs 402.<\/strong> We grabbed a library voice in ElevenLabs with an honest-looking "available to free users" tag \u2014 and got paid_plan_required<\/code>. As it turns out, on the free tier the API only serves premade voices; shared\/library ones need a paid plan even when the catalog hints otherwise. free_users_allowed: true<\/code> in the catalog does not mean the right to synthesize with a free key. For a "native" Russian timbre you need a paid plan or a different provider.<\/p>\n
ffprobe-static<\/code> is not ffmpeg-static<\/code>.<\/strong> One npm package exports a string path, the other an object { path }<\/code>. Mix them up and spawn<\/code> falls over with undefined<\/code> instead of a path. The cure is one line everywhere you call a binary: typeof mod === "string" ? mod : mod?.path<\/code>.<\/p>\n
Windows with no ffmpeg and a fake Python.<\/strong> On the test machine the system python<\/code> opened the app store (a WindowsApps<\/code> stub), and ffmpeg<\/code> wasn't in PATH at all. A Node-only pipeline plus static binaries pulled in by npm install<\/code> into the skills closed the question without any "please install ffmpeg for us."<\/p>\n
Where the money is<\/h2>\n
The economics of one run over four inserts, 480p, short lines:<\/p>\n\n\n\n\n\n\n\n\n\n
Stage<\/th>\nOrder of magnitude<\/th>\n<\/tr>\n<\/thead>\n
Slot fill (cheap chat model)<\/td>\n~$0.01<\/td>\n<\/tr>\n
ElevenLabs TTS (free quota)<\/td>\n$0 (or cents on paid)<\/td>\n<\/tr>\n
Seedance \u00d7 4<\/td>\n~$0.20\u20130.35<\/td>\n<\/tr>\n
Mux and concat<\/td>\n$0 (local CPU)<\/td>\n<\/tr>\n
API total<\/strong><\/td>\n~$0.25\u20130.40<\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
The expensive part here isn't the API \u2014 it's the human: filming the live scaffold, the edit review, the legal read of the offer. The pipeline cheapens exactly the insert iterations \u2014 regenerating one CUT is cheaper than reshooting the whole reel. Which is precisely why the ten-second cap per clip and the ban on the model's native audio aren't micro-optimizations but product decisions. The OpenRouter video-model collection, by the way, sorts by popularity, not by price \u2014 it's easy to pick an expensive model by not paying attention.<\/p>\n
What carries over to other builds<\/h2>\n
Three takeaways transfer almost verbatim to any multimodal pipeline, not just reels.<\/p>\n
First: separate the immutable scaffold from the generated slots<\/strong> \u2014 in the data, in the code, and in the model's permissions. Let an LLM rewrite everything and it will rewrite everything.<\/p>\n
Second: the duration of a multimodal clip is measured, not guessed.<\/strong> TTS \u2192 ffprobe \u2192 video length is the only order that doesn't collapse into drift.<\/p>\n
Third: model constraints are part of the product design, not an integration bug.<\/strong> The 4\u201312 second range, free versus library voices, the cost of native audio \u2014 you bake these into the schema up front, not catch them at runtime on an HTTP 400.<\/p>\n","protected":false},"excerpt":{"rendered":"
The expert films the talking-head himself; the cutaways between his points are AI-generated \u2014 faceless video under synthesized speech. The story of how a one-off script grew into a five-stage pipeline, with the gotchas: hypnotic loops, ‘free’ voices that answer 402, and a Seedance that refuses to shoot 3 seconds.<\/p>\n","protected":false},"author":0,"featured_media":79,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[24],"tags":[65,66,67,63,64],"class_list":["post-80","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-engineering","tag-ffmpeg","tag-openrouter","tag-pipeline","tag-text-to-video","tag-tts"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t