Inside our resumable, review-gated video production pipeline

Seven steps and one command

The pipeline runs seven steps in a fixed order: script, narration, visuals, captions, render, description and thumbnail. Each one is its own small Python script that does one job and hands its results to the next. You can run any of them on their own when you want to poke at something, but most days we type one command and let the whole line roll. We like this shape because it keeps problems small. When the thumbnail looks odd, we know exactly which script to open, and we do not have to scroll through a thousand lines of everything else to find it. It also means each step can be improved, tested or swapped without the rest of the line noticing that anything changed at all.

The first step writes the script. An AI model drafts it, and it grounds the draft in web search so the history comes from somewhere real rather than from a confident guess. The script is split into segments, and each segment later gets its own narration, pictures and captions, so the numbering matters a great deal. We could let the model number the segments itself, and it would probably be fine most of the time, which is exactly the kind of fine that ruins an evening. So the pipeline assigns the segment numbers on its own after the text comes back. The model is great at writing sentences about history. Counting to twelve reliably, it turns out, is a job we would rather keep for ourselves.

Teaching a voice to keep time

Narration comes from ElevenLabs, using its text to speech option that returns timestamps along with the audio. That detail sounds dull until you need captions that land on the right word at the right moment. The timestamps arrive per character, which is wonderfully precise and also slightly absurd for our purposes, since nobody reads captions one letter at a time. So the pipeline walks through those characters and stitches them back into words, giving each word a start time and an end time. From there, everything downstream knows exactly when each word is spoken. The captions use it, the render uses it to size each clip, and nobody has to sit with a stopwatch nudging subtitles by hand at eleven at night.

The run that died at step six

Picture this. It is nearly midnight, the run has been going for a while, and it dies at step six. Maybe the laptop went to sleep, maybe a network call gave up, maybe someone closed the wrong terminal window. Without a memory, that would mean starting again from the script and making all that narration a second time. Now, every time a step finishes cleanly, the pipeline writes it down in a small state file. Run the command again and it reads that file, skips everything already done, and picks up right where it stopped. Step six gets another go, and steps one through five stay politely untouched. The whole recovery takes about as long as a tired sigh.

The same memory works as a guard rail. If you try to start the pipeline at a later step, say jumping straight to the render, it checks the state file first. When the earlier steps are not recorded as finished, it refuses to run and tells you why. That can feel a little stubborn in the moment, especially when you are sure the files are all there. But the alternative is worse. A render that quietly uses half finished captions or last week's narration produces a video that looks fine at a glance and wrong on a second watch. We would rather be told no at the start than discover the mistake after the whole thing has been assembled and is waiting in the queue.

When a photo says one thing and shows another

For pictures, the pipeline searches two wonderful public archives, the Library of Congress and Wikimedia Commons, using keywords pulled from each segment of the script. Searches like that return plenty of candidates, some brilliant and some baffling. An AI model looks over the results for each keyword and picks the best one or two. That would be the end of it, except archives are written by people, and people label things in all sorts of ways. A photo's title might name a famous general while the image itself shows the general's horse, a crowd in a field, or the building where the photo was later stored. The credit line might describe the collection rather than the subject. The words and the picture do not always agree.

So there is a second check, and it does not read the label at all. A vision model looks at the actual pixels of each chosen image and asks a simple question: does this picture show what the segment is talking about. If the script is describing a steam locomotive and the photo is a portrait of a man in a very fine hat, the photo is out, no matter how confidently its title says locomotive. This step catches the mix ups that text alone never would. It also saves us from the particular embarrassment of narrating a dramatic moment over a picture of an unrelated train station in winter, which is a sentence we are glad we only have to write hypothetically.

Captions, slow zooms and a patient queue

Captions are grouped into chunks of about five words, and the grouping pays attention to sentences, so a caption does not end on a dangling word that belongs with the next line. Once the groups are set, ffmpeg burns them straight into the video frames. The render step then builds the pictures into slow pan and zoom clips, the gentle drift across a still photo often called the Ken Burns effect. Each clip is cut to the exact length of its narration, using those word timings from earlier, so the image changes when the voice moves on and not a beat before. It is a small thing, but a still photo that lingers after its sentence has ended feels oddly like someone forgot to turn the page.

The last two steps write a description and make a thumbnail, and then the pipeline does something we are quite proud of. It stops. Nothing is uploaded automatically, ever. The finished video, with its description and thumbnail, goes into a queue marked as pending review, and it sits there until a person watches it and decides what happens next. That review is not a formality. It is the point where a human catches the odd pronunciation, the caption that splits a name in half, or the photo that passed every check and still feels wrong. Automation gets the video most of the way there. A person decides whether it is actually ready to be seen by anyone else.

The hardest decision was how to fail

The decision we went back and forth on the most was not about pictures or voices. It was about what to do when something fails. An early version of the pipeline had a habit we did not notice at first. When the model review of the images failed, it quietly swapped in a set of simple rules instead and carried on as if nothing had happened. Later, a wrapper around the vision check did something similar, skipping the check entirely whenever a call failed, and saying nothing about it. The videos still came out. They just came out having skipped the very steps that were meant to protect them, and there was no sign in the logs that anything had gone missing.

So we removed the silent fallbacks. Now, when a model call fails or comes back empty, the pipeline waits and tries again, first after 10 seconds, then 30, then 60, and it keeps trying until it gets a real answer. During the long waits it prints a heartbeat line to the log, so a quiet stretch looks like patience rather than a frozen screen. Network calls that hit a rate limit, the familiar 429 response, are treated differently. They wait at most three times and then fail only that one keyword, leaving the rest of the run alone. Slow and honest beats fast and wrong, especially when the wrong version would have looked perfectly fine.