← Work · 03 / Pipelines · media

What, Then When

The model decides what each scene shows. Real audio decides when it shows it.

4
components in the pilot library
1
alignment, driving sync and captions both
0
model-authored timestamps
Python · Node.js · LangGraph · Remotion · WhisperX · FFmpegunder development[ year ][ repo ]
A scene spec beside its frame, one screenshot The authored scene block on the left, the frame it produced on the right — narration span, component, labels, and the cue that fired. Two panes, no arrows, no callouts. The correspondence should be obvious without explanation.
Authored once as content. Everything visual is derived.
01

The problem

Ask a model to write a video and it will hand you timestamps. It has to — a scene needs a start and an end, so it produces some. Those numbers are fiction. The narration doesn't exist yet, so nothing in the model's context could tell it when the voice will actually reach the word the animation is supposed to land on.

You find out at the end, after the render. The arrow draws two seconds before the sentence about the arrow. Nothing crashed, nothing logged an error — the video is just subtly wrong in the specific way that makes an explainer feel amateur.

The compounding version is worse. Insert any video-only time into a continuous narration track — a pad, a transition, a held frame — and picture and sound separate a little more at every scene. By minute four it's visible. By then it's baked into a render that took an hour.

02

What changed

BeforeAfter
TimingModel emits timestampsDerived from word-level alignment of real audio
CaptionsA second generation stepThe same alignment output that drives timing
VisualsModel writes render codeModel picks from an enum; a validator enforces it
CuesAnchored to a word the model countsAnchored to a phrase, resolved in code
Motion too longTruncate it, or pad the audioSpeed up within a cap, flagged; halt past it
PreviewAfter the full renderOne still per scene, before anything slow

The structural decision under all of it: the narration track is the master timeline, and video-only time is never inserted into it. Total video length equals total audio length. A scene that needs to hold longer holds by moving its cut later, not by stretching the timeline. Drift stops being something to manage carefully and becomes something that cannot occur.

03

How it works

The composer reads the script and returns a content spec: per scene, which visual component, what data fills it, the span of narration it covers, and any cues. It never returns a time. A validator checks that spec the moment it lands — the component must exist in the library, its payload must match what that component declares, and every cue must point at an element actually present in the scene. A violation halts the run before a second of voice or render time is spent.

Then the audio gets made, as one continuous track. Forced alignment produces word-level timestamps, and a pure function merges those into the approved spec to produce absolute frames. That same alignment output writes the caption file — one measurement, two consumers, no second subsystem to drift out of sync with the first.

04

Architecture

  1. Compose. One model call. What each scene shows, and which phrase each cue anchors to. No times.
  2. Validate. Code, hard gate. Component in the enum, payload shape correct, cues resolve. Fails before anything expensive runs.
  3. Preview. One still per scene, approved by a human — before rendering, because rendering is the cost.
  4. Voice. Continuous narration, one track, phonetic overrides for technical terms.
  5. Bind. Pure function. Word timestamps merged into the spec, cues resolved to frames, motion checked against each component's minimum duration.
  6. Render. Against the compiled spec. Loudness normalised, intro and outro attached, captions shipped as a sidecar rather than burned in.
Architecture diagram A straight chain with the audio track drawn underneath as a fixed ruler the visual track pins to. Mark the preview gate — everything expensive lives to the right of it, and the drawing should make that obvious.
The audio is the ruler. Everything else is measured against it.
05

The decision

A model asked for a timestamp will always return something timestamp-shaped.

Asking the model for times directly is the obvious build, and it nearly works. The model has just read the script, it knows roughly how long a sentence takes to say, and it will return clean well-formed numbers for every scene. You get a complete timeline in one call, previewable end to end, with no second pipeline stage.

Refusing to accept those numbers costs the preview. Because timing can't exist until the voice does, a human approves content — stills, labels, which component — without seeing motion. Timing problems surface later, after audio generation, and when a scene's animation won't fit its narration the run either compresses within a fixed cap or halts and sends the operator back to rewrite. That's a rework loop the timestamp version never has.

It buys a failure that is loud instead of silent. An animation that can't fit its line stops the pipeline and says so. The alternative is a number that looked right, rendered fine, and put the arrow two seconds early in a finished video nobody flagged until it was published.

06

What's next

The pilot's measured numbers — wall-clock render time on the actual hardware, and whether alignment holds within tolerance on a sentence deliberately packed with technical terms. Both are go/no-go, and neither is worth claiming before it's measured. After that: the component library grows, because with four bricks the videos will visually rhyme no matter how the composer is prompted. [ Confirm before publishing ] A second render lane for mathematical visuals, and hero animations graduating into reusable components once one has been built twice.