Scene Studio — architecture (design doc)

Status: design / not built. This is the plan for the first step toward a "movie creation studio." It extends the Rehearsal Room Promo Studio (INDEX → Promo Studio, services/rehearsal-promo.ts) from a 26-second promo to an N-shot dramatized scene pointed at a Graphene season chapter. See the north-star framing in docs/marketing/promo-videos/ + the memory project_movie_studio_vision.

Framing: movie → script → story (we own the bottom two)

A film decomposes into a script that dramatizes a story. Graphene already produces story + script + audio; the Promo Studio already does brief → shot list → stills → assemble. Scene Studio generalizes that loop and points it at a chapter, so the unit of output is a watchable scene (then a trailer, then a short).

Honest constraints (these shape the design)

• Clips are ~5–10s. A "movie" is not the near-term unit. The deliverables are: dramatized scene → trailer → short film. Design around short clips stitched together, not long generations.
• Consistency across shots is THE hard problem (character face, wardrobe, style). Mitigations are first-class in this design (below), not afterthoughts.
• Cost + runtime scale fast. Everything is admin-triggered, per-shot, and budgeted — mirror the Promo Studio's gating.
• Art-first (Odessa). The point is that the footage comes from stories people live in + rehearse — not generic AI video. Keep the experience story-driven; the renderer is downstream.

Pipeline (extends the Promo Studio loop)

Chapter (prose + bible + cast voices)
  → 1. Shot breakdown        (LLM → structured scene script: shots, dialogue, who speaks)
  → 2. Keyframe stills        (Flux per shot; image-to-video start frames → consistency)
  → 3. Per-shot video clips   (Replicate image-to-video, ~5–10s each)
  → 4. Dialogue/narration audio (reuse season-audio: per-character voices + narrator)
  → 5. Assemble               (ffmpeg: clips in order, mux audio, optional music bed)
  → Scene MP4 (re-hosted in storage)

1. Shot breakdown (script → shots)

An LLM call (same shape as generatePromoBrief) takes the chapter prose + the season's story bible (character descriptions, the look) and emits a structured scene script: ordered shots, each with { visual_prompt, dialogue?, speaker_persona_id?, duration_s, shot_type }. The bible is what keeps character/world descriptions consistent across shots.

2. Keyframe stills (consistency anchor)

Generate one keyframe still per shot with Flux (reuse renderImage). Crucially, these become the start frames for image-to-video — the single biggest consistency lever we have. Per-character model-sheet stills (a locked reference image per persona, generated once and reused) + a per-project style anchor string pin the look. Lean stylized/graphic-novel where helpful (drift reads as aesthetic, not error — and we already have graphic-novel mode).

3. Per-shot video (Replicate, image-to-video)

Reuse the new Replicate({auth}).run(...) pattern (already proven in season-music.ts) — no new dependency/secret. Candidate image-to-video models (verify exact slugs at build time; this space moves fast):

• Kling (kwaivgi/kling-*) — strong motion + image-to-video, good value.
• Minimax / Hailuo (minimax/video-01) — good adherence + start-frame.
• Google Veo (if available on Replicate) — quality + native audio (could reduce our audio-mux work for non-dialogue shots). Prefer image-to-video from the Flux keyframe over text-to-video — it's the consistency anchor. Download + re-host each clip in storage (don't hotlink expiring URLs), exactly like the music/stills paths.

4. Audio (reuse what exists)

Dialogue + narration come from the existing audio stack: each shot's dialogue is rendered with the speaker persona's assigned voice (assignVoicesToCast / VOICE_BANK) via ttsSegment; narration uses the narrator voice. We already render and store per-chapter audio — Scene Studio should reuse a chapter's audio takes where they exist rather than re-rendering. Optional music bed via the existing season-music themes.

5. Assembly (ffmpeg)

ffmpeg is already used server-side (concatMp3Buffers, the season media pipeline). Assemble the scene: order clips (trim/pad to the dialogue length), concat (concat demuxer), mux the dialogue/narration track + music bed, export MP4 9:16 (or 16:9 for a "cinema" cut). Re-host in storage.

Data model (proposed)

Mirror the promo tables. One project per chapter (or per scene):

scene_projects (
  id uuid pk,
  season_id uuid,             -- the Graphene season
  episode_id uuid,            -- the chapter
  style_anchor text,          -- locked look string for the whole scene
  script jsonb,               -- the shot breakdown (ordered shots + dialogue)
  status text,                -- draft → shots → clips → final → published
  video_url text,             -- assembled scene
  created_at, updated_at
)
scene_shots (
  id uuid pk,
  scene_project_id uuid,
  idx int,                    -- order
  visual_prompt text,
  speaker_persona_id uuid null,
  dialogue text null,
  duration_s numeric,
  keyframe_url text null,     -- Flux still (start frame)
  clip_url text null,         -- generated video clip
  audio_url text null,        -- rendered dialogue/narration for this shot
  status text
)
persona_model_sheets (        -- consistency: one locked reference per character
  persona_id uuid pk,
  reference_image_url text,
  created_at
)

UI surface

A Scene Studio module — either its own admin page (like Promo Studio) or a panel on the season page (/seasons/[id]). Per chapter: Generate shot breakdown → review/edit shots → generate keyframes → generate clips (per shot, with retry) → assemble → preview + publish. Per-shot regeneration is essential (you'll re-roll individual shots constantly).

Phasing

1. Scene Studio (one chapter → one scene). Proves the whole chain end to end; reuses ~80% of the Promo Studio + audio stack. Start here.
2. Trailer mode. Stitch the best shots across a season into a 30–60s trailer — the fastest distribution win; feeds straight back into the Promo Studio + marketing kit.
3. Episode / short film. Chain scenes for a full chapter; push the consistency tooling (model sheets, style locks) hard.
4. Story → Script generator. From a note/premise, generate the season AND its shootable script — closing journal-entry → story → script → screen.

Risks / open questions

• Consistency ceiling: how good is image-to-video from a fixed keyframe in practice? Validate on one scene before committing to phase 2+.
• Per-shot cost at scale (a 12-shot scene = 12 video gens + stills + audio); needs a budget guard + a clear per-scene cost readout (extend the existing TTS spend pattern to video).
• Model churn: pin model versions; expect to swap engines as the field moves (the renderImage engine-selector pattern generalizes to video).
• Runtime/latency: scene generation is minutes, not seconds — needs async job handling + progress UI (heartbeat pattern from the crons doc).

Brand alignment

Ships under EmberKiln (graphene.fm), not as a generic AI-video tool. The wedge is that the footage is born from stories people actually inhabit and rehearse — the fiction→real-change loop (Odessa), now with a screen at the end of it. Keep the story/script experience primary; the renderer serves it.