AI Folks
  • Courses
  • Web Of AI
  • Pricing
  • Enterprise
Log in
PreviousMotion with SeedanceNextConsistent characters

Vibe Coder · Unit 21: AI Video Production · Lesson 7 of 14

Open video models in ComfyUI

Run an open video model yourself. A video workflow in ComfyUI is the image workflow with frames added: load the model, give it a start image and a prompt, sample, and save a clip.

Read the short notes and pictures.Do everything inside a blue frame.

What you already know

In the image unit you learned to read a ComfyUI graph: load a model, encode a prompt, sample in the latent, decode, save. A video workflow has the same shape. The model is a video model, the latent holds many frames instead of one, and the last node writes a clip.

A dark circuit, hand-drawn. Six nodes: load video model, Load Image for the start frame, prompt for the motion, sampler, decode, save video.
The image graph, with a video model and a clip at the end.

Start from a template

  1. 1

    Open the workflow templates

    And find the video section. There are templates for current open video models.

  2. 2

    Pick image to video

    So you can use the stills you already approved.

  3. 3

    Download what it asks for

    Video models, like Flux, come as several files that go in the folders the template names.

  4. 4

    Load your start image

    In the Load Image node.

  5. 5

    Write a motion prompt

    What moves and how the camera moves. The still already shows what is in the frame.

Which open model

Makers publish their own requirements, and they differ widely. As examples from published material: Lightricks’ LTX models have first-party ComfyUI templates and can generate audio with the picture; Alibaba’s Wan family is released under Apache 2.0 and includes a small model meant for modest graphics cards alongside much larger ones. Read the model’s page for its current version, memory needs and licence.

Frames

A dark circuit, hand-drawn. A table: number of frames decides the length, frames per second decides smoothness, frames divided by frames per second gives seconds.
Frames divided by frames per second is the length in seconds.

Two settings are new. The number of frames sets how long the clip is. Frames per second sets how smoothly it plays. Divide one by the other for the length: 48 frames at 24 per second is a two-second clip. Each model has a frame count it was trained around; the template starts you at a sensible value.

A motion prompt that fights the still

A modern office with a laptop and a
plant and a mug and morning light and
a person walking in

A motion prompt that works

Slow push-in toward the laptop screen.
Steam rises gently from the mug.
Soft morning light, no other movement.
Describe motion and camera only. Keep it to one or two movements per clip.

Drafts and finals

A video generation does many times the work of an image. On your own hardware a clip can take minutes. So work in two passes, as you would with a film camera.

  1. 1

    Draft small

    Low resolution, few frames, a few seeds. You are judging the motion.

  2. 2

    Choose

    Keep the seed and prompt of the take that moves correctly.

  3. 3

    Render the final

    Same seed and prompt, full size and length. Only for takes you will use.

  4. 4

    Save the workflow

    With the clip. The settings are how you remake it.

Flicker or warping

Simplify the motion prompt; reduce how much moves at once.

The subject changes

Use a clearer start image, or a model that accepts a reference.

Out of memory

Fewer frames, lower resolution, a smaller or quantized model, or Comfy Cloud.

Nothing moves

Say what moves, and name the camera movement.

No suitable graphics card

Comfy Cloud runs the same workflows in the browser on cloud GPUs, and its templates open there directly. Hosted video models remain the practical route for many people; this lesson is so that you understand what they are doing.

Licence and labelling still apply

An open model’s licence governs what you may do with the clips. Running it on your own machine does not change that. Label AI-made video where required.

ActivityYour turnWrite it yourself first, then compare.

Your draft clip is 16 frames at 16 frames per second and you need a 4-second shot. What do you change, and what might go wrong?

ActivityBuild stepA launch video

Animate one shot with an open model

In ComfyUI or Comfy Cloud, open an image-to-video template for an open model. Animate one approved still from your storyboard: draft small with a few seeds, pick the best, and render one final clip. Then publish your video files.

Done when

Compare this clip with the hosted version of the same shot, and use the better one in your edit.

Keep these

  • A video workflow is the image graph with frames added.
  • Frames divided by frames per second gives the length.
  • Draft small, then render only the takes you will use.
Previous lessonMotion with SeedanceNext lessonConsistent characters