How Image-to-Video Generation Works: From a Still Image to a Usable Moving Clip

A single still image enters the model, and for four seconds the model predicts what happens next by guessing which pixels move and which stay still. That guess is the entire technical reality of image-to-video generation. The model does not know the scene has a floor, a ceiling, or a person with two arms. It knows only that in its training data, when a certain arrangement of pixels was followed by a certain arrangement of motion vectors, the output looked plausible. This page explains how image-to-video generation works, what the models actually do inside their latent diffusion and temporal attention layers, and how you get from a source image to an MP4 container holding H.264-encoded frames that you can use in a project.

• Architecture: Latent diffusion with temporal attention layers; no persistent 3D scene model
• Input: Single still RGB image, PNG or JPEG; resolution varies by model
• Output: MP4 container, H.264 or H.265 codec; 24 or 30 fps; no native audio
• Native Clip Length: 2-10 seconds per generation; usable motion coherence typically 4-6 seconds
• Native Resolution: 576×1024 to 1080×1920; effective detail often one tier below advertised
• Key Control Types: Camera presets (pan, tilt, zoom, roll, dolly, crane, orbit); motion strength scalars; region masks
• Licence Categories: Proprietary (web/API); open-weights (Apache 2.0 or non-commercial); no fully open-source models
• Post-Processing Standard: FFmpeg for frame extraction, concatenation, format conversion, metadata handling

Every model listed here, Runway Gen-3 Alpha, Kling 1.6, Luma Dream Machine, Pika 2.0, OpenAI Sora, Google Veo 2, Hailuo Minimax, Wan 2.1, Stable Video Diffusion, LTX-Video, and Seedance 1.0, shares a core architectural family. The source image is first encoded into a compressed latent space using a VAE. Noise is added to that latent representation, and a denoising network then predicts the noise that was added. The critical addition for video is the temporal attention mechanism: the model attends not only to spatial relationships within a single frame but to the same spatial location across multiple frames. This is what produces the illusion of motion rather than a sequence of unrelated stills.

Temporal attention layers operate over a window of frames, 4 to 25 frames depending on the model and the clip length. Each frame's latent representation queries the representations of adjacent frames, learning which patches are likely to persist and which are likely to shift. The result is a motion prediction that is, in the strict sense, a pixel-displacement estimate rather than a scene understanding. The model does not infer that a chair is a solid object; it infers that the chair's pixels in frame 1 correlate with a patch in its training data where similar pixels moved in a particular way.

What This Means for the Output

Because the model lacks a persistent 3D world model, every failure mode in the list below follows directly from the architecture. Subject drift occurs when the temporal attention loses the conditioning signal from the source image. Limb mutation happens because the model has no skeletal model, it has no concept of joints or bone structure. Background sliding arises because the model cannot infer a consistent 3D scene from a single 2D image. These are not bugs that will be fixed in a future version; they are structural consequences of the approach.

Before any of the advice here applies, the source image must meet conditions that are independent of the model you choose. If the still fails these, the output will fail regardless of the model's capabilities.

The image must be in an aspect ratio the model supports. Training data for video models is dominated by 16:9, 9:16, and 1:1 ratios. If you supply a 4:3 or 21:9 image, most services centre-crop it to the nearest supported ratio, which may cut off the subject. The safest choice is 16:9 at the model's native resolution. The image resolution must sit between the model's minimum and maximum. Below the minimum, the service upscales, introducing softening. Above the maximum, it downscales, discarding detail. Neither outcome helps motion coherence.

High-frequency detail in the source image, text, fine patterns, intricate textures, degrades during generation across all models. The latent-space compression discards the high-frequency information first, and the temporal attention layers propagate the degraded version. If your image contains a sign with readable text, expect the text to become unreadable or to morph into different characters within the first two seconds of the clip. Faces in the source image should occupy at least 10 to 15 percent of the frame height. Below that threshold, facial features become unstable and identity shift becomes likely.

Aspect-Ratio Constraint and Pre-Processing

The pre-processing step in the image-to-video pipeline is where most aspect-ratio distortion originates. The source image is resized and centre-cropped to match the model's training aspect ratios. If your subject is positioned near the edge of the frame, centre-cropping will displace it. If the subject fills the frame in the source image, the crop may truncate limbs. The only reliable mitigation is to compose the source image with generous padding around the subject, anticipating a crop of up to 15 percent on each side.

Model	Max Clip Length	Max Native Resolution	Camera Control Axes	Start/End Frame	Region Mask	Open Weights	Licence	Release Date
Runway Gen-3 Alpha	10 s	1080×1920	6 (pan, tilt, zoom, roll, crane, dolly)	Yes	Yes (5 regions)	No	Proprietary	2024-06-17
Kling 1.6	10 s (standard), 5 s (pro)	1080p	4 (pan, tilt, zoom, roll)	Yes	Yes	No	Proprietary	2024-12-19
Luma Dream Machine	5 s	1080×1920	7 (pan, tilt, zoom, roll, dolly, crane, orbit)	Yes (2 frames only)	No	No	Proprietary	2024-06-12
Pika 2.0	8 s	1080p	4 (pan, tilt, zoom, roll)	Yes	Yes	No	Proprietary	2024-12-16
OpenAI Sora	20 s	1080p	Storyboard controls	Yes	No	No	Proprietary	2024-12-09
Google Veo 2	8 s	1080×1920	Prompt-based	Yes	No	No	Proprietary	2024-12-16
Hailuo Minimax	6 s	1080p	Not specified	Yes	No	No	Proprietary	2024-09-02
Wan 2.1	5 s (480p), 3 s (720p)	720p	Motion magnitude parameter	Yes	No	Yes (1.3B, 14B)	Apache 2.0	2025-02-25
Stable Video Diffusion	4 s (SVD), 6 s (SVD-XT)	576×1024	Motion bucket ID (1–255)	No	No	Yes	Non-commercial / limited commercial	2023-11-21
LTX-Video	6 s	768×512	Not specified	No (text-to-video base)	No	Yes	Apache 2.0	2024-11-21
Seedance 1.0	10 s	1080p	Camera presets (LoRAs)	Yes	No	No	Proprietary	2025-01-22

The image-to-video pipeline steps are consistent across every tool listed. Understanding the sequence lets you predict where problems will appear and which parameter changes will actually affect the output.

Step one is pre-processing. The source image is resized and centre-cropped to the model's supported aspect ratio. The service applies any user-provided motion brush masks or region masks at this stage, converting them into latent-space coordinates. Step two is VAE encode: the pre-processed image is compressed into a latent representation. Step three is the core inference: the model iteratively denoises the latent across the target number of frames, with temporal attention layers enforcing frame-to-frame consistency. Step four is VAE decode, which expands the latent back into pixel space. Step five is post-processing, which is where you intervene with FFmpeg.

Reading the Specification Table

The table above lists claimed maximums and documented limits from each model's documentation. The column order prioritises the attributes that determine whether a clip is usable for a particular purpose. Clip length comes first because the structural trade-off between motion magnitude and source fidelity means that longer clips degrade more. Resolution comes second because the claimed figure and the effective detail are not the same: independent frame analysis shows that models advertising 1080p output often produce effective detail closer to 720p upscaled. Camera control axes come third because this is the attribute that distinguishes models for professional use. Start-and-end-frame support and region-mask support are binary features that enable particular workflows. The open-weights column tells you whether the model can run locally, and the licence column tells you what you are allowed to do with the output.

Claimed Versus Real Figures

Models advertise 10-second generation, but motion coherence degrades visibly after 4 to 6 seconds across all models. The gap is 40 to 60 percent of the claimed duration. The cause is the temporal attention mechanism losing the source-image conditioning signal over long sequences. The same pattern applies to resolution: a model may output 1080p files, but the effective detail is closer to 720p upscaled, a gap of roughly one resolution tier. The cause is the latent-space compression used in the generation pipeline. For photorealism, marketing materials show cherry-picked best-of-N outputs; the median generation from the same model on an uncurated source image shows visible AI artefacts. The gap is the difference between best-of-100 and median-of-1.

Every failure mode listed below is a documented behaviour of current image-to-video models. Each one has a particular architectural cause. Knowing the cause lets you decide whether the failure is worth working around or whether it tells you the model is wrong for your source image.

Subject Drift

Subject drift is the gradual transformation of the person or object in the source image into something else over the clip duration. The cause is the model's temporal attention losing the conditioning signal from the source image as the frame count increases. The model begins to interpolate toward the distribution of subjects in its training data rather than anchoring to your particular image. Subject drift is most visible in the last two seconds of a clip and becomes severe beyond six seconds regardless of the model.

Limb Mutation

Limb mutation describes arms, legs, or fingers that warp, multiply, or detach during motion. The cause is the model's lack of a skeletal or physical model of the person. The temporal attention layers have no concept of joints or limb connectivity; they predict pixel motion based purely on statistical correlation. When the training data contains a wide variety of limb configurations, the model samples from that distribution rather than maintaining the particular limb arrangement in the source image.

Background Sliding

Background sliding occurs when the background moves independently of the foreground subject, as if on a separate layer. The cause is the model failing to infer a consistent 3D scene structure from the single image. In the latent space, foreground and background patches are processed with the same temporal attention, but without a depth map or parallax model, the network has no way to enforce their relative motion. The result is a parallax effect that is physically wrong.

Identity Shift

Identity shift is a particular form of subject drift where a face changes ethnicity, age, or gender during the clip. The cause is the model interpolating toward the distribution of faces in its training data rather than anchoring to the source image. This is most common when the source image contains a face that is statistically rare in the training data: a non-frontal angle, an unusual expression, or a face with features underrepresented in the training set. If you are animating a real person, get explicit written consent before generation. The model can alter identity in ways the person may not have anticipated or agreed to.

Flicker

Flicker is rapid frame-to-frame changes in brightness, colour, or texture that are not part of the intended motion. The cause is inconsistent latent-space decoding across frames. The VAE decode step is not deterministic across a sequence of frames; minor variations in the latent representation at each frame produce visible luminance shifts. Flicker is most visible in flat, uniform areas such as skies or walls.

Seam Artefact

A seam artefact is a visible jump or glitch where two generated clips are concatenated. The cause is the model not supporting true start-and-end-frame conditioning for the join. Some models support specifying both the first and last frame of the generated clip, but the intermediate frames are interpolated from those two references alone. The seam artefact appears when the generated clip does not match the following clip's first frame.

Motion Collapse

Motion collapse is the generation of a nearly static clip despite a prompt requesting significant movement. The cause is the model's bias toward low-motion outputs, which have fewer visible artefacts during training. The model learns that static or near-static outputs are penalised less during training than high-motion outputs that produce subject drift or limb mutation. Motion collapse is most common on models with a single motion-strength scalar rather than multi-axis camera controls.

Camera-motion control types differentiate models for professional use. The number of distinct controllable axes determines whether you can achieve a particular shot type. Runway Gen-3 Alpha, as of its 2024-06 release, offers six axes: pan, tilt, zoom, roll, crane, and dolly. Luma Dream Machine, as of its 2024-06 release, offers seven axes: the same six plus orbit. Pika 2.0, as of its 2024-12 release, offers four axes: pan, tilt, zoom, and roll. Kling 1.6, as of its 2024-12 release, offers four axes matching Pika's set. Seedance 1.0, as of its 2025-01 release, offers camera presets via LoRAs, which is a distinct approach: instead of a slider, you select a pre-trained motion style.

The distinction between motion strength and camera motion is critical. Motion strength animates subjects within the frame: a person waves, a flag flaps, water ripples. Camera motion moves the virtual camera through the scene: the camera pans left, tilts up, zooms in. A model with only a motion-strength scalar cannot produce a dolly shot. A model with camera presets but no motion-strength scalar cannot animate the subject independently of the camera. Each model's documentation specifies which type it supports, and the table above lists the number of controllable axes for each model.

The source image is half the outcome. A photograph of a static subject against a simple background, with good lighting and the subject centred, produces a usable clip. A photograph of a complex scene with multiple overlapping subjects, fine text, or a non-standard aspect ratio produces a clip with visible artefacts regardless of the model. This is not a limitation that will disappear with model improvements; it is a structural consequence of the single-reference-frame limitation. All current image-to-video models generate motion from one still image. Multi-image conditioning is an emerging feature, Kling 1.6 and Luma Dream Machine support start-and-end-frame pairs, but the single-image case remains the baseline problem.

The Durable Pipeline Structure

The image-to-video pipeline structure is source image, pre-processing, model inference, and post-processing. Post-processing uses FFmpeg for frame extraction, concatenation, format conversion, and metadata stripping. No consumer tool eliminates this step. If you need to concatenate multiple generated clips into a longer sequence, handle the seam artefacts yourself using FFmpeg's crossfade or frame-interpolation filters. If you need to remove the model's metadata watermark, use FFmpeg or exiftool to strip the metadata fields. The model outputs an MP4 container with H.264 or H.265 encoding at 24 or 30 fps, and that is the format you work with.

The Trade-Off Between Motion Magnitude and Source Fidelity

Stronger motion always increases the risk of subject drift and artefacting. This is a structural consequence of predicting pixel displacement from a single reference frame. Every model listed exhibits this trade-off. If your output requires the subject to remain recognisable, keep motion strength low. If your output requires dramatic camera movement, accept that the subject will drift and plan to mask, re-generate, or accept the artefact. The models with multi-axis camera controls, Runway Gen-3 Alpha and Luma Dream Machine, allow you to distribute motion across axes, which spreads the displacement and reduces the peak drift per axis.

The model outputs a clip that is almost never ready for final use. Post-processing with FFmpeg is the step that turns a generated clip into an asset you can edit into a project. The three most common operations are frame extraction, concatenation, and format conversion. Use frame extraction when you need individual frames for manual selection or as training data for another model. Use concatenation when you generate multiple short clips and need a longer sequence. Use format conversion when the delivery platform requires a particular codec or container.

Frame Extraction

To extract every frame from a generated clip as PNG files, run ffmpeg -i input.mp4 -vf fps=24 output_%04d.png. Replace the frame rate value with the clip's actual frame rate. The output files are numbered sequentially. Inspect them frame by frame to identify flicker, seam artefacts, or subject drift at particular points.

Concatenation

To concatenate two clips without a visible seam, use the concat demuxer with a text file listing the input files. The command is ffmpeg -f concat -i filelist.txt -c copy output.mp4. This method works only if the clips have identical codec, resolution, and frame rate. If they do not, re-encode them to a common format first, which introduces generation loss.

Format Conversion

Most generated clips are already in an MP4 container with H.264 encoding, but some platforms require H.265 for smaller file sizes or particular metadata fields. Convert with ffmpeg -i input.mp4 -c:v libx265 -crf 23 output.mp4. The CRF value controls quality; lower values produce higher quality at larger file sizes. The 2024-03 YouTube policy requiring disclosure of altered or synthetic content does not mandate a particular encoding format, but it does require the uploader to declare the content's synthetic origin.

If you have not run an image-to-video model before, start with a source image that is a single subject against a plain background, at 16:9 aspect ratio, at the model's native resolution. Use the model with the lowest motion strength setting and a simple camera pan. Run that single generation. Inspect every frame. Note which failure modes appear. The first-generation output is your baseline. Every adjustment you make, changing the source image, adjusting the motion strength, switching models, is a measurable change against that baseline. No summary or recommendation replaces that one test.