Understanding Image-to-Video Quality Benchmarks Including VBench and Human Preference Evaluations
Image-to-video benchmarks like VBench measure motion smoothness and aesthetic quality but miss identity drift, limb mutation, and source-image fidelity at clip end.
Understanding Image-to-Video Quality Benchmarks Including VBench and Human Preference Evaluations
The most common mistake newcomers make about image-to-video quality evaluation is assuming a high score means the clip will look physically coherent. It does not. Benchmarks like VBench and human preference tests measure specific proxy signals: motion smoothness, frame-to-frame consistency, aesthetic quality. They routinely miss the failures that break a clip for real use. Identity shift. Limb mutation. Background sliding. A model can score well on VBench and still morph your subject's face into someone else by the fifth second. Understand what each test actually tracks, and what it systematically ignores, before you pick a model for a job.
VBench Image-to-Video Evaluation Metrics
VBench is the most widely cited evaluation suite for image-to-video generation. Published by a team from Nanyang Technological University and released alongside a labeled evaluation dataset, VBench defines a series of per-frame and per-clip metrics. The official paper lists 16 dimensions including motion smoothness, temporal flickering, object class consistency, and aesthetic quality. Each dimension is scored by comparing the generated clip against a reference or against learned classifiers trained on real video.
What VBench Measures
Motion smoothness checks whether the optical flow between consecutive frames is continuous. Temporal flickering captures rapid brightness or color shifts that do not correspond to intended motion. Object class consistency verifies whether an object detected in frame 1 is still detected as the same class in frame 10. Aesthetic quality uses a learned predictor trained on curated photo datasets.
What VBench Ignores
What VBench does not measure matters more. It does not measure identity preservation across a clip: whether a face stays recognisably the same person. It does not measure limb coherence: whether arms and legs maintain anatomical structure during motion. It does not measure source-image fidelity at clip end: whether the subject still looks like the original input after the full duration. These are the failure modes that matter most to a filmmaker or content creator, and VBench is silent on them.
The Dataset Problem
The evaluation dataset VBench uses is a curated set of prompts and reference videos. The provenance of those reference videos is disclosed in the paper: they are drawn from publicly available stock footage and synthetic sources. The dataset is static. That means it can be gamed. A model tuned to perform well on VBench's specific test cases may still fail badly on an uncurated source image from a user's phone.
Human Preference Test Image-to-Video Evaluation
Human preference tests address some of what VBench misses by asking raters to compare two clips head-to-head. The protocol: present a rater with two generated clips from the same source image and prompt, and ask which looks better. Aggregate thousands of such pairwise comparisons and you get a preference score. This method captures holistic quality. A rater can penalise a clip that has smooth motion but a distorted face, even if VBench gave it high marks.
The gap between claimed and real performance is where the human preference test exposes the most trouble. Marketing materials and model demos show cherry-picked output, selected as the best from 50 or 100 attempts. This is the best-of-N problem. In a human preference test, the median-of-1 output, the single attempt an ordinary user would get on their first try, scores noticeably lower. The dossier's claimed-vs-real analysis documents that the gap between best-of-100 and median-of-1 can be one to two full points on a five-point quality scale. The median clip from an uncurated source image shows visible AI artefacts: flicker, subject drift, background sliding. The cherry-picked clip the company posted on social media does not.
Image-to-Video Benchmark Limitations
No single evaluation captures the full failure-mode profile of an image-to-video model. VBench misses identity preservation and limb coherence. Human preference tests capture holistic quality but are expensive to run at scale and depend on rater pool composition. Raters from one geography may prefer different motion styles than those from another. Neither test measures source-image fidelity at clip end systematically. Neither measures prompt adherence to motion descriptions: the text prompt that says "the car turns left" while the generated clip drifts right.
Selection Bias Inflates Every Score
The evaluation protocol itself introduces bias. Best-of-N selection inflates every score. Models that produce more varied output benefit more from N selections than models that produce consistent but lower-variance output. A model that always generates a decent clip will lose to a model that generates one great clip and nine duds, when only the great one is shown.
Match The Test To Your Task
Cross-reference scores with the specific failure modes that matter for your use case. If you need a face to stay recognisable across a 10-second clip, VBench's object class consistency score is irrelevant. You need human preference data filtered for identity preservation. Or you need to run your own test.
Evaluating Image-to-Video Model Quality
To evaluate a model for a specific use, cross-reference three things.
Check The Published Scores, Footnotes First
Start with the VBench scores, taken from the published paper or the model's own documentation. Read the footnotes. Many models report scores on a subset of VBench dimensions and omit the ones they score poorly on. Runway Gen-4 publishes scores on motion smoothness and temporal consistency but does not report source-image fidelity at clip end. Kling 1.6 publishes scores on aesthetic quality and object class consistency but does not publish on limb coherence.
Find Task-Specific Human Preference Data
Look for human preference test results for the exact task you need. If you need controlled camera movement, find preference tests that include camera-motion control as a variable. If you need start-and-end-frame support, look for tests that specifically evaluate the smoothness of the transition between the specified end frame and the generated clip. Luma Dream Machine includes keyframe support in its documentation, and human preference tests published by third-party evaluators show higher scores for end-frame fidelity over models that do not support it.
Run Your Own 20-Minute Test
Pick three source images of the type you will actually use: a product shot, a portrait, a landscape. Generate one clip from each model. Do not cherry-pick. Take the first output. Compare source-image fidelity at clip end, limb coherence, and background sliding. This takes 20 minutes and is worth more than any published score.
Frequently Asked Questions
What is the biggest failure mode that VBench misses?
Identity preservation across a clip. VBench measures whether an object stays the same class, but not whether a face stays recognisably the same person. A face can shift ethnicity or gender and still score well.
Why do human preference tests show lower scores than company demos?
Company demos show best-of-N output, usually the best from 50 or 100 attempts. Human preference tests evaluate median-of-1 output, what a user gets on the first try. The gap is typically one to two points on a five-point scale.
Can I trust a model's published VBench score?
Only if the model reports scores on all 16 VBench dimensions. Many publish on a subset and omit dimensions they score poorly on, such as temporal flickering or object class consistency. Check the footnotes in the paper or documentation.
What is the single most useful benchmark for my use case?
Run your own test with three representative source images. Take the first output from each model without cherry-picking. Compare source-image fidelity at clip end, limb coherence, and background sliding. That test beats any published benchmark for your specific needs.