GLM-5V-Turbo

Paper Summary: A Very Big Video Reasoning Suite (VBVR)
1. Overview
As video generation models (such as Sora and Veo) continue to improve rapidly in visual quality, their limitations in reasoning are becoming increasingly apparent. Existing video reasoning benchmarks are often relatively small in scale and lack systematic theoretical grounding. To address this gap, this paper introduces the VBVR (Very Big Video Reasoning) suite, which includes a large-scale dataset, VBVR-Dataset, and a verifiable evaluation framework, VBVR-Bench.
VBVR is designed to advance video reasoning models beyond pure “visual generation” toward capabilities grounded in “physical-world commonsense and logical reasoning” through a systematic task design based on cognitive architecture.
Figure 1: Overview of VBVR. The left panel shows a representative task grid covering five cognitive faculties; the center panel compares the scale of VBVR with other datasets (2.015M samples); the radar chart in the lower-right corner shows model performance across different cognitive faculties.
2. VBVR-Dataset: A Large-Scale and Systematic Dataset
2.1 Cognitive Architecture and Task Design
The core innovation of VBVR-Dataset lies in its task taxonomy, which is built on theories from human cognitive science (such as Kant and Anderson). The dataset decomposes video reasoning into five core cognitive faculties:
Perception: extracting structured representations from sensory input.
Transformation: operating on and composing mental representations.
Spatiality: intuitive understanding of position and navigation.
Abstraction: distilling general patterns from concrete experience.
Knowledge: applying prior knowledge and logical rules.
The dataset contains 200 tasks and approximately 2.015 million samples in total (1 million training samples and 1 million test samples).
2.2 Scale Comparison
Compared with existing video reasoning benchmarks (such as Video-Zero-Shot and Ruler-Bench), VBVR-Dataset improves by an order of magnitude in both number of tasks and dataset size, providing sufficient “fuel” for model training.
Table 1: Comparison between VBVR-Dataset and existing video reasoning benchmarks. VBVR far exceeds previous datasets in both task count (200) and sample size (2M+).
2.3 Task Examples
VBVR features highly diverse task designs, ranging from simple geometric shape recognition to complex physical simulation and logical planning. The figure below shows sample tasks, including polygon recognition, pipe connection, grid navigation, maze solving, and sliding puzzle tasks.
Figure 2: Sample tasks generated by the VBVR parametric task suite. Each sequence illustrates the reasoning process required to arrive at a valid solution, covering perception, spatial reasoning, and logical operations.
2.4 Data Generation Pipeline
To ensure both data quality and scale, VBVR adopts a distributed parametric generation pipeline. Tasks are first rigorously designed and reviewed, then implemented through standardized generator templates, and finally generated and stored at scale in parallel using cloud services (AWS Lambda).
Figure 3: System pipeline for VBVR data generation. Starting from cognitive architecture, tasks proceed through task design and generator implementation, and are ultimately written to S3 storage through distributed Lambda workers.
3. VBVR-Bench: A Verifiable Evaluation Framework
3.1 Evaluation Results
VBVR-Bench provides a rule-based, reproducible, and interpretable evaluation framework. The research team conducted a comprehensive evaluation of current state-of-the-art video generation models, including open-source models such as CogVideoX and closed-source models such as Sora 2 and Veo 3.1.
The results show that even the best-performing model (such as Sora 2) achieves an overall score of only around 0.546, far below human performance (0.974). This indicates that current models still face major challenges on video tasks requiring strict logical reasoning and physical consistency.
Table 3: Benchmark results on VBVR-Bench. Shows model performance on in-domain and out-of-domain tasks. Bold indicates the best result, and underlining indicates the second best.
3.2 Human Alignment Analysis
To validate the effectiveness of VBVR-Bench, the study conducted a large-scale human preference alignment analysis. The results show that VBVR’s automatic evaluation scores are highly correlated with human preference scores (correlation coefficient ρ>0.93), demonstrating that the benchmark accurately reflects models’ actual reasoning capabilities.
Figure 4: Alignment analysis between VBVR-Bench evaluation results and human preferences. The left panel shows in-domain tasks, and the right panel shows out-of-domain tasks; both exhibit strong linear correlation.
4. Scaling Laws and Emergent Capabilities
4.1 The Effect of Data Scale
The study further explores how data scale affects model reasoning ability. By fine-tuning the Wan2.2 model on VBVR-Dataset, the team found that as training data increased from 0 to 500K, model performance showed a steady upward trend across all metrics. This demonstrates the critical role of high-quality, large-scale reasoning data in improving model performance.
Table 4: Model performance at different data scales (VBVR-Wan2.2). As the amount of training data increases, the model’s scores on both in-domain and out-of-domain tasks improve significantly.
4.2 Qualitative Analysis and Emergent Behaviors
In qualitative analysis, the model fine-tuned on VBVR data (VBVR-Wan2.2) demonstrated more controllable execution than Sora 2. For example, on tasks such as “delete a specific symbol” or “precisely rotate an object,” VBVR-Wan2.2 was able to follow instructions strictly, whereas Sora 2 was more prone to object deformation or execution failure.
In addition, the study observed several emergent behaviors, such as the model spontaneously choosing a completion strategy (Self-chosen completion policy) or providing “rationalized” interpretations of a scene (Rationalizing), suggesting that large-scale reasoning training may unlock deeper cognitive capabilities.
Figure 6: Qualitative comparison between VBVR-Wan2.2 and Sora 2. Panel A shows VBVR-Wan2.2’s advantages on controlled execution tasks; Panel B shows an emergent self-chosen completion policy; Panel C shows the model’s limitations when handling complex constraints.
5. Conclusion
By providing the largest and most systematic video reasoning dataset and evaluation benchmark to date, the VBVR suite reveals the shortcomings of current video generation models in logical reasoning and validates the feasibility of improving model capabilities through more high-quality reasoning data. This lays the foundation for future development of general-purpose video agents with genuine physical-world commonsense and logical reasoning abilities.

{
    "0": [
        {
            "label": "horse-0",
            "bbox_2d": [574, 512, 641, 602]
        }
    ],
    "1": [
        {
            "label": "horse-0",
            "bbox_2d": [556, 469, 605, 585]
        }
    ],
    "2": [
        {
            "label": "horse-0",
            "bbox_2d": [532, 458, 579, 562]
        }
    ],
    "3": [
        {
            "label": "horse-0",
            "bbox_2d": [518, 497, 592, 637]
        }
    ],
    "4": [
        {
            "label": "horse-0",
            "bbox_2d": [534, 597, 614, 758]
        }
    ],
    "5": [
        {
            "label": "horse-0",
            "bbox_2d": [529, 627, 602, 779]
        }
    ],
    "6": [
        {
            "label": "horse-0",
            "bbox_2d": [509, 609, 582, 787]
        }
    ],
    "7": [
        {
            "label": "horse-0",
            "bbox_2d": [472, 593, 579, 809]
        }
    ],
    "8": [
        {
            "label": "horse-0",
            "bbox_2d": [450, 605, 592, 825]
        }
    ],
    "9": [
        {
            "label": "horse-0",
            "bbox_2d": [455, 600, 614, 838]
        }
    ],
    "10": [
        {
            "label": "horse-0",
            "bbox_2d": [448, 569, 659, 852]
        }
    ],
    "11": [
        {
            "label": "horse-0",
            "bbox_2d": [436, 562, 686, 862]
        }
    ],
    "12": [
        {
            "label": "horse-0",
            "bbox_2d": [419, 553, 709, 889]
        }
    ],
    "13": [
        {
            "label": "horse-0",
            "bbox_2d": [409, 554, 721, 924]
        }
    ],
    "14": [
        {
            "label": "horse-0",
            "bbox_2d": [398, 555, 705, 930]
        }
    ],
    "15": [
        {
            "label": "horse-0",
            "bbox_2d": [435, 531, 763, 920]
        }
    ],
    "16": [
        {
            "label": "horse-0",
            "bbox_2d": [480, 533, 779, 909]
        }
    ],
    "17": [
        {
            "label": "horse-0",
            "bbox_2d": [528, 574, 769, 900]
        }
    ]
}