Towards Universal Video Retrieval: Generalizing Video Embedding via Synthesized Multimodal Pyramid Curriculum

The prevailing video retrieval evaluation suffers from fragmented benchmarks that obscure true model generalization. To diagnose this limitation, we conduct a structured assessment across 16 established public datasets, revealing critical capability gaps, particularly pronounced deficiencies in fine-grained, temporal, and partially relevant video retrieval. The discoveries directly motivate the holistic design of our framework: V-SynFlow synthesizes 1.55 million high-quality, multi-task video-text pairs explicitly targeting the uncovered deficiencies, while the Modality Pyramid curriculum trains the General Video Embedder (GVE) by leveraging inherent task hierarchies for progressive learning. GVE achieves state-of-the-art zero-shot performance across diverse scenarios. Our analysis further demonstrates that conventional benchmarks poorly correlate with holistic retrieval capability, and identifies partially relevant retrieval as a pivotal yet overlooked real-world challenge. This work establishes a practical pathway toward universal video retrieval by tightly coupling evaluation diagnostics with structured training on diverse synthesized data.

We propose a co-designed framework that jointly rethinks evaluation, data, and modeling to advance video retrieval toward universality—defined as robust zero-shot performance across diverse query formats (textual, composed, visual), granularities (coarse- to fine-grained), and domains (spatial, temporal, partially relevant, long-context).

1. Universal Video Retrieval Benchmark (UVRB). We establish UVRB, a diagnostic evaluation suite comprising 16 datasets spanning 9 abilities: 3 query formats (Textual, Composed, Visual) and 6 domain-specific challenges (Coarse-Grained, Spatial, Temporal, Partially Relevant, Long-Context, and their combinations). Unlike conventional benchmarks that measure only end performance, UVRB enables fine-grained diagnosis of capability gaps and inter-task correlations.

2. V-SynFlow: Scalable Synthesis of High-Fidelity Training Data. Guided by UVRB 's diagnostics, we introduce V-SynFlow, a multi-stage synthesis pipeline that transforms weakly annotated web videos (e.g., WebVid, InternVid) into UVRD—a high-quality dataset of 1.55 million video retrieval pairs. V-SynFlow first applies multi-granular filtering (annotation rectification, cross-modal consistency, temporal dynamics) to construct a clean asset pool. It then leverages a multimodal large language model (MLLM) to enrich semantic content with controlled emphasis on spatial, temporal, and abstract attributes. Finally, it synthesizes diverse task formats—including text-to-video, image-to-video, and text+image-to-video—by conditioning MLLM generation on visual inputs and structured prompts.

3. Modality Pyramid: Curriculum Learning for Generalizable Embeddings. To effectively learn from UVRD 's heterogeneous tasks, we devise the Modality Pyramid, a bottom-up curriculum that exploits latent dependencies among retrieval abilities. Foundational tasks (e.g., image-video alignment) are prioritized early in training, while complex, composite tasks (e.g., long-context composed retrieval) are gradually introduced as the model 's representational capacity matures. Task scheduling is governed by an alignment-aware mechanism that estimates current task difficulty using a prober model and anneals sampling probabilities over epochs. The model—General Video Embedder (GVE), derived from Qwen2.5-VL—is trained with a symmetric InfoNCE loss, hard negative mining, and cross-device in-batch negatives, using parameter-efficient LoRA fine-tuning.

This tightly integrated benchmark-data-curriculum pipeline enables GVE to achieve state-of-the-art zero-shot generalization on UVRB, while revealing critical insights: (i) partially relevant retrieval is a strong proxy for overall universality; (ii) spatial and temporal reasoning remain largely disentangled; and (iii) conventional benchmarks poorly predict cross-task generalization. Our framework thus provides a principled path toward truly universal video retrieval.