Skip to content
BranchesTags

Repository files navigation

CLIP-CC-Bench — Evaluating Paragraph-Level Video Descriptions in Video–Language Models

CLIP-CC-Bench: Evaluating Paragraph-Level Video Descriptions in Video–Language Models

A comprehensive evaluation framework that uses state-of-the-art text embedding models to measure the quality of video descriptions generated by vision-language models (VLMs). The framework computes semantic similarity between model predictions and reference descriptions at both document-level (coarse-grained) and sentence-level (fine-grained) granularity, and aggregates an ensemble of five embedding judges with Borda count.

Links: Dataset — GitHub · Hugging Face | Leaderboard website — multimodal-intelligence-lab.github.io/CLIP-CC-Bench | Paper — arXiv:2608.04302 (EvalMG @ SIGIR 2026; see Citation)

CLIP-CC-Bench is the evaluation framework and leaderboard; CLIP-CC is the underlying dataset of 200 human-narrated movie-clip descriptions it scores against.

Overview

CLIP-CC-Bench is a text embedding-based evaluation framework that:

  • Evaluates 17 vision-language models (VLMs) using an ensemble of 5 state-of-the-art text embedding models
  • Ranks VLMs based on aggregated performance across all embedding model judges

Leaderboard

Final ranking of the 17 VLMs on CLIP-CC-Bench: Borda = Borda count across the 5 embedding judges; Mean HM-CF = average harmonic mean of coarse- and fine-grained similarity across judges. Full per-judge scores are in the results/embedding_models/aggregated_results/ folder.

Rank VLM Borda Mean HM-CF
1 VideoLLaMA3 80 0.67
2 mPLUG-Owl3 75 0.66
3 LLaVA-OneVision 67 0.64
4 ViLAMP 67 0.64
5 LongVU 61 0.63
6 Qwen2.5-72B 55 0.62
7 Qwen2.5-32B 48 0.61
8 VideoChat-Flash 42 0.60
9 MiniCPM-V 42 0.60
10 Video-XL 36 0.59
11 ShareGPT4Video 29 0.58
12 InternVL2 27 0.58
13 TimeChat 20 0.56
14 LLaVA-NeXT-Video 16 0.55
15 TS-LLaVA 9 0.53
16 Oryx 6 0.52
17 LongVA 0 0.48

Project Structure

CLIP-CC-Bench/
├── src/
│   ├── configs/                      # Embedding model configurations (YAML)
│   │   ├── nv-embed.yaml             # NV-Embed-v2 configuration
│   │   ├── kalm.yaml                 # KaLM-Embedding-Gemma3-12B-2511
│   │   ├── nemo.yaml                 # llama-embed-nemotron-8b
│   │   ├── gte.yaml                  # gte-Qwen2-7B-instruct
│   │   └── qwen.yaml                 # Qwen3-Embedding-8B
│   ├── scripts/                      # Evaluation and ranking scripts
│   │   ├── run_nv_embed_evaluation.py
│   │   ├── run_kalm_evaluation.py
│   │   ├── run_nemo_evaluation.py
│   │   ├── run_gte_evaluation.py
│   │   ├── run_qwen_evaluation.py
│   │   └── rank_vlms.py              # VLM ranking algorithm
│   └── utils/                        # Model implementations and utilities
├── embedding_models/                 # Embedding-judge weights go here (not included)
│   ├── NV-Embed-v2/                  # See embedding_models/README.md
│   ├── KaLM-Embedding-Gemma3-12B-2511/
│   ├── llama-embed-nemotron-8b/
│   ├── gte-Qwen2-7B-instruct/
│   └── Qwen3-Embedding-8B/
├── venv_configs/                     # Per-judge virtual-environment setup scripts
├── data/
│   ├── ground_truth/
│   │   └── clip_cc_dataset.json      # Reference descriptions (frozen CLIP-CC snapshot)
│   └── models/                       # VLM-generated descriptions (17 models)
│       ├── internvl.json
│       ├── llava_next_video.json
│       └── ... (17 VLM models)
├── results/
│   └── embedding_models/
│       └── aggregated_results/       # Published per-judge scores (shipped)
│           ├── aggregated_results.csv    # 17 VLMs × 5 judges (coarse / fine / HM-CF)
│           ├── cross_decoder_stats.json
│           └── <judge>/<vlm>.json
├── run_all_evaluations.sh            # Master evaluation script
├── LICENSE                           # MIT (code)
├── DATA_LICENSE                      # CC-BY-4.0 (reference summaries)
└── README.md

Running the pipeline regenerates the full results tree, including results/embedding_models/logs/, results/embedding_models/individual_results/ (per-video detail), and results/ranking/ (vlm_overall_ranking.csv, vlm_per_judge_metrics.csv). This repository ships the aggregated results behind the published numbers; the larger per-video artifacts regenerate on a run.

Embedding Models

The framework uses an ensemble of 5 state-of-the-art text embedding models as judges:

Embedding Model Parameters Embedding Dim Max Length Implementation
NV-Embed-v2 (NVIDIA) - 4,096 32,768 Custom local model
KaLM-Embedding-Gemma3-12B-2511 (PGFoundation) 12B - - SentenceTransformer
llama-embed-nemotron-8b (NVIDIA) 8B 4,096 4,096 SentenceTransformer
gte-Qwen2-7B-instruct (Alibaba) 7B 3,584 32,768 SentenceTransformer
Qwen3-Embedding-8B (Qwen) 8B Variable 32,768 SentenceTransformer

All models use official/recommended implementations from their respective HuggingFace model cards.

Video Language Models Evaluated

The framework evaluates predictions from 17 vision-language models:

  • InternVL (internvl)
  • LLaVA-NeXT-Video (llava_next_video)
  • LLaVA-OneVision (llava_one_vision)
  • LongVA (longva)
  • LongVU (longvu)
  • MiniCPM (minicpm)
  • mPLUG (mplug)
  • Oryx (oryx)
  • Qwen2.5-32B (Qwen2.5-32B)
  • Qwen2.5-72B (Qwen2.5-72B)
  • ShareGPT4 (sharegpt4)
  • TimeChat (timechat)
  • TS-LLaVA (ts_llava)
  • Video-XL (video_xl)
  • VideoChatFlash (videochatflash)
  • VideoLLaMA3 (videollama3)
  • ViLaMP (vilamp)

Setup

Prerequisites

  • Python 3.9+
  • CUDA-capable GPU (24GB+ VRAM recommended for large embedding models)
  • ~70GB disk space for all embedding models
  • PyTorch 2.0+
  • transformers >= 4.42.0
  • sentence-transformers >= 2.7.0

Installation

1. Clone the repository

git clone https://github.com/Multimodal-Intelligence-Lab/CLIP-CC-Bench.git
cd CLIP-CC-Bench

2. Setup virtual environments

Each embedding model requires an isolated virtual environment with specific dependencies. See venv_configs/README.md for detailed setup instructions:

# Setup all 5 virtual environments
# Follow instructions in venv_configs/README.md

Quick activation reference:

source venv_configs/activate_nv-embed_env.sh
source venv_configs/activate_kalm_env.sh
source venv_configs/activate_nemo_env.sh
source venv_configs/activate_gte_env.sh
source venv_configs/activate_qwen_env.sh

# Deactivate when done
deactivate

3. Download embedding models

Download the required embedding model weights from HuggingFace and place them directly in their respective directories under embedding_models/. Ensure model files are placed directly without creating subdirectories. Total storage required: ~70GB.

4. Prepare evaluation data

# Reference descriptions (ground truth)
data/ground_truth/clip_cc_dataset.json

# VLM-generated predictions
data/models/{vlm_name}.json

Reproducing Results

Option 1: Run All Evaluations (Recommended)

The master script runs all 5 embedding model evaluations sequentially and computes final VLM rankings:

./run_all_evaluations.sh

This script will:

  1. Activate each embedding model's virtual environment
  2. Run evaluation for all 17 VLMs using that embedding model
  3. Generate individual results, aggregated statistics, and logs
  4. Move to the next embedding model
  5. Compute final VLM rankings across all embedding models
  6. Save comprehensive results to results/ directory

Option 2: Run Individual Embedding Models

Evaluate using a specific embedding model:

# Activate the target environment
source venv_configs/activate_nv-embed_env.sh

# Run evaluation with config file
python src/scripts/run_nv_embed_evaluation.py --config src/configs/nv-embed.yaml

# Deactivate when done
deactivate

Repeat for the other embedding models (run_kalm_evaluation.py, run_nemo_evaluation.py, run_gte_evaluation.py, run_qwen_evaluation.py) with their respective configs and environments.

Option 3: Evaluate Specific VLMs Only

To evaluate a subset of VLMs (useful for testing):

source venv_configs/activate_nv-embed_env.sh

python src/scripts/run_nv_embed_evaluation.py \
  --config src/configs/nv-embed.yaml \
  --models internvl llava_one_vision longvu

deactivate

Generate VLM Rankings

After running all embedding model evaluations, generate the final rankings:

python src/scripts/rank_vlms.py

This creates results/ranking/vlm_overall_ranking.csv (final rankings) and results/ranking/vlm_per_judge_metrics.csv (per-embedding-model metrics).

Results

The published numbers reported in the paper are summarized in the Leaderboard above. The full per-judge scores for all 17 VLMs are shipped in:

  • results/embedding_models/aggregated_results/aggregated_results.csv — cross-model summary table (each VLM × each judge: coarse, fine-F1, HM-CF)
  • results/embedding_models/aggregated_results/<judge>/<vlm>.json — per-judge, per-VLM aggregated detail

Running the pipeline additionally produces execution logs, per-video detailed results, and the ranking CSVs under results/ (see the note in Project Structure).

Data Format

Reference Descriptions (Ground Truth)

File: data/ground_truth/clip_cc_dataset.json — a frozen snapshot of the CLIP-CC dataset included here so the benchmark reproduces exactly.

[
  {
    "id": "001",
    "file_link": "https://www.youtube.com/watch?v=...",
    "summary": "The video begins with a man wearing sunglasses and a light-colored shirt driving a car..."
  }
]

VLM Predictions

File: data/models/{vlm_name}.json

{
  "001": "The video shows a man driving a car while appearing tense...",
  "002": "A person is shown lying down with headphones on, eyes closed..."
}

Adding New Models

To evaluate a VLM not currently in the benchmark:

  1. Generate predictions in the required format: 
    {
  "001": "Your VLM's predicted description for video 001...",
  "002": "Your VLM's predicted description for video 002..."
}
  2. Save predictions to data/models/your_vlm_name.json
  3. Update configs to include your VLM (add "your_vlm_name" to the models list in src/configs/)
  4. Run evaluation: 
    ./run_all_evaluations.sh
  5. Check results in results/embedding_models/aggregated_results/ and results/ranking/vlm_overall_ranking.csv

Citation

If you use CLIP-CC-Bench in your research, please cite:

@misc{ali2026clipccbench,
  title         = {{CLIP-CC-Bench}: Evaluating Paragraph-Level Video Descriptions in Video--Language Models},
  author        = {Ali, Mukhtiar and Dubey, Harsh and Mishra, Sugam and Pack, Chulwoo},
  year          = {2026},
  eprint        = {2608.04302},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  note          = {Presented at the 2nd Workshop on Evaluation for Multimodal Generation (EvalMG), ACM SIGIR 2026},
  url           = {https://arxiv.org/abs/2608.04302}
}

License

The evaluation code is released under the MIT License. The human-written reference summaries in data/ground_truth/ are released under CC-BY-4.0; the linked source videos remain the property of their original copyright holders (see DATA_LICENSE).

Acknowledgments

  • NVIDIA for NV-Embed-v2 and llama-embed-nemotron-8b models
  • PGFoundation for KaLM-Embedding-Gemma3-12B-2511 model
  • Alibaba NLP for gte-Qwen2-7B-instruct model
  • Qwen Team for Qwen3-Embedding-8B model
  • HuggingFace for model hosting and the transformers library
  • Sentence-Transformers for the embedding framework

Contact

For questions or issues, please open an issue on this repository, or contact the corresponding author: Chulwoo Pack (chulwoo.pack@sdstate.edu).

About

Evaluation framework and leaderboard for paragraph-level, minute-scale video description: 200 movie clips, 17 video-language models, 5 embedding judges (EvalMG @ SIGIR 2026).

multimodal-intelligence-lab.github.io/CLIP-CC-Bench/

Topics

benchmarkevaluationllm-evaluationmultimodalvideo-captioningvideo-language-modelvideo-understandingvision-language-models

Resources

Readme
MIT license
Activity
Custom properties

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Used by

Contributors

Languages