# CHNB

The Classical Hollywood Narrative Benchmark

## Summary

This repository contains benchmark questions/answers for multimodal narrative understanding of 61 popular Hollywood movies, with questions pertaining to temporality, plot, character, setting, perspective, representation, symbolism and object identification. This research is described in the following article:

David Bamman, Kent K. Chang, Allison Cooper, Juishan Hsu, Reina Kushihashi, Madison Mar, Arnav Podichetty, Rachael Samberg, Ipek Nil Sancak and Yuhan Shao (2026), "Evaluating Multimodal Narrative Understanding of Popular Hollywood Films", [preprint](https://people.ischool.berkeley.edu/~dbamman/pubs/pdf/chnb.pdf).

Information on identifying popular Hollywood movies (that forms the basis of this collection) can be found on a companion repository: [https://github.com/bamman-group/variety-boxoffice/](https://github.com/bamman-group/variety-boxoffice/).


## Data

`data/movie_metadata.tsv` contains metadata for each movie, including:

* title
* year of release
* IMDB id
* video_file_id
* genre
* director
* AFI id
* copyright registration number
* copyright registration date
* Internet Archive URL for mp4 file

`data/benchmark.jsonl` contains the 628 benchmark questions

`data/non_benchmark.jsonl` contains the 151 non-benchmark (memorized) questions

Each jsonl file contains the following fields:

* question ID (qid)
* video_file_id (keyed to same field in `movie_metadata.tsv`)
* category (Temporality, Plot, Character, etc.)
* question
* option_A
* option_B
* option_C
* option_D
* answer (choice among A B C D)

---

## Download movies

```bash
cd scripts
chmod 755 download_movies.sh
./download_movies.sh
```

## Paths and file structure

The code resolves all filesystem locations through `util.paths`. Defaults use
the repository layout:

```text
    chnb/
    ├── data/
    │   ├── benchmark.jsonl
    │   ├── non_benchmark.jsonl
    │   ├── movie_metadata.tsv
    │   ├── mp4/                <- populated by scripts/download_movies.sh
    │   ├── frames/             <- pre-computed frame extractions
    │   ├── clip_embeddings/    <- CLIP features
    │   ├── clips/              <- shot-grouped video chunks
    │   ├── captions/           <- per-chunk captions
    │   └── subtitles/          <- per-movie ASR transcripts
    ├── output/
    │   ├── results/                <- one CSV per (experiment, model, config)
    │   └── world_state_histories/  <- cached chronological caption assemblies (Socratic)
    └── ...
```

Common path overrides:

| env var | default |
|---|---|
| `CHNB_DATA_ROOT` | `data/` |
| `CHNB_OUTPUT_ROOT` | `output/` |
| `CHNB_BENCHMARK` | `$CHNB_DATA_ROOT/benchmark.jsonl` |
| `CHNB_MOVIE_METADATA` | `$CHNB_DATA_ROOT/movie_metadata.tsv` |
| `CHNB_VIDEO_DIR` | `$CHNB_DATA_ROOT/mp4` |
| `CHNB_SUBTITLES_DIR` | `$CHNB_DATA_ROOT/subtitles` |
| `CHNB_FRAMES_DIR` | `$CHNB_DATA_ROOT/frames` |
| `CHNB_CLIP_EMBEDDINGS_DIR` | `$CHNB_DATA_ROOT/clip_embeddings` |
| `CHNB_CLIPS_DIR` | `$CHNB_DATA_ROOT/clips` |
| `CHNB_CAPTIONS_DIR` | `$CHNB_DATA_ROOT/captions` |
| `CHNB_SHOTS_DIR` | `$CHNB_DATA_ROOT/shots` |
| `CHNB_RESULTS_DIR` | `$CHNB_OUTPUT_ROOT/results` |
| `CHNB_WSH_DIR` | `$CHNB_OUTPUT_ROOT/world_state_histories` |

Before launching a server run, confirm the resolved paths:

```bash
python -m util.paths
```

## Demo notebooks

Two minimal end-to-end examples that load one benchmark question, call one
model, and print the answer alongside the gold:

- **`demo/demo_vertex_full_video.ipynb`** — Gemini 3 Flash answering with native long-video access. Requires `GOOGLE_CLOUD_PROJECT` and an mp4 in `data/mp4/`.
- **`demo/demo_qwen3vl_frames.ipynb`** — Qwen3-VL-8B answering with a 64-frame uniform sample. Requires a CUDA GPU; loads weights from HuggingFace on first run.

## Run experiments

Set up the environment:

```bash
pip install -r requirements.txt
cp .env.example .env
```

Pre-processing the videos: First look into the shot detection pipeline from [Bamman et al. 2024](https://github.com/dbamman/movie-representation/tree/main/pipeline).


```bash
bash scripts/preprocess_all.sh
```


To run the evaluation:

```bash
# Run all experiments
bash scripts/run_all_experiments.sh

# Evaluate a single paradigm/model
bash experiments/end-to-end/gemini3flash/run.sh
```

Or experiment with just one question:

```bash
bash proc_one.sh --qid 1 --models all --frames 64
```

---

## Notes

The films that comprise this benchmark were selected based on publication years prior to 1931 or to lack of evidence of required registration renewal; while we exclude several categories of protected films (e.g., adaptations of copyrighted works), inclusion does not constitute a legal warranty of public domain status.