Datasets8 min read

Person ReID Dataset Benchmarks: Protocols, Bias and Limits

Compare Person ReID benchmarks by capture domain, protocol and bias, then learn what a reproducible score can—and cannot—support in a production workflow.

PolyReID Engineering25 July 2026

Take an illustrative comparison: Model A reports 92.4 mAP and Model B reports 89.7. The winner seems obvious—until you notice that one result removes difficult detections, uses a different query list and adds re-ranking. The decimal places are precise; the comparison is not.

A Person ReID dataset is more than a folder of labeled crops. It defines the game: which identities appear, how cameras and sessions differ, what becomes a query, what enters the gallery and which matches are excluded. A benchmark score only makes sense inside those rules.

That is why “trained on dataset X” and “achieved mAP Y” leave out the part you need to judge. Two teams can start with the same images and still evaluate different tasks. Before you compare models, compare how each number was produced.

For the wider system context, start with the Person ReID guide. This article focuses on dataset and benchmark design rather than explaining Rank-k or mAP in depth; those definitions live in the Person ReID evaluation metrics guide.

Read the rules before the score

A reproducible image-based ReID benchmark should make at least six elements explicit:

Identity universe. Training and test identities are normally disjoint when the goal is retrieval of previously unseen people.
Capture domains. Cameras, locations, times, weather, lighting and image-processing pipelines determine the visual variation.
Query construction. The query set defines which observations initiate retrieval.
Gallery construction. The gallery contains relevant images, distractors and possibly identities absent from the query set.
Exclusion rules. Same-camera matches, junk detections and invalid identities may be ignored under a published protocol.
Scoring pipeline. Distance, normalization, optional re-ranking and aggregation rules must remain fixed.

The last two points are easy to miss because they often live in code rather than the abstract. If one implementation removes difficult same-camera images while another treats them as negatives, their results are not directly comparable. If you tune hyperparameters on the final test identities, the test set has quietly become development data.

Three datasets, three visual worlds

The following examples illustrate why dataset choice is part of the research question rather than a neutral download decision.

Dataset	Capture design	What it helps test	Important limitation
Market-1501	Six cameras at one campus; detector-produced pedestrian boxes; more than 32,000 annotated boxes in the original paper	Standard image retrieval, detected-box noise and large galleries	One site and an older surveillance domain cannot represent modern event photography or every population
MSMT17	Fifteen indoor and outdoor cameras across multiple time periods; 4,101 identities and 126,441 boxes in the paper	Lighting, scene and camera diversity; harder cross-domain evaluation	Still a specific surveillance network with its own collection and annotation choices
VAST-ReID	Low-light videos from 21 locations; 169 identities; a BoxTrack-ReID subset with identity-aligned crops	Low light, degraded imagery, temporal context and attribute-rich evaluation	Smaller identity count and a deliberately specialized low-light domain

Market-1501 was influential because it moved beyond tiny hand-cropped sets. Its authors used a pedestrian detector, included multiple observations per identity and evaluated image search at a larger scale. Those design decisions made the benchmark more realistic for its time, but they are not timeless assumptions.

MSMT17 was introduced with more cameras, indoor and outdoor scenes, longer time coverage and stronger lighting variation. The paper also demonstrated a pronounced domain gap: performance on one collection did not transfer cleanly to another. That remains an essential lesson—dataset size alone does not guarantee deployment relevance.

VAST-ReID, published at WACV 2026, intentionally concentrates on low-light conditions. It is useful when that is the target failure mode, not because it supersedes every earlier dataset.

No row wins this table. Choose the dataset whose capture process stresses the behavior you need to measure, then add target-domain evidence. A low-light benchmark answers a sharper question than “does this work everywhere?”

One once-common dataset is intentionally absent from this comparison. DukeMTMC-ReID was taken down, and a NeurIPS 2020 ethics review required its removal from an accepted paper and stated that it should no longer be used. Legacy results can be discussed as historical context only when that withdrawal is disclosed; do not recommend downloading or starting new experiments with the dataset.

One photo burst can quietly leak across a split

Consider an event-photo system in which one participant appears in dozens of near-duplicate images. A benchmark can be made artificially easy by placing frames from one burst on both sides of the query-gallery boundary. The model may appear to retrieve identity while exploiting near-duplicate scene cues.

Conversely, a protocol can become unrealistically hard by declaring every major wardrobe change a positive match even though the model is explicitly appearance-based and receives no temporal, face or roster context. Difficulty is not automatically realism.

Write a plain-language task definition before generating files:

Given one authorized crop from session A,
rank crops from sessions B and C.
Training identities do not occur in the final test.
The gallery includes people absent from the query set.
Near-duplicate exports of the same source frame stay in one split.

That paragraph resolves more ambiguity than a model name. It also exposes whether the intended task is closed-set ranking, open-set retrieval, cross-camera matching, cross-session matching or end-to-end person search from uncropped frames.

Look for the shortcuts hiding in the frame

Demographic representation matters, but ReID dataset bias also includes:

clothing conventions and the frequency of similar uniforms;
body visibility, pose and mobility;
camera height, focal length and viewpoint;
venue, flooring, walls and repeated background;
detector behavior and box tightness;
image compression, color processing and resolution;
time gaps and wardrobe changes;
which people were annotated, omitted or labeled as distractors.

A model can exploit a camera-specific background or event-specific costume distribution without learning the intended invariance. A high same-dataset score may therefore measure compatibility with collection artifacts.

Create dataset slices before inspecting model results. Examples include indoor versus outdoor, bright versus low light, full-body versus partial, detector-generated versus manually corrected, and same-event versus cross-event. Record the number of identities and queries in every slice; a percentage without its denominator hides uncertainty.

A seed cannot reproduce an undocumented task

A useful reproduction package should freeze:

source dataset version, checksum and access date;
permitted uses, license or data-use agreement;
identity mapping and all split files;
raw-to-crop provenance and bounding boxes;
invalid and junk-image rules;
orientation, resize, padding and normalization;
model checkpoint hash and preprocessing revision;
distance definition and whether vectors are L2-normalized;
exact query, gallery and distractor lists;
re-ranking settings, if any;
software versions and deterministic seeds;
per-query outputs, not only final averages.

Seeds cannot repair a leaked split or an undocumented detector. Reproducibility means another team can reconstruct the same task and explain any remaining nondeterminism.

Write the dataset’s biography

The Datasheets for Datasets proposal recommends documenting motivation, composition, collection, preprocessing, uses, distribution and maintenance. Applied to Person ReID, a compact datasheet should answer:

Area	Questions to record
Purpose	Which retrieval task is supported, and which uses are out of scope?
People	Who is represented, how were images obtained, and what consent or lawful basis applies?
Labels	Who annotated identities and boxes, with what instructions and disagreement process?
Structure	How are source assets, crops, identities, cameras and sessions connected?
Splits	Are identities, source bursts and events isolated as intended?
Quality	Which missing, ambiguous, occluded or multi-person samples remain?
Access	Who may download it, for how long, and under which restrictions?
Maintenance	How are corrections, removals and versions communicated?

Do not treat a public download link as permission for every use. Dataset availability, license terms and ethical suitability are separate questions, and links or access processes can change.

Build a benchmark you can explain six months later

Use public benchmarks for regression testing and comparison with published baselines, but build a governed target-domain set for product decisions.

Select one public protocol relevant to the technical question.
Reproduce a documented baseline before modifying the pipeline.
Freeze an identity-disjoint target-domain test set.
Include realistic distractors and hard lookalikes.
Report ranking results with and without any re-ranking stage.
Publish slices, query counts and failure examples.
Keep calibration separate from final testing.
Re-run when crops, cameras, models or gallery rules change.

The goal is not to collect the largest benchmark table. It is to know exactly what each number supports. A public leaderboard can show that an implementation is technically plausible; only a representative, versioned protocol can show how well it fits a particular workflow.

What the leaderboard still cannot tell you

A public benchmark result does not validate a private dataset, checkpoint or serving revision. The internal DINO-ReID report uses private event-photo data and project-specific splits; those results are internal, non-peer-reviewed evidence rather than a new public benchmark. They cannot establish production API performance unless the served checkpoint, preprocessing and target-domain protocol are shown to match the evaluated system.

Likewise, one high mAP or Rank-1 value does not isolate a backbone, loss or augmentation when several factors changed together. Component claims require controlled ablations on the same data and protocol.