Person ReID evaluation metrics: Rank-k, mAP and production tests

A Person ReID system should not be reduced to one “accuracy” percentage. The useful answer is a protocol plus a metric set: define realistic queries and galleries, measure ranking quality with Rank-k and mean Average Precision (mAP), then measure the actual product decision—review, accept or reject—at its chosen operating point.

That distinction matters because Person ReID is normally a retrieval task. A model can place one relevant image first while ordering the remaining relevant images poorly. It can also produce a convincing benchmark ranking while failing on a new camera, a weak crop or a gallery full of similar clothing. Start with the broader practical Person ReID guide if the system boundary is not yet clear.

Freeze the evaluation protocol before computing metrics

For each query crop, the gallery contains candidates to rank. A candidate is relevant when it belongs to the same evaluation identity under the documented ground truth. Before running the model, write down:

• how identities are separated between training, calibration and final test sets;
• whether query and gallery images can come from the same camera or event;
• which same-camera observations, duplicate frames or invalid crops are excluded;
• whether every query has at least one valid relevant item in the gallery;
• how multiple outfits, sessions and venues are represented;
• which model and preprocessing revisions generate the embeddings.

This is not administrative detail. Changing an exclusion rule changes the problem. Letting near-duplicate frames appear on both sides can make retrieval look easier than the intended workflow. Selecting a threshold on the final test set turns that test into calibration data.

The Market-1501 benchmark paper is a useful primary reference for query-gallery evaluation with Cumulative Matching Characteristics (CMC) and mAP. Its protocol is not a universal production protocol; copy the definitions you need, not the dataset assumptions. The open-source Torchreid project is also useful for inspecting a concrete implementation of common ReID evaluation rules.

Rank-k answers an early-hit question

Rank-k asks whether at least one relevant gallery item occurs within the first (k) returned results. For (Q) valid queries:

Rank-k = (1 / Q) × Σ 1[first relevant rank for query q ≤ k]

Rank-1 therefore describes how often the first candidate is relevant. Rank-5 describes how often a reviewer could find at least one relevant candidate within five results. Report the values that match the interface: Rank-20 is less informative when operators only ever see five cards.

Rank-k is intuitive, but it ignores what happens after the first relevant result. A query with one correct image at rank 1 and every other relevant image buried deep can receive the same Rank-1 outcome as a perfectly ordered result set. It also says nothing about how many irrelevant candidates a reviewer must dismiss.

mAP measures the ordering of multiple relevant results

Average Precision rewards rankings that place all relevant items early. For a query (q) with (R_q) relevant gallery items:

AP(q) = (1 / Rq) × Σ Precision@k × relevant(k)
mAP   = average AP(q) over valid queries

Here, relevant(k) is 1 when the result at rank (k) is relevant and 0 otherwise. Precision is sampled at each relevant result. mAP is especially helpful when a person can appear in many gallery images, as in an event-photo collection.

Still, mAP depends on gallery composition. Adding many easy negatives, hard lookalikes or additional relevant frames changes the result. Always publish it with the protocol, gallery-size distribution and model revision. Do not compare two mAP values produced by different filtering rules as if they measured the same task.

Add metrics that match the product

Rank-k and mAP evaluate ranking. Production often adds a score threshold and human review. That creates different questions:

• Precision among surfaced candidates: of the candidates shown or accepted, how many are relevant?
• Recall: of the relevant gallery items, how many were surfaced?
• False matches per query: how many irrelevant candidates enter the review queue?
• No-result rate: how often does the system return nothing above the review threshold?
• Reviewer workload: median and tail number of candidates inspected per query.
• Correction rate: how often does a reviewer reject, merge or split the proposed group?

These quantities must be computed at the operating rule being tested. The companion guide on choosing a Person ReID similarity threshold explains why there is no reusable universal cutoff. Use a separate calibration split to choose the rule, then report its performance once on untouched test data.

For a system that only proposes a ranked list, avoid presenting pairwise “verification accuracy” as the whole result. For a system that automatically accepts pairs, retrieval-only metrics are equally incomplete. Measure the decision the software actually makes.

Slice results before trusting an average

An aggregate can hide a systematic failure. Define slices before looking at the test result, using attributes relevant to the intended environment:

• camera or capture position;
• venue and lighting condition;
• crop source and detector revision;
• blur, truncation and occlusion level;
• outfit change or visually similar uniforms;
• gallery size and number of relevant items;
• demographic or operational groups where lawful, appropriate and adequately governed.

Report the number of queries in every slice. A dramatic percentage from a tiny slice is uncertainty, not a conclusion. When sample sizes permit, add confidence intervals by resampling queries rather than treating every gallery pair as independent.

Input quality deserves its own analysis. The person-crop quality checklist separates representation errors from crops that contain two people, too little of the body or severe blur. Environment changes need a separate domain-shift evaluation, because a stable score distribution alone does not prove stable relevance.

Keep embedding and retrieval variables visible

Store the model revision, preprocessing version, normalization state and distance definition with every run. The guide to ReID embeddings and vector databases explains why vectors from incompatible revisions should not share one comparison space.

Also evaluate the retrieval index. An approximate nearest-neighbour configuration can omit candidates that exact search would have returned. Compare the configured index against an exact-search reference on representative data, then separate index recall from model ranking quality. Otherwise a retrieval configuration change may be mistaken for a model regression.

The public PolyReID API overview describes the embedding endpoint and its version fields. It does not turn a benchmark result into a guarantee for a customer dataset.

A practical evaluation report

A reviewable report can fit on a few pages if it includes the essentials:

1. intended use and prohibited interpretations;
2. dataset origin, consent or lawful basis, and split construction;
3. query, gallery and exclusion rules;
4. crop-generation and quality policy;
5. immutable model, preprocessing and index revisions;
6. Rank-k and mAP with query counts;
7. threshold precision, recall and workload where applicable;
8. predefined slices and uncertainty;
9. known failure examples, not only successful rankings;
10. approval owner and conditions that trigger reevaluation.

Re-run the test when the model, crop pipeline, cameras, index settings or operating rule changes. The objective is not to manufacture the largest number. It is to produce evidence that a specific, versioned workflow behaves acceptably for a specific use, while making its blind spots visible.