CLIP and Semantic Image Search: Find a Scene with Words

Imagine several thousand photos captured during a dance competition. You do not know the exact filenames or the time of every heat, but you can describe what you need: “dancers in black dresses,” “a couple in their final pose under a blue spotlight,” or “a judge watching the floor from the front row.”

Conventional keyword search only works when someone has already attached the right labels. Semantic image–text search takes another route: it turns photos and descriptions into comparable vectors, then ranks the images nearest to the query. The same mechanism can help explore a sports gallery, an archive, a product catalog or a collection of nature observations.

The system does not understand every scene the way a person does. It produces an ordered shortlist that a team can inspect, correct and combine with domain metadata.

CLIP brings images and words closer

CLIP, or Contrastive Language–Image Pre-training, popularized learning a shared space for images and text. In that space, a photo and a description that appear to match are placed closer together. The foundational work also showed how the representation can transfer to new categories without training a separate classifier for each one (CLIP research paper).

In conventional supervised classification, the categories are fixed in advance: black dress, red dress, cha-cha, waltz and so on. Asking a new question may require more labels and sometimes more training. An image–text encoder lets an operator test phrases directly:

• “dancer in a black dress”;
• “black outfit with long sleeves”;
• “dance couple seen from behind”;
• “fast movement with spinning fabric”;
• “award ceremony on the podium.”

That flexibility is useful when a collection does not yet have a settled taxonomy. It does not turn a proximity score into a probability, proof of identity or definitive account of the scene.

PolyReID exposes a SigLIP2 model for this workflow. SigLIP2 belongs to a multilingual family of image–text encoders, and its paper reports improvements for image–text retrieval and visual representations (SigLIP2 paper). In this guide, “CLIP” names the broader semantic-search pattern. Every saved vector must remain associated with its exact model and preprocessing revision.

Three ways to search a collection

The same vector space supports complementary workflows.

Search	Input	Example	Expected result
Text to image	A free-text description	“dancers in black dresses”	Photos nearest to the description
Image to image	A reference photo	One selected gallery image	Visually neighbouring scenes
Text to groups	Several candidate descriptions	“empty floor,” “warm-up,” “heat,” “podium”	A shortlist to organize and verify

Text-to-image search is intuitive when the user can name a color, outfit, action or atmosphere. Image-to-image search helps when they already have a representative photo. Several short prompts can help a team prepare groups without imposing a final taxonomy at the beginning.

The Embeddings API page documents the current routes, limits, retention and technical contract. This article explains practical uses; it does not replace that contract.

Example: find dancers in black dresses

Take a reference image of a dancer wearing a black dress during a figure. The competition gallery also contains dark-blue dresses, black suits worn by partners, dark backgrounds and several dancers in similar outfits.

1. Encode the gallery

Each relevant image or crop receives an embedding. Store the metadata you already know beside the vector: competition, day, session, heat, camera, source-file ID and model revision.

Framing matters. A wide photo may rank highly because of the floor or lighting, while a person-centered crop gives more weight to clothing and silhouette. When an image contains several couples, searching separate crops may be more useful, provided each crop keeps a link to the original photo.

2. Try several prompts

One sentence is not a complete test. Compare the first results for closely related wording:

dancer in a black dress
ballroom dancer, black dress with sparkling details
couple on the floor, female partner dressed in black
black dance outfit in motion

The first results are candidates. A navy dress may appear, as may a male partner whose black suit fills much of the frame. Inspect the rank of relevant images, repeated confusions and how stable the ordering remains across prompts.

3. Add competition context

Metadata can reduce the search space before any vector comparison. If the request concerns Saturday afternoon, one discipline or a specific floor, apply those filters. A useful engine combines visual meaning with simple facts: schedule, category, heat number, a separately read bib or an already reviewed group.

4. Review before publishing

Search can accelerate a photographer’s or operator’s work, but images proposed to a participant should remain verifiable and correctable. Two competitors may wear nearly identical dresses. Finding “a black dress” is not the same as identifying a person.

For a search focused on the same bodily appearance across photos, Person ReID answers a narrower question. To read a visible number, use a bib-number OCR workflow. These signals can complement one another, but they should not be confused.

Original searches that help event teams

Semantic search is not limited to the color of a dress. It can help a photo team cover an event and prepare a gallery that is easier to browse.

A photographer or editor could search for:

• “couples entering the floor” to prepare a chronological story;
• “final pose with both dancers visible” to find closing images;
• “audience applause” or “reaction after the results” to enrich the event narrative;
• “trophy presentation with the judges” to retrieve institutional images;
• “wide shot showing the complete venue” to document atmosphere and sponsors;
• “dancer partially hidden by another couple” to prepare a review queue;
• “dark image with strong motion blur” to shortlist photos for inspection without automatically judging quality.

Judges and organizers may also navigate authorized archives with descriptions such as “start of the couple’s heat,” “side-by-side figure” or “leaving the floor.” The similarity result must not assign a technical score, infer a fault or replace sporting judgment. The tool retrieves and organizes; authorized people remain responsible for decisions affecting participants.

Beyond dance: nature, archives and catalogs

The same mechanism can explore very different collections, provided it is evaluated on their actual images.

Nature photography

A photographer could search for “bird in flight above a wetland,” “bee on a purple flower,” “misty woodland” or “rocky coast at sunset.” An association might shortlist observations that show a habitat, season or visible behavior.

Semantic search does not replace scientific species identification. Closely related species, juveniles and distant subjects require expertise, metadata and sometimes specialized models. Locations of sensitive species also need protection.

Cultural archives and media

In a digitized collection, queries such as “crowd outside a theater at night,” “costume rehearsal” or “outdoor group portrait” can surface material with incomplete captions. An archivist still confirms the context, date and people shown.

Product catalogs and creative work

A creative team might search for “black jacket with gold detail,” “minimal light-wood interior” or “packaging photographed in a natural setting.” Similarity supports visual navigation, while product references, inventory and usage rights remain in the relevant business systems.

Inspection and documentation

Queries such as “rusted surface,” “partially obscured sign” or “loose cable” can help explore technical imagery. Concluding that a defect exists or triggering maintenance still requires an inspection protocol, validated thresholds and often a specialized model.

What semantic search does not guarantee

A general model may favor a color, background, composition or secondary object that does not match the user’s intent. It can miss an important distinction: black versus navy, rehearsal versus competition, or one species versus a close relative.

Build the workflow so that it:

1. stores the model and preprocessing revision with every embedding;
2. evaluates queries on a representative corpus, including difficult cases;
3. measures relevance across the first results, not only the top score;
4. tests several phrasings and documents recurring confusions;
5. applies available business filters before vector search;
6. presents results as ranked candidates;
7. provides human review and a correction path;
8. reindexes and reevaluates the collection after a model change.

A high score only means that two inputs are close in the model’s space. It proves neither a person’s identity, nor a species label, nor the artistic quality of a photograph.

Start with a question you can verify

A useful pilot can be small: choose a bounded collection, write five to ten practical queries, encode the images, inspect the first results, label the errors and measure the review time actually saved.

For a competition, “find dancers in black dresses in this session” is a better starting point than a general promise to understand every image. The question is clear, results can be checked and confusions are easy to document. The team can then test more original searches for gestures, program moments, judges, audience reactions or atmosphere.

CLIP’s value is not eliminating editorial or domain work. It makes a collection searchable with words, then lets a team turn the shortlist into a dependable gallery, archive or service.