How a face search works

A face search engine answers one question: where else on the public web does a face like this one appear? The answer is produced by a pipeline that has looked the same in outline since deep-learning face models became practical around 2015, whichever engine runs it: PimEyes, FaceCheck.ID, FaceSearch and Lenso.ai all follow it, with different indexes, thresholds and policies.

1. Step 1

   Crawl

   A crawler fetches publicly accessible pages and the images on them, as a web search engine does for text.

   ›
2. Step 2

   Detect and align

   Each image is scanned for faces; every face found is cropped and rotated to a standard pose.

   ›
3. Step 3

   Embed

   A neural network turns each aligned face into an embedding, a vector of a few hundred numbers.

   ›
4. Step 4

   Index

   Embeddings are stored with the image URL and page URL in a structure built for nearest-neighbour lookup.

   ›
5. Step 5

   Query

   Your uploaded photo goes through the same detect, align and embed steps.

   ›
6. Step 6

   Rank

   The index returns the stored embeddings closest to yours; results are ordered by distance.

   ›
7. Step 7

   Show sources

   Each result is shown as the matched image plus the page it was found on.

What happens to the photo you upload?

The upload is decoded, usually stripped of EXIF metadata and re-encoded, then passed to a face detector. If no face is found, the search stops. If several are found, engines either ask you to pick one or search each in turn. The chosen face is aligned (eyes level, nose centred) and resized to the input size the embedding model expects, commonly 112 by 112 pixels. Whether the upload is kept afterwards is a policy decision that differs by engine; the engines index records each one's stated retention.

What is an embedding and why does it matter?

An embedding is the model's description of a face as numbers, arranged so that two photos of the same person land close together and photos of different people land far apart. The model learns this arrangement from millions of labelled face pairs. The embedding is not an image and cannot be turned back into one with any fidelity, but it is derived from the face and is treated as biometric data in the GDPR, Illinois BIPA and similar laws. See embeddings and similarity for the geometry.

How does the engine find matches among billions of faces?

Comparing your embedding with every stored embedding one by one would be too slow at web scale. Engines use approximate nearest-neighbour indexes (graph-based structures such as HNSW, or clustered inverted lists) that find the closest vectors by visiting a small fraction of the data. The trade-off is that an approximate index can miss a close match that an exhaustive scan would find, which is one reason the same photo can return slightly different results on different days. Building a web-scale face index covers the engineering.

What does the score mean?

The similarity score is the distance between two embeddings, often rescaled to a percentage. It measures how alike two faces look to the model, not the probability that they are the same person. Above the engine's threshold a pair is shown as a match; below it, hidden. Thresholds are set by each operator and are rarely published. Look-alikes, relatives and low-quality photos push scores up or down in ways the number does not explain, which is the doppelganger effect.

Why do results differ between engines?

Three things differ: the index (which sites were crawled and how recently), the model (which training data and architecture produced the embeddings) and the policy (thresholds, how many results are shown, whether source links are free). Two engines with equally good models return different pages because they crawled different parts of the web. This is why comparisons that count matches on a single photo say more about index coverage than about the model.

How is this different from Google Lens or TinEye?

Reverse image search engines compare whole images, usually with a perceptual hash or a general visual embedding, and find copies, crops and edits of one picture. They can return the same person by accident when the photo is a copy. A face search deliberately discards everything except the face, so a different photo taken years later can still match. Google Lens and TinEye are the first kind; the engines listed above are the second.