Building a web-scale face index

Users judge a face search by what it finds, and what it finds is decided before they upload anything. The index is the engine's memory of the web: every face it has seen, where it saw it, and the embedding it computed. Building and maintaining that memory is the expensive part of running an engine.

What does the crawler fetch?

A crawler starts from seed URLs and follows links, fetching pages and the images they reference, the same way a text search engine does. Engines differ in scope. PimEyes states that it excludes social media and video platforms; FaceCheck.ID emphasises public social profiles and news; Search4Faces covers specific social networks only. Most honour robots.txt and skip pages that require a login, which is what public web means in practice: pages that any visitor can open without an account.

1. Step 1

   Seed and discover

   Start from known sites and sitemaps; follow links; prioritise pages likely to carry photos of people.

   ›
2. Step 2

   Fetch and filter

   Download pages and images; drop icons, logos and images below a size threshold.

   ›
3. Step 3

   Detect and embed

   Run detection on every image; compute an embedding for every face above the quality bar.

   ›
4. Step 4

   Store

   Write embedding, image URL, page URL, crawl date and a thumbnail reference to the index.

   ›
5. Step 5

   Recrawl

   Revisit pages on a schedule; mark removed images; apply opt-out block lists.

How are billions of embeddings searched in milliseconds?

Exact nearest-neighbour search compares the query with every stored vector; at a billion 512-dimensional vectors that is far too slow per query. Engines use approximate nearest-neighbour (ANN) structures. Graph-based methods such as HNSW build a navigable graph over the vectors and walk it greedily towards the query. Quantisation methods (IVF-PQ and relatives) cluster the space and compress each vector so that millions of comparisons fit in memory. Both return most of the true nearest neighbours most of the time, with a tunable recall, which is why a face can be found one day and missed the next after the index is rebuilt.

Why is the index always out of date?

The web changes faster than any crawler can revisit it. A page crawled in March and deleted in April still has its faces in the index until the recrawl, so results point at dead links. A photo posted yesterday is absent until the crawler reaches it. Engines that advertise a deep search are usually querying a second, larger or older partition of the index that is too slow to search by default. None of this is visible in the result count, which is why a count says little.

What leaves the index?

Three mechanisms remove faces. Recrawls drop images that are gone from their source page. Site-level exclusions remove whole domains that opt out or block the crawler. Person-level opt-outs block a specific face: the engine stores an embedding of the requester and suppresses results that match it. The last mechanism means the engine keeps a record of the very face it has promised not to show, a point regulators have raised in the PimEyes and Clearview proceedings.

How large are the indexes?

30bn+

Images Clearview AI reported in its database, 2024 (Dutch DPA decision)

Autoriteit Persoonsgegevens, 3 September 2024

Not published

Index size of most consumer engines

Engine websites, checked November 2026

Days to months

Typical recrawl interval, inferred from dead-link rates

Observation; no engine publishes a schedule

Clearview is the exception that publishes a figure because regulators forced it into the record. Consumer engines describe their coverage in words ('billions of images', 'the public web') rather than numbers. Treat any size claim as marketing unless it comes from a regulator's decision or a court filing.

Why is scale itself a legal question?

Crawling public pages for text has decades of settled practice. Crawling them to build a biometric database of people who never consented is what the Clearview decisions in the Netherlands, France, Italy, the UK and Australia turned on, and what the EU AI Act now prohibits outright for untargeted scraping of facial images. The index is the product, and the index is the thing the law examines.