Embeddings and similarity

Every face search, and every face verification system from phone unlock to airport gates, rests on the same object: the face embedding. It is the point at which a picture stops being pixels and becomes something a computer can compare at scale.

What is an embedding?

An embedding is a list of numbers, typically 128 to 512 of them, that a trained network outputs for an aligned face crop. Each number has no human meaning on its own. Together they place the face at a point in a high-dimensional space. The training objective shapes that space so that distance corresponds to identity: the same person photographed twice should land at nearby points, two different people at distant ones, regardless of lighting, expression or age.

How does a network learn to do that?

Two families of training objective dominate. Metric losses such as the triplet loss used by FaceNet (2015) take an anchor face, a second photo of the same person and a photo of someone else, and push the network to make the first pair closer than the second by a margin. Classification-style losses with angular margins, such as SphereFace, CosFace and ArcFace (2018 to 2019), train the network to classify millions of identities while forcing a gap between classes on the surface of a sphere. ArcFace's additive angular margin became the common baseline for open-source and commercial systems. Both approaches need large labelled datasets, which is where the controversies documented under datasets begin.

How is similarity calculated?

Most systems normalise embeddings to unit length and use cosine similarity: the cosine of the angle between two vectors, from 1 (identical direction) down to -1. Equivalent formulations use Euclidean distance between the normalised vectors. Engines then rescale the raw value into a 0 to 100 score for display. The rescaling is arbitrary and differs by engine, so a 90 on one service and a 90 on another are not comparable.

The two curves overlap. Where the operator places the threshold decides the trade-off: a high threshold produces few false matches but misses true ones; a low threshold finds more true matches and more strangers. In a one-to-many search against millions of faces, even a tiny per-comparison false-match rate produces look-alikes in the top results, because there are so many comparisons. That is the arithmetic behind the doppelganger effect.

What does a score not tell you?

A similarity score is not a probability that two photos show the same person, and engines that present it as one are simplifying. The probability depends on the prior: how many people in the index look like the query. For a distinctive face in a small index, a score of 80 may be near certain; for a common face type in a billion-face index, the same score may be a coincidence. NIST's evaluations report false-match rates at fixed thresholds precisely because the raw score has no absolute meaning; see accuracy, bias and testing.

Can an embedding be reversed into a face?

Not faithfully, but approximately. Research from 2019 onward has shown that a network trained to invert embeddings can produce a face that the original system scores as a match, even though it does not look much like the person to a human. This is why regulators treat embeddings as biometric data rather than as anonymous numbers: the vector can be used to recognise the person it was computed from. The GDPR definition in Article 4(14) and the Illinois BIPA definition of a biometric identifier both reach stored face geometry.