Concept · Introductory
Face detection and alignment
Face detection finds the faces in an image and returns a box around each one, usually with the positions of the eyes, nose and mouth. Alignment then rotates, scales and crops each face so that those landmarks sit in standard positions. Both steps happen before any comparison, and most search failures that users blame on the matching model begin here.
Updated November 6, 2026 · 4 min read · reviewed November 6, 2026
Before a face can be compared with any other, software has to decide that there is a face in the picture and where it is. That is face detection. Alignment follows: the face is straightened and cropped to a fixed frame so that the embedding model always sees eyes, nose and mouth in roughly the same place. Together they turn an arbitrary photograph into the standard input a recognition model expects.
- Step 1
Scan
The detector slides over the image at several scales looking for face-like patterns.
- Step 2
Box
Each candidate gets a bounding box and a confidence score; weak candidates are dropped.
- Step 3
Landmarks
Five or more points are placed: eye centres, nose tip, mouth corners.
- Step 4
Align
A similarity transform rotates and scales the face so the landmarks match a template.
- Step 5
Crop
The aligned face is cut to the model's input size, often 112 by 112 pixels.
How do detectors work?
Early detectors, such as the Viola-Jones cascade of 2001, matched hand-designed brightness patterns and worked only on near-frontal faces. Modern detectors are convolutional networks trained on datasets of annotated faces (WIDER FACE is the standard benchmark) and locate faces across poses, scales and partial occlusion. MTCNN (2016) and RetinaFace (2019) are the designs most engines and open-source toolkits derive from. RetinaFace reports landmark positions alongside each box, which is what alignment needs.
Why does alignment matter so much?
Embedding models are trained on aligned faces. Feed one a tilted or off-centre crop and the numbers it produces drift, so a genuine match scores lower and a look-alike may score higher. Alignment removes rotation and scale so that the model only has to cope with identity, expression, lighting and age. It cannot remove pose: a face turned 60 degrees still shows the model half the information of a frontal one, which is why profile shots match poorly.
What makes a face undetectable?
| Property | Effect | Practical threshold |
|---|---|---|
| Face size | Below a minimum pixel size the detector finds nothing | About 40 to 60 pixels between the eyes for reliable detection |
| Pose | Yaw beyond roughly 45 degrees halves usable features | Frontal to three-quarter view works; full profile often fails |
| Occlusion | Sunglasses, masks and hands hide landmarks | Eyes and mouth visible is the usual requirement |
| Blur and compression | Edges vanish; landmarks become uncertain | Screenshots of screenshots are the common failure |
| Lighting | Strong backlight or shadow removes contrast | Even, frontal light is best |
| Multiple faces | All are detected; the engine must choose one | Crop to the face of interest before uploading |
What are landmarks used for beyond alignment?
The same landmarks support quality checks (is the face frontal enough to search), liveness detection in verification systems (does the face move like a live person), and deduplication (two detections of the same face in one image). Some engines reject uploads whose landmarks indicate a child's face proportions; FaceCheck.ID states such a rule, though no engine publishes its method.
How accurate are detectors?
- Viola-Jones cascade (2001) 20 % Approximate; the method predates the benchmark and fails on hard faces
- MTCNN (2016) 60 % As reported in later comparisons
- RetinaFace (2019) 91 % 91.4% AP, hard subset, RetinaFace paper
Source: Deng et al., RetinaFace (2019), Table 4; earlier figures from the WIDER FACE leaderboard as summarised in that paper
The hard subset of WIDER FACE contains tiny, crowded and occluded faces; on ordinary portraits every modern detector is close to perfect. The failures users meet in practice come from the thresholds above, not from the detector's design.
What should you do before uploading a photo?
- Crop so that one face fills most of the frame.
- Prefer the original image over a screenshot, and a screenshot over a photo of a screen.
- Choose a frontal or three-quarter view with both eyes visible.
- Avoid heavy filters; they change the texture the model relies on.
Questions
Why did the engine pick the wrong face in a group photo?
Detectors return every face; the engine then selects the largest or most confident one unless you choose. Crop to the face you want first.
Does alignment change my photo?
Only the copy used for the search. The face is rotated and cropped in memory before the embedding is computed.
Can a detector find a face in a drawing or a statue?
Often yes, since it looks for face-like patterns. The embedding step decides whether it matches anyone.
Why does a group photo return nothing?
A detector returns every face it finds, but a search engine usually takes the largest one or asks you to choose. Small faces in a crowd fall below the detector's size threshold, and the chosen face may not be the one you meant. Crop to one face before uploading.
Does alignment change the photo I uploaded?
Only a working copy. The cropped, rotated and scaled face is what the embedding model sees; the uploaded file itself is not altered by this step. Engines that strip metadata or re-encode uploads do that separately, before detection.
Sources
- 1.Deng et al., RetinaFace: Single-stage Dense Face Localisation in the Wild (2019) (checked May 2, 2019)
- 2.Zhang et al., Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks (2016) (checked April 11, 2016)
- 3.Yang et al., WIDER FACE: A Face Detection Benchmark (2016) (checked November 20, 2015)
Entry reviewed November 6, 2026. Found an error? Tell us.
Related entries
- Concept How a face search works The face search pipeline step by step: detection, alignment, embedding, index lookup, ranking and source pages, with the limits at each stage.
- Concept Embeddings and similarity What a face embedding is, how networks place the same face close together, how cosine similarity becomes a score, and why a high score is not proof of identity.
- Term Face detection Face detection defined: locating faces in an image with boxes and landmarks, how it differs from recognition, common detectors, and the failures that limit a search.
- Term Liveness detection Liveness detection defined: how systems tell a live, present face from a photo, replay or deepfake, the active and passive methods, and the ISO standard.
Cite this entry
FaceSearch Wiki editors. "Face detection and alignment". FaceSearch Wiki, updated November 6, 2026. https://facesearch.wiki/concepts/face-detection-and-alignment/