Glossary · Search
Crawler
A crawler is the program that fetches web pages, follows links and collects the images on them so a search engine can index them. Face search engines run crawlers like text search engines do, but keep only images in which a face is detected. Crawlers can only reach pages that are publicly accessible, and site owners can tell them to stay away with robots rules.
Updated November 6, 2026 · 2 min read
How does a crawler work?
It starts from a list of known URLs, downloads each page, extracts links and image addresses, and queues them. Pages are revisited on a schedule that depends on how often they change. For a face search engine the images are the point: each one is run through face detection and the faces, with their source page address, go into the index.
What stops a crawler?
| Barrier | Effect |
|---|---|
| Login walls | Private profiles, messages and members-only content are invisible |
| robots.txt | A site's instruction to crawlers; well-behaved crawlers obey it, not all do |
| Rate limits and blocking | Sites can refuse or throttle crawlers by IP or user agent |
| JavaScript-only content | Images loaded by scripts may be missed by simpler crawlers |
| Scale | No crawler covers the whole web; coverage is always partial |
Do face search engines respect robots rules?
Policies differ and are not independently audited. Some engines state they index only pages open to any visitor; several platforms assert that scraping their public pages breaches their terms, a claim tested in the cases against Clearview AI. The EU AI Act now prohibits untargeted scraping of facial images to build recognition databases, which targets the crawl itself rather than the search.
Where does it go wrong?
- Stale results: a page changed after the last crawl still shows the old image.
- Gaps: small sites and new pages may never be crawled.
- Over-reach: a crawler that ignores robots rules collects what site owners meant to withhold.
- Cost: crawling and embedding billions of images is expensive, which is why index size is a competitive secret.
How do you keep your own pages out?
A robots rule, a login wall, or a noindex header keeps compliant crawlers away. None of them removes what was already crawled; that needs the engine's opt-out, listed in the opt-out routes directory.
Questions
Can a crawler see my private Instagram?
No. Private accounts require a login, and crawlers do not log in. Public profiles are a different matter and depend on the platform's rules.
Does blocking crawlers remove old results?
No. It prevents new crawls. Existing index entries persist until the engine recrawls or you opt out.
Sources
- 1.Google Search Central, How Search works (checked November 6, 2026)
- 2.RFC 9309, Robots Exclusion Protocol (checked September 1, 2022)
Entry reviewed November 6, 2026. Found an error? Tell us.
Related entries
- Term Index (face search) Index defined for face search: the stored set of face embeddings and source URLs built from crawled public pages, how it is refreshed, and why its scope matters.
- Term Public web Public web defined: the pages anyone can load without a login, the only part of the internet a face search engine can index, and what falls outside it.
- Term Source page Source page defined: the web page where a face search found a matching image, why it is the actionable part of a result, and how engines differ in showing it.
- Concept Building a web-scale face index How face search engines crawl the public web, store billions of embeddings for nearest-neighbour lookup, keep the index fresh, and decide what counts as public.
- Law EU Artificial Intelligence Act and the ban on scraping faces EU AI Act Article 5(1)(e): the ban on untargeted scraping of facial images for recognition databases, in force since February 2025, fines up to 7% of turnover.
Cite this entry
FaceSearch Wiki editors. "Crawler". FaceSearch Wiki, updated November 6, 2026. https://facesearch.wiki/glossary/crawler/