How do search engines work? Three stages, executed continuously at absurd scale: crawl the web, index what was found, then rank the matches for your query. Critically, you are not searching the live web — you are searching a stored copy that a crawler built, which is why new pages do not appear instantly.
Stage One: Crawling
Automated programs called crawlers (or spiders) start from known pages, read them, extract every link, and add those to a queue. Then they fetch those, and so on. It is a never-ending traversal of a graph with trillions of edges.
Crawling is bounded by real constraints:
- Crawl budget — no engine can fetch everything constantly, so popular and frequently updated pages get visited more often
- robots.txt — a file site owners use to request that certain paths not be crawled
- Server load — crawlers deliberately rate-limit so they do not hammer small sites
Anyone running a website and wondering why their new page is not showing up should know this: submitting a sitemap through the search engine’s own webmaster tools is the direct way to get in the queue.
Stage Two: Indexing
Raw pages are useless for fast search, so engines build an inverted index: instead of “document → words,” it stores “word → list of documents containing it.”
That flip is what makes searching billions of pages take milliseconds. Look up a term, get a list, intersect the lists for multiple terms.
Modern indexing stores considerably more than word presence: position on the page, whether it is in a heading or bold, how often it appears, links between pages, structured data markup, page load performance, and mobile-friendliness.
Stage Three: Ranking
Finding matches is easy. Choosing the order is the hard part, and it uses hundreds of signals. The well-known families:
| Signal family | What it measures |
|---|---|
| Relevance | Does the page actually address the query? Term usage, synonyms, entity matching |
| Authority / links | How many reputable pages link to it — the original PageRank idea |
| Content quality | Depth, originality, whether it adds anything beyond what already exists |
| User experience | Load speed, mobile layout, intrusive ads, layout stability |
| Expertise signals | Author credentials, site reputation on the topic |
| Context | Your location, language, search history, device |
The original insight behind Google was using links as votes: a link from page A to page B is an endorsement, and endorsements from well-linked pages count for more. That idea was transformative in 1998, though modern ranking uses far more than links.
Why Results Differ Between People
Personalization is real but often overestimated. The largest source of difference is location — “coffee near me” obviously depends on where you are. Language and country settings matter a lot. Search history has a smaller effect than most people assume for most queries.
Search engines also classify queries. A navigational query (“facebook login”) wants one specific page. An informational query (“how do search engines work”) wants explanation. A transactional query wants somewhere to buy. The engine adjusts what it shows, sometimes serving entirely different page formats — shopping results, knowledge panels, maps.
Why Not Everything Shows Up
All of which is a fair description of how do search engines work at their edges.
- The crawler has not visited the page yet — normal for new sites, typically days to weeks
- The page is blocked by robots.txt or a noindex tag
- It is genuinely low quality or duplicative, and filtered
- The index is a sample, not a mirror — no engine holds the entire web
That last point surprises people. The index is a curated subset, biased toward pages that seem worth storing.
What Changed With AI Answers
Search engines increasingly generate a summarized answer at the top of results, synthesized from multiple sources, rather than only listing links. That is convenient and it changes the economics of publishing: if the answer appears without a click, the site that produced it gets less traffic.
It also introduces a familiar failure mode. Generated summaries can state things confidently that no source actually said, so the summary is a starting point rather than a citation — worth checking against the linked sources below it.
Keep Exploring
- What is an algorithm? — what all this runs on
- How does machine learning work? — ranking increasingly learned from data
- What is a neural network? — the modern ranking machinery
