A web crawler (or spider) is a program that systematically traverses the web by downloading pages and following hyperlinks from one page to the next. Web crawlers power search engine indexing, dataset collection for Large Language Model (LLM) training, content monitoring, and archival projects. The core challenge is doing this at internet scale, across billions of pages, while being polite to site operators and resilient to the messy, failure-prone nature of the open web.