Courses/System Design Case Studies/The Search Engine

Keeping Up With a Billion Pages

After this lesson you can lay out a crawler that is polite per site and fast overall, decide how often to re-crawl by change rate, and build the index in batches that swap in without a pause.