AlgoFoxGet AlgoFox App

CoursesSystem Design Case Studies

The Search Engine

Search over a billion documents: an index sharded by document with a query fanned out and merged, an indexing pipeline that keeps up with the web's changes, autocomplete inside a keystroke, and the whole design under a query that half the world types at once.

Lessons
5
Exercises
29
Minutes
34

Start An Index Too Big for One Machine

  1. 1

    An Index Too Big for One Machine

    After this lesson you can shard an inverted index by document, fan a query out to every shard and merge the top results, and say why sharding by term is the tempting wrong answer for most queries.

    DebugMultiple choiceShort answerTrace
    5 exercises
    6 min
  2. 2

    Keeping Up With a Billion Pages

    After this lesson you can lay out a crawler that is polite per site and fast overall, decide how often to re-crawl by change rate, and build the index in batches that swap in without a pause.

    DebugMultiple choiceShort answer
    5 exercises
    6 min
  3. 3

    Autocomplete Inside a Keystroke

    After this lesson you can serve query suggestions in a few milliseconds from a prefix structure built offline, keep it fresh for trending queries, and keep it from suggesting what it should not.

    DebugMultiple choiceShort answer
    5 exercises
    6 min
  4. 4

    Checkpoint: Search VariationsCheckpoint

    The same design under different numbers: an internal wiki, an app-store search, and a log search.

    DebugMultiple choiceShort answer
    6 exercises
    6 min
  5. 5

    Boss: The Search EngineBoss

    The whole design: the question, the numbers, the shards and replicas, the pipeline, the hot query, the shard that vanished, the revised serving path, and the tradeoff you defend.

    Code orderMultiple choiceShort answerTrace
    8 exercises
    10 min