Courses

Distributed Reliability

Keep a system correct when parts of it fail: replication and the consistency it can promise, coordination and leases, transactions across services, and the recovery that gets the data back.

Islands
4
Lessons
24
Exercises
140
DebugCode orderFill blankMultiple choiceShort answerTrace

Start the first lesson

  1. Island 1

    Replication Models

    One leader, several leaders or none; the quorum arithmetic that makes a leaderless read see the last write; the conflict two leaders create and the ways to resolve it; and the guarantees a session can be given.

    1. One Leader, Several, or None5 exercises, 6 min
    2. Quorum Arithmetic5 exercises, 6 min
    3. Two Writers, One Row5 exercises, 6 min
    4. What a Session Can Be Promised5 exercises, 6 min
    5. Checkpoint: Replication ModelsCheckpoint7 exercises, 6 min
    6. Boss: The Multi-Region Profile StoreBoss8 exercises, 9 min
  2. Island 2

    Coordination and Leases

    Why a lock across machines is a lease, the fencing token that makes a stale holder harmless, how a leader is elected and what it may assume, and how far a clock can be trusted.

    1. A Lock Across Machines Is a Lease5 exercises, 6 min
    2. The Fencing Token5 exercises, 6 min
    3. Electing a Leader5 exercises, 6 min
    4. What a Clock Can Tell You5 exercises, 6 min
    5. Checkpoint: Coordination and LeasesCheckpoint7 exercises, 6 min
    6. Boss: The SchedulerBoss8 exercises, 9 min
  3. Island 3

    Transactions Across Services

    A saga of local transactions with compensations, the state machine that lets it resume after a crash, why two-phase commit is the last resort, and the anomalies a saga lets through that a transaction would not.

    1. A Saga of Local Transactions5 exercises, 6 min
    2. The Saga That Survives a Crash5 exercises, 6 min
    3. Two-Phase Commit, and Why It Is Last5 exercises, 6 min
    4. What a Saga Lets Through5 exercises, 6 min
    5. Checkpoint: Transactions Across ServicesCheckpoint7 exercises, 6 min
    6. Boss: The Trip Booking SagaBoss8 exercises, 9 min
  4. Island 4

    Backups, Recovery and Chaos

    A backup that has been restored, recovery to a point in time, a region failover that has been rehearsed, and the cascade a bulkhead stops before it reaches everything.

    1. A Backup Is a Restore That Worked5 exercises, 6 min
    2. Recovery to a Point in Time5 exercises, 6 min
    3. Failing Over a Region5 exercises, 6 min
    4. The Cascade, and the Bulkhead5 exercises, 6 min
    5. Checkpoint: Backups, Recovery and ChaosCheckpoint7 exercises, 6 min
    6. Boss: The Region FailoverBoss8 exercises, 9 min