Databases · Lesson 5.1
Promoting a replica
So server A and server B send every write to Simplified's primary.
As the primary adds each change to its write-ahead log, it sends the replica a copy.
The replica adds that copy to its own log, then makes the same change.
So it has a copy of every row, and takes the reads, but refuses every write.
But what happens when the primary itself dies?
Say, to keep it simple, Simplified runs a primary and just one replica.
And say the primary dies.
Reads still work, because the replica still has a copy of every row.
So server B can still load Ada's progress from it.
But writes fail, because only the primary takes writes, and the replica is read-only.
So Ada can't finish lesson 3: server A can't save her 10 XP.
So how does Simplified take writes again?
An operator, a person who looks after Simplified's database servers, steps in.
They make the replica stop being read-only, so it can take writes.
This is promoting the replica: it becomes the new primary.
Then the operator points server A and server B at the new primary.
So Ada finishes lesson 3, and her 130 XP is saved there.
This is called failover.
Failover means promoting a replica and pointing server A and server B at it, so writes work again.
When the primary dies, failover turns a replica into the new primary, so writes work again.
Simplified has a primary and one replica, and Dee has 300 XP on both. The primary dies. An operator promotes the replica, and points server A and server B at it. Then server B sends UPDATE users SET xp = xp + 10 WHERE id = 4; to it.
What does the new primary send back?
Explanation
First, promoting the replica made it stop being read-only, so it now takes writes.
Then, it runs the UPDATE, and Dee's XP goes from 300 to 310.
Therefore, the new primary sends back UPDATE 1, not the read-only error.
Simplified's primary died, and an operator promoted its replica, then pointed server A and server B at it.
Where does server A send Ada's next write?
Databases · Lesson 5.2
The commit that never arrived
So when the primary dies, an operator promotes the replica, and it takes the writes.
And with asynchronous replication, the primary replied COMMIT before the replica had the change.
But what if the primary dies before that change reaches the replica?
Say the replication lag is 50 milliseconds.
Ada's COMMIT reaches the primary's log at 0 milliseconds, and it replies COMMIT.
So Ada sees her tick and her 130 XP.
At 20 milliseconds, the primary dies, before her change has reached the replica.
So the replica still has Ada at 120.
Then an operator promotes the replica, so it's the new primary.
Later, Ada opens her dashboard, which reads her XP from the new primary, and gets 120.
Her 130 is gone, though Postgres replied COMMIT.
But isn't COMMIT the line, after which every change is kept?
On one machine it is: its write-ahead log on disk keeps the change.
Here, the log that has Ada's 130 is on the old primary, the machine that died.
The new primary's log never got that entry.
So nothing on it can put Ada's 130 back.
With asynchronous replication, you risk a failover losing a change Postgres already replied COMMIT to.
Simplified uses asynchronous replication, and its replication lag is 50 milliseconds. Ben has 40 XP. The COMMIT for his lesson, worth 10 XP, reaches the primary's log at 0 milliseconds, and the primary replies COMMIT. At 30 milliseconds, the primary dies, and an operator promotes the replica.
What XP does the new primary have for Ben?
Explanation
First, the change takes 50 milliseconds to reach the replica, and the primary died at 30 milliseconds.
Then, the change never reached the replica, so its log never got the entry.
Therefore, the new primary has 40 XP for Ben: his COMMIT is lost.
Simplified uses asynchronous replication, and its replication lag is 50 milliseconds. Dee has 300 XP. The COMMIT for her lesson, worth 20 XP, reaches the primary's log at 0 milliseconds, and the primary replies COMMIT. At 70 milliseconds, the primary dies, and an operator promotes the replica.
How much XP does the new primary have for Dee?
Simplified uses asynchronous replication, and its replication lag is 50 milliseconds. Eli has 40 XP. The COMMIT for one lesson, worth 10 XP, reaches the primary at 0 milliseconds, and the COMMIT for another, worth 20 XP, at 40 milliseconds. Postgres replies COMMIT to both. At 60 milliseconds, the primary dies, and an operator promotes the replica.
How much XP does the new primary have for Eli?
Simplified uses asynchronous replication. Postgres replied COMMIT to Ada's 130 XP, then the primary died before the change reached the replica, and the promoted replica has 120. COMMIT is the line on one database server.
Why isn't it here?
Simplified uses asynchronous replication. Its primary dies, and an operator promotes the replica.
Which of these changes is sure to be on the new primary?
Say a small app runs one database server, with no replicas. Postgres replies COMMIT to Ada's change from 120 to 130 XP. A moment later, the database server crashes, then restarts.
What does it have for Ada?
Simplified uses asynchronous replication.
When can a failover lose a change Postgres already replied COMMIT to?
Databases · Lesson 5.3
Losing nothing in a failover
So if the primary dies at 20 milliseconds, Ada's 130 never reaches the replica, and failover loses it.
Because the primary replies COMMIT without waiting for the replica.
So how can Simplified make sure no COMMIT is lost in a failover?
The primary can wait for the replica.
First it adds Ada's change and the COMMIT to its own write-ahead log, and sends the replica a copy.
The replica adds them to its own write-ahead log, on its disk.
Then the replica tells the primary it has the COMMIT.
Only then does the primary reply COMMIT to server A.
This is called synchronous replication.
Synchronous replication means the primary replies COMMIT only once a replica has the COMMIT in its write-ahead log, on disk.
So what happens now, if the primary dies just after its COMMIT?
Say the primary dies at 20 milliseconds again, just after replying COMMIT.
The replica already has Ada's change and the COMMIT in its write-ahead log.
So once it's promoted, Ada still has 130.
With synchronous replication, a failover loses nothing Postgres replied COMMIT to.
Simplified uses synchronous replication. Ben has 40 XP on the primary and its replica. The primary replies COMMIT to Ben's lesson, worth 10 XP. Then the primary dies, and an operator promotes the replica.
What XP does the new primary have for Ben?
Explanation
First, with synchronous replication, the primary replied COMMIT only once the replica had the COMMIT in its write-ahead log.
Then, the replica had Ben's change before the primary died, so it has 50.
Therefore, the new primary has 50 XP for Ben: nothing Postgres replied COMMIT to is lost.
Simplified uses synchronous replication. Eli has 40 XP on the primary and its replica. The primary replies COMMIT to Eli's lesson worth 10 XP. Then it takes his next lesson, worth 20 XP: it adds that change and the COMMIT to its own log. It dies before its copy of the second change reaches the replica. An operator promotes the replica.
How much XP does the new primary have for Eli?
Simplified uses synchronous replication. The primary adds Dee's change and the COMMIT to its own write-ahead log, and sends the replica a copy of each entry. Before the replica tells it that it has the COMMIT, the primary dies.
Did server A get a COMMIT reply for Dee's change?
Simplified's primary uses synchronous replication.
What must happen before it replies COMMIT to server A?
Simplified's replication lag is 50 milliseconds. Postgres replies COMMIT to Ada's 130 XP, and the primary dies 20 milliseconds later. An operator promotes the replica.
With which kind of replication is Ada sure to still have 130?
Simplified uses synchronous replication. The primary has added Dee's change and the COMMIT to its own write-ahead log, and sent the replica its copy. The replica hasn't answered yet.
Can Simplified's server show Dee her new XP yet?
Simplified's primary replies COMMIT only once a replica has the COMMIT in its own write-ahead log, on disk.
What is this called?
Databases · Lesson 5.4
What does waiting cost?
So with synchronous replication, the primary waits for the replica before it replies COMMIT.
And when the primary died just after Ada's COMMIT, her 130 was already safe on the replica.
But what does that wait cost every commit?
Every commit now waits for a message to reach the replica, and for its answer to come back.
That time is called a round trip.
A round trip is the time for a message to reach another machine and come back.
Say the COMMIT reaches the primary's own disk in 1 millisecond, and a round trip takes 5.
So an asynchronous commit takes 1 millisecond.
And a synchronous one takes about 6 milliseconds: 1 on disk, plus the 5-millisecond round trip.
So which should an app pick?
Say Venmo keeps balances on a primary and a replica.
Losing a payment Venmo already showed as sent would be very bad.
So Venmo might pay the 5 extra milliseconds, with synchronous replication.
Simplified's XP is different: losing a rare award of 10 XP is a small cost.
So Simplified takes that risk, in return for 1-millisecond commits instead of about 6.
Every synchronous commit waits a round trip, so pick it where losing a COMMIT is unacceptable.
Say a COMMIT takes 2 milliseconds to reach Simplified's primary's own disk, and a round trip to its replica takes 4 milliseconds. Simplified uses synchronous replication.
Roughly how long does a synchronous commit take?
Explanation
First, the COMMIT reaches the primary's own disk: 2 milliseconds.
Then, the primary waits one round trip for the replica's answer: 4 milliseconds more.
Therefore, a synchronous commit takes about 6 milliseconds, against 2 for an asynchronous one.
Say a COMMIT takes 2 milliseconds to reach the primary's own disk, and a round trip to its replica, there and back, takes 8 milliseconds.
Roughly how many milliseconds longer does a synchronous commit take than an asynchronous one?
Say Simplified's replica is in another city, so a round trip to it takes 40 milliseconds. A COMMIT takes 1 millisecond to reach the primary's own disk.
With asynchronous replication, about how many milliseconds does a commit take?
Simplified uses synchronous replication.
Why does every commit now wait one round trip?
Say Venmo keeps balances on a primary and a replica. A payment Venmo has shown as sent must never be lost, even if the primary dies.
Which replication should it pick, and why?
A Simplified commit takes 1 millisecond with asynchronous replication, and about 6 with synchronous.
Why might Simplified keep asynchronous replication for XP?
With synchronous replication, each commit waits for a message to reach the replica and for its answer to come back.
What is that time called?