How to Create Highly Available MongoDB Databases with Replica Sets

In this article, we will explore in detail the various configurations of MongoDB replica sets.

The replica set is initialized through the <b>replSetInitiate</b> command (or <b>rs.initiate()</b> for mongo shell). After the initialization, various members start to send heartbeat messages and initialize the primary node election. The node that wins the majority of votes is elected as the primary node while the remaining ones become the secondary nodes.

Initialize the replica set

Suppose that the number of voting members (to be introduced later) in the replica set is N. Majority is then defined by the formula (N/2) + 1, or (N+1)/2, for even and odd values of N, respectively. If the number of members alive in the replica set is less than the majority, the replica set cannot elect the primary node, and is not able to provide write services. The replica set is in the read-only status.

Number of voting members

Majority

Number of tolerable failures

1

2

3

4

5

6

7

You are recommended to set the number of members in a replica set to an odd number for better failure tolerance. From the table above, we can see that the replica sets with two nodes and three nodes need a majority of two nodes, but only the odd-numbered set can tolerate a failed node. This means that the odd-numbered set has better service availability and can provide more reliable data storage.

In normal cases, the secondary node of the replica set will participate in the primary node election (it may also be elected as the primary node), and synchronize the data last written from the primary node to ensure its data consistency with the primary node.

The secondary node can provide the reading service. Increasing the number of secondary nodes can enhance the reading service capability of the replica set and improve the availability of the replica set. In addition, the MongoDB supports flexible configurations on the secondary nodes in the replica set to adapt to the demands of a variety of scenarios.

The arbiter node only participates in the voting. It cannot be elected as the primary node and does not synchronize data from the primary node.

For example, if you deploy a replica set with two nodes, you will have a primary node and a secondary node. If either node fails, the replica set will not be able to provide services (cannot elect the primary node). However, if you add an arbiter node to the replica set, the primary can still be elected, even if a node fails.

The arbiter node itself does not store data and is a very lightweight service. When the number of members in the replica set is an even number, you should add an arbiter node to improve the availability of the replica set.

The election priority of a Priority0 node is 0 and the Priority0 node will not be selected as the primary node.

For example, if you deploy a replica set across server rooms A and B, you can specify that the primary node must be in server room A. You can do this by setting the priority of replica set members in server room B to 0, so that the primary node must be a member in server room A. If you deploy the replica set like this, you should deploy the majority of nodes in server room A. Otherwise, the primary node may fail to be elected during network partitioning.

In MongoDB 3.0, you can set a maximum number of 50 replica set members, and a maximum number of 7 members participating in the primary node election. The vote attributes of other members (Vote0) must be set to 0, that is, they do not participate in the voting.

The hidden node cannot be selected as the primary node (its Priority is 0) and is invisible to the Driver.

Because the hidden node will not accept requests from the Driver, you can use the hidden node for data backup, offline computing, and other tasks without affecting the service provided by the replica set.

The delayed node must be a hidden node, and its data lags behind that on the primary node for some time (this is configurable, such as one hour).

Because the data on the delayed node lags behind that on the primary node, you can recover data on the primary node by using historical data on the delayed node.

For example, the oplog format below contains the ts, h, op, ns, and o fields.

ts: Operation time. The current timestamp + counter, and the counter is reset every second.

h: The global unique identifier of the operation.

v: The oplog version information.

op: Operation type.

i: Insert operation.

u: Update operation.

d: Delete operation.

c: Execute commands (such as createDatabase, and dropDatabase)

n: Null operation. It is for some special purposes.

ns: The targeted set of the operation.

o: Operation content. If it is an update operation:

o2: The operation query condition. Only the update operation contains this field.

The init sync process contains the following steps:

At T1, the secondary node synchronizes the data of all the databases on the primary node (except local) through the sensitive command combination of listDatabases + listCollections + cloneCollection. We suppose all the operations are completed at T2.

Apply all the oplogs from the period of [T1-T2] from the primary node. Some operations may have been included in Step 1. But because of the idempotence of the oplog, the oplog can be applied repeatedly.

Create indexes for corresponding sets on the secondary node according to the indexing settings of various sets on the primary node. (The _id index of every set has been completed in Step 1.)

The size of the oplog set should be reasonably configured based on the database scale and application writing requirements. If the set is too large, it will waste storage space. If the set is too small, the "init sync" operation of secondary nodes may fail. For example, if there is too much data in the database in Step 1 and the size of the oplog is too small, the oplog cannot store all the oplogs during the period of [T1, T2]. As a result, the secondary node cannot fully synchronize with the datasets from the primary node.

When you need to modify the replica set, such as adding a member, deleting a member, or modifying the member configuration (such as priority, vote, hidden, and delayed among other attributes), you can use the replSetReconfig command (rs.reconfig()) to re-configure the replica set.

For example, to set the priority of the second member in the replica set to 2, you can execute the following commands:

Apart from at the replica set initialization, the primary node election may also occur in the following scenarios:

The replica set is re-configured

The secondary node will trigger a new round of primary node election when it detects the primary node failure.

When the primary node performs an active stepDown (actively downgrade to the secondary node), a new round of primary node election will be triggered.

The primary node election is affected by multiple factors including the inter-node heartbeats, priority, and the latest oplog time.

Members in a replica set will send a heartbeat message between each other every two seconds by default. If the heartbeat message of a node is not received for 10 seconds, the node is considered to have failed. If the failed node is the primary node, the secondary node (the premise is that it can be voted as the primary node) will initiate a new round of primary node election.

Every node is inclined to vote the node with the highest priority as the primary node.

A node with the priority of 0 will not take the initiative to trigger the primary node election.

When the primary node discovers a secondary node with a higher priority, and the data latency on the secondary node is within 10 seconds, the primary node will perform an active stepDown and make the secondary node with a higher priority eligible for being the primary node.

Only the node with the latest optime (the timestamp of the most recent oplog record) can be elected as the primary node.

A node is eligible to be elected as the primary node only if it remains connected with a majority of voting nodes. If the primary node is disconnected from a majority of nodes, the primary node will take the initiative to downgrade to a secondary node. During network partitioning, multiple primary nodes may appear within a short period of time. To avoid this from happening, you should set the majority policy when you write data to the driver. This ensures that even if multiple primary nodes appear, only one primary node can successfully write data to the majority of nodes.

Primary: The default rule is that all the read requests are sent to the primary node.

PrimaryPreferred: The primary enjoys priority. If the primary node is unreachable, the requests are sent to the secondary nodes.

Secondary: All the read requests are sent to the secondary node.

SecondaryPreferred: The secondary node enjoys priority. When all the secondary nodes are unreachable, the requests are sent to the primary node.

Nearest: The read requests are sent to the nearest reachable node (detected through the ping).

For example, the write concern rule below states that the write operation must be successful on a majority of nodes and the timeout value is 5 seconds.

The setting above is for a single request. You can also modify the default write concern of the replica set so that you do not need to set it separately for every single request.

When the primary node is down and the primary node re-joins the set, if some data is not synchronized to the secondary node and there have been some write operations on the new primary node, the old primary node needs to roll back some operations to ensure the consistency of the dataset with the new primary node.