Sometimes our replica set members fall off the oplog and the node needs to be resynced. When this happens, an Initial Sync is required, which does the following:
Clones all databases except the local database. To clone, the mongod scans every collection in each source database and inserts all data into its own copies of these collections.
Applies all changes to the data set. Using the oplog from the source, the mongod updates its data set to reflect the current state of the replica set.
When the initial sync finishes, the member transitions from STARTUP2 to SECONDARY.
Some common questions when performing an initial sync of a Replica Set Member are:
How do I know if the sync is progressing?
How long will this take to complete?
Determining if the sync is progressing can be done by either checking the size of the dbPath of the syncing node or by running the db.adminCommand({ replSetGetStatus: 1, initialSync: 1 }) command while connected to the SECONDARY via the mongo shell.
Checking the directory size of the SECONDARY that is being initial sync’ed will provide a good approximation as to how much data still remains to be copied. Note that as the WiredTiger storage engine doesn’t “release” space when documents are deleted there is a high probability that the SECONDARY will have a smaller total directory size than the sync source.
The second step (after cloning) where the oplog entries are applied will also affect the overall time required to sync from the sync source.
The replSetGetStatus command will produce a JSON document similar to the following. This document contains extensive details as to how the database/collection cloning is progressing, as well as any errors that have occurred during the process.
Depending on the number of databases and collections being sync’ed, the size of this document can be quite large and difficult to visually parse.
To improve this situation I’ve created the following script.
By running this against the SECONDARY from the mongo shell, a more concise representation of the initialSyncStatus document is produced:
The script will also let you know if there have been any sync failures recorded, as well as what the last failure was.
Hopefully you’ll find this useful when the time comes to resync one of your nodes.
Comments powered by Disqus.