While doing evalauation of NoSQL databases, we had a 10 node riak cluster and wanted check how a similar setup would work with mongodb. So started to setup a 10 node mongodb cluster. Since this was for initial spikes, we decided to set this up on a single machine as with the other test setup using Riak.
Before I explain how we setup 10 node mongodb ReplicaSet, let me talk about replica sets. MongoDB implements replication, providing high availability using replica sets. In a replica set, there are two or more nodes participating in an asynchronous master-slave replication. The replica-set nodes elect the master node, or primary node, among themselves and when the primary node goes down, the rest of the node elect the new primary node.
When setting up ReplicaSets we need to know the ports and node names/address on which all the mongod instances are running. In this example we are using mongodb 2.6.1, the shell script below downloads mongodb and extracts into mongodb-2-6-1 and creates 10 folders to represent the 10 nodes where data will be stored. The replicaset needs a name and this example its named as snow
#!/bin/bash -ex
workingdir=`pwd`
nodedirectory='mongo-node'
#!/bin/bash
echo "Checking if mongo is downloaded"
if [ ! -f "mongodb-osx-x86_64-2.6.1.tgz" ]; then
rm -rf mongodb-2-6-1
wget http://fastdl.mongodb.org/osx/mongodb-osx-x86_64-2.6.1.tgz
fi
echo "Unzip and Setup MongoDB"
if [ ! -d "mongodb-2-6-1" ]; then
tar -xvf mongodb-osx-x86_64-2.6.1.tgz
mv mongodb-osx-x86_64-2.6.1 mongodb-2-6-1
echo "Setting up nodes"
for i in {1..10}
do
cd $workingdir
mongo_port=`expr 10000 + $i`
nodeid=$nodedirectory$i
mkdir $nodeid
cd $nodeid
mkdir data
mkdir log
$workingdir/mongodb-2-6-1/bin/mongod --dbpath=$workingdir/$nodeid/data --logpath=$workingdir/$nodeid/log/mongo.log --fork --port=$mongo_port --replSet snow
done
fi
echo "Waiting for all nodes to initialize.."
sleep 5
$workingdir/mongodb-2-6-1/bin/mongo localhost:10001 $workingdir/initialize-replicaset.js
This script works on my mac and has been tested with OSX-10.9
Once all the nodes are started, we need to initialize the replicaset, in this javascript you will notice that three of the nodes have no votes this is because mongodb does not allow more than 7 nodes to particiapte in voting to choose the master. These 3 nodes can still be used as secondaries (slaves) and can still be used to read from.
"use admin;"
cfg = {"_id" : "snow", "members" : [
{ "_id" : 0, "host" : "localhost:10001"},
{ "_id" : 1, "host" : "localhost:10002"},
{ "_id" : 2, "host" : "localhost:10003"},
{ "_id" : 3, "host" : "localhost:10004"},
{ "_id" : 4, "host" : "localhost:10005"},
{ "_id" : 5, "host" : "localhost:10006"},
{ "_id" : 6, "host" : "localhost:10007"},
{ "_id" : 7, "host" : "localhost:10008", "votes":0},
{ "_id" : 8, "host" : "localhost:10009", "votes":0},
{ "_id" : 9, "host" : "localhost:10010", "votes":0}
]
};
rs.initiate(cfg);
print("Waiting for Replica-Set to initialize..");
var rsState = rs.status().myState;
do
{
rsState = rs.status().myState;
}
while (rsState == undefined)
printjson(rs.status());
print("Replica-Set initialized..")
This script works on my mac and has been tested with OSX-10.9
Upon initilization of the replica set, we can connect to the replicaset by connecting to a single node using mongo localhost:10001 and query the replicaset status using rs.status() here is a screen shot using the admin interface of the 10 nodes and their status along with the votes each of the nodes have.
After the testing of the replicaset is done, we can shutdown down the replicaset and clean up all the data from all the nodes using the shell script.
#!/bin/bash
echo "Stopping all mongo processes"
ps -ef |grep [m]ongo | awk '{print $2}' | xargs kill -2
echo "Cleaning up folders"
read -p "Press any key to continue...Deletes all mongo-node folders... " -n1 -s
rm -rf mongo-node*
rm -rf mongodb-*
echo "Done.."
This script works on my mac and has been tested with OSX-10.9
Setting up and testing a multi-node replicaset is easy and can be done on a single machine. Offcourse we are ignoring network issues, firewalls when crossing multiple data centers and other mayrid challenges, but we can still test how the replicaset behaves and how our code would interact with the replicaset, especially when nodes go down.