MongoDB Recipes: Update existing MongoDB replica set to use access control

The Story:I have a MongoDB replica set setup of three nodes running in a private environment. MongoDB was not configured to use access control, so anybody can connect and do anything (actually it was protected by firewall and only the application server can connect to this setup, so some level of protection was there). One  day I was informed that our analytics team needs access to this mongodb server and they should only be given read access to the pekay database. Also the sales team’s application needs read-write access to the pekay database. So I have to create users with proper roles and for that I have to enable access control in my MongoDB replica set.
In this story, my 3 node mongodb replica set is running mongodb 3.2.1 on three Ubuntu 14.04 VMs.
So to enable access control we have to restart each node with proper config options.
Normally with standalone mongodb server, we have to set the command line option  --auth or security.authorization configuration file setting. You may also read this related post http://pe-kay.blogspot.in/2016/02/mongodb-enable-access-control.html
Lets try that:
I have added the following line in my mongodb config file in all three replica set members and restarted the mongod servers.
security:
  authorization: enabled
Now after starting my mongod servers when I connect any of them, I get:

All the 3 mongod servers are in RECOVERING state and my mongod log files are getting filled with below error messages:
2016-02-22T14:43:14.400+0530 I ACCESS   [conn1] Unauthorized not authorized on admin to execute command { replSetHeartbeat: "rep1", configVersion: 3, from: "server3:27017", fromId: 1, term: 1 }
2016-02-22T14:43:16.146+0530 I ACCESS   [conn2] Unauthorized not authorized on admin to execute command { replSetHeartbeat: "rep1", configVersion: 3, from: "server2:27017", fromId: 0, term: 1 }
2016-02-22T14:43:18.341+0530 I REPL     [ReplicationExecutor] Error in heartbeat request to server2:27017; Unauthorized not authorized on admin to execute command { replSetHeartbeat: "rep1", configVersion: 3, from: "server4:27017", fromId: 2, term: 1 }
2016-02-22T14:43:18.341+0530 I REPL     [ReplicationExecutor] Error in heartbeat request to server3:27017; Unauthorized not authorized on admin to execute command { replSetHeartbeat: "rep1", configVersion: 3, from: "server4:27017", fromId: 2, term: 1 }

So all are authorization issues. What went wrong????
Well, we have to enable Internal Authentication for enabling access control on replica sets and sharded clusters. We can use keyfile or x.509 for internal authentication. In this document I am going to write about the keyfile, which is simplest to setup.
First we have to create a keyfile and the keyfile has to be present in all the nodes of my replica set. This keyfile contains a password and all the nodes should share the same password. Be sure to have a strong and long password in the keyfile. As this is my test setup, so I am choosing a simple password:
$ echo "mypassword" > /mongodb/config/key
Also you can use openssl or md5sum command to generate random password. Set the same password in all three server’s keyfile.
Change the permission of this keyfile to 600.
$ chmod 600 /mongodb/config/key
I am changing my mongod config file to set the security.keyFile option, also you can start the mongod with  --keyFile option.
Note: Enabling security.keyFile option also enables authorization: enabled option. So we do not have to set the authorization: enabled in our config file.
security:
       keyFile: /mongodb/config/key

So starting the mongod servers of our replica set with keyFile option, and this time the replica set started normally .
Now using the localhost exception, I will connect to the primary and create the initial user administrator.

Authenticate with the newly created user and create the additional users.
rep1:PRIMARY> db.auth("uadmin", "abc123");
Create a user with root role, so that using this user we can run database/cluster administration commands.
rep1:PRIMARY> db.createUser({
... user: "rootuser",
... pwd: "abc123",
... roles: [{ role: "root", db: "admin"}]
... })
Successfully added user: {
        "user" : "rootuser",
        "roles" : [
                {
                        "role" : "root",
                        "db" : "admin"
                }
        ]
}
rep1:PRIMARY>

User for analytics team and sales team on pekay database:

That’s it, our replica set is authentication enabled now