MongoDB Recipes: Change mongod’s log level

To view the current log verbosity levels, use the db.getLogComponents() method.
In the below mongod instance, the verbosity is 0 (default informational level). Here we can see that the verbosity of individual components are -1, this means the component inherits the log level of its parent.

We can configure log verbosity levels by

1. Method 1: Using mongod’s startup settings
2. Method 2: Using the logComponentVerbosity parameter
3. Method 3: Using the db.setLogLevel() method.

Method 1:

We can configure global verbosity level using: the systemLog.verbosity settings

Also we can change verbosity level of an individual log component using the systemLog.component.<name>.verbosity setting for that component.

For example we are changing the verbosity level of network component to 0.
systemLog.component.network.verbosity

Method 2:

To change log verbosity level using the logComponentVerbosity parameter, pass a document with the required verbosity settings.
Example, we are going to set the default verbosity level to 2, verbosity level of storage to 1, and the storage.journal to 0
> use admin
> db.runCommand( { setParameter: 1, logComponentVerbosity:
{
verbosity: 2,
storage: {
verbosity: 1,
journal:
{
verbosity: 0
}
}
}
} )

Method 3:

We can use the db.setLogLevel() method to update a single component’s log level.
Example:

• Change the default verbosity level to 0:
  
• Change the storage.journal log level to 4:
  

MongoDB Receipes: Change chunk size in a shared cluster

The default chunk size in a shared cluster is 64MB. If we want to change it to a smaller/larger size (the allowed range of the chunk size is between 1 and 1024 MB):

Method 1:

This method works before/after the shared cluster is initialized and is recommended method

• Login to any mongos of the shared cluster:
  
• To change the global chunk size, update the value field of chunksize in the config database, here we are changing the chunk size to 5MB:
  

Method 2:

This method works only when you initialize the cluster for the first time. After the cluster is initialized, this method will not change the chunk size of the cluster.
Start the mongos with sharding.chunkSize (using config file) or --chunkSize (command line option) and set this option to desired chunk size value.

MongoDB Recipes: Disable replica set chaining

By default MongoDB allows replica set chaining. That means it allows a secondary member to sync from another secondary. Suppose we want our secondaries only to sync from the primary not from any other secondary. In that case we can disable replica set chaining.

• Save replica set configuration in a variable:
  repl1:PRIMARY> cfg = rs.conf()

• If settings sub-document is not present in the config, then add it:
  repl1:PRIMARY> cfg.settings = {}
• Set the chainingAllowed property to false from default value true in the cfg variable
  repl1:PRIMARY> cfg.settings.chainingAllowed = false
• Set the new configuration from cfg variable
  repl1:PRIMARY> rs.reconfig(cfg)
• Check the new settings:
  repl1:PRIMARY> rs.conf()

MongoDB Recipes: How to change MongoDB’s sort buffer size

When MongoDB could not use an index obtain the sort order for a query, then it sorts the results in memory. If the sort operation consumes more than 32 megabytes, MongoDB returns an error:
"Executor error during find command: OperationFailed: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit."


As written in MongoDB document, to avoid this error we can either create an index supporting the sort operation or we can use sort() in conjunction with limit()
Also memory usage limit for sorts can be configured via the internalQueryExecMaxBlockingSortBytes parameter. In the following example I am setting the sort buffer size to 128MB:
> db.adminCommand({"setParameter": 1, "internalQueryExecMaxBlockingSortBytes" : 134217728})

Now my MongoDB will use 128MB memory for the sorts that could not use an index .

MongoDB Recipes: MongoDB point-in-time recovery using oplog

MongoDB oplog is a capped collection that records all the data change operations in the databases. As the oplog is a capped collection, so it can record database changes only for a particular period of time.
Consider the following scenario:Say our oplog can store changes for last 24 hours. Our daily backup takes place at 3AM. Suppose one user fired a drop command by mistake, and dropped a collection at 11AM. To recover the dropped collection we can restore the backup of 3AM and then we can apply the oplog and recover it till 11AM (till the drop statement).
Again suppose if our oplog size is small and it can store changes only for 4 hours. In this case, we will not be able to perform the point-in-time recovery for the above scenario. So be sure that you have a large enough oplog or you take backup of your oplog frequently (before getting overwritten).
Oplog is enabled when we start a mongodb replica set. In case of standalone mongodb instance, if we need oplog then we can start the mongod instance as master (master-slave replication) or as a single node replica set.

Hands On:

Note: I have used MongoDB 3.2.4 on Ubuntu 14.04 for this tutorial.
In this example I am going to show the recovery process in a standalone mongodb instance. Also I am going to enable oplog in this instance by starting it as a master node of master-slave replication (using master-slave replication is not recommended, and it is recommended to use replica set. As here it is just for tutorial purpose, so I am using master-slave replication).
To start MongoDB with oplog in (master-slave replication) use the  --master option.
mongod --config /etc/mongod.conf --master
I will create a collection mycoll in the database mydb with 10000 documents.

At this point we will take a backup, for backup I am going to use mongodump.

After taking backup, we will do some more changes in the database
Inserting another 10000 documents:

Update all the documents and increment the score of all the documents by 10:

Again add another 5000 documents:

We have 25,000 documents in our mycoll collection.
Now I will run a query that will remove 5000 documents.

Lets suppose the above remove command was fired by mistake and we want to recover the deleted documents.
We have monogdump backup, but that backup was taken when the collection had only 10000 documents. So this backup will not be sufficient to recover the deleted documents. So we are going to use the oplog, our saviour .
Take backup of the oplog using mongodump:MongoDB stores oplog in the local database. The collection in which the oplog is stored depends on the replication type:
1) Using master-slave type replication: oplog is stored in oplog.$main collection
2) Using replica-set: oplog is stored in oplog.rs collection
For this example we are using master-slave replication, so I am taking mongodump backup of oplog.$main collection.

I created oplogbackup directory and in that directory I took mongodump of the oplog.$main collection.
Find the recovery point:As we have our mongod running and oplog available, we can query the oplog.$main collection and find the recovery point. We executed the delete command on mydb.mycoll collection, so our query will be as follows:
> d=db.getSiblingDB("local")
> d.oplog.$main.find({"ns":"mydb.mycoll","op":"d"}).sort({$natural: 1}).limit(5)

Our aim is to find the "ts" field for first delete operation. In our case it is  "ts" : Timestamp(1460026341, 1). Note this value as  1460026341:1 and it is our recovery point, we will have to use this value in mongorestore.
Also we can run the bsondump tool to generate json file of our oplog backup and then searching the json file for the recovery point.


Recovering Database:
First I will restore the mongodump backup of the database. I am restoring the backup in a fresh mongod instance:

mongorestore command restored 10,000 documents of our mydb.mycoll collection. The remaining recovery will be done using the oplog backup.
We can replay oplog using the mongorestore command’s --oplogReplay option. To replay oplog, mongorestore command requires oplog file backup to be named as oplog.bson. So I am going to move and rename our oplog backup file oplog.$main.bson into another directory as oplog.bson.

For point-in-time recovery, mongorestore has another option --oplogLimit. This option allows us to specify timestamp (in <seconds>[:ordinal] format). --oplogLimit instructs mongorestore to include oplog entries before the provided timestamp. So restore operation will run till the provided timestamp.
So running our oplog replay:

Database recovery is completed, now lets check the recovered data

Yes, we can see all the 25,000 documents are back. So we are able to recover the 5,000 deleted documents.

Also our updated 20,000 documents with incremented score of 10, also present .
So our point in time recovery is successful

MongoDB Recipes: Securing MongoDB using x.509 Certificate

Enabling SSL x.509 certificate to authenticate the members of a replica set

Instead of using simple plain-text keyfiles, MongoDB replica set or sharded cluster members can use x.509 certificates to verify their membership.
The member certificate, used for internal authentication to verify membership to the sharded cluster or a replica set, must have the following properties (Source: https://docs.mongodb.org/manual/tutorial/configure-x509-member-authentication/#member-x-509-certificate) :

• A single Certificate Authority (CA) must issue all the x.509 certificates for the members of a sharded cluster or a replica set.
• The Distinguished Name (DN), found in the member certificate’s subject, must specify a non-empty value for at least one of the following attributes: Organization (O), the Organizational Unit (OU) or the Domain Component (DC).
• The Organization attributes (O‘s), the Organizational Unit attributes (OU‘s), and the Domain Components (DC‘s) must match those from the certificates for the other cluster members.
• Either the Common Name (CN) or one of the Subject Alternative Name (SAN) entries must match the hostname of the server, used by the other members of the cluster.
• If the certificate includes the Extended Key Usage (extendedKeyUsage) setting, the value must include clientAuth (“TLS Web Client Authentication”).

We have an existing setup of MongoDB 3.2.1 replica set running on three Ubuntu 14.04 VMs. The VMs are named as server2, server3 and server4. This mongodb replica set do not have authentication enabled. We are going to use this replica set and update it to use x.509 certificates for member authentication.
For this document I am going to use self-signed certificates, but self-signed certificates are insecure as we can’t trust the authenticity of the self-signed certificates. For our internal private setup, use we can use self-signed certificates.
First we will create our own CA (certificate authority) which will sign the certificates of our mongodb servers.
First I am going to create private key for my CA server.
openssl genrsa -out mongoPrivate.key -aes256


Next I will create the CA certificate:
openssl req -x509 -new -extensions v3_ca -key mongoPrivate.key -days 1000 -out mongo-CA-cert.crt


Our CA is ready to sign our certificates, now we will create private keys for our mongodb servers and will generate CSR (certificate signing request) files for each mongodb server. We will use the CA certificate that we created above and generate  certificates for our mongod servers using the CSR files.
We will use a single command which will create the private key and CSR for a particular server. We have to create private key and CSR in all the mongodb server nodes. So we have three servers server2, server3 and server4 and we will run this command in all the three servers.

openssl req -new -nodes -newkey rsa:2048 -keyout server2.key -out server2.csr

While generating the CSR in our 3 servers, we will keep all fields in the certificate request same, except the Common Name. We will set the Common Name as hostname of that respective server on which we run the command.
We are using the -nodes option, this will leave the private key unencrypted, so that we don’t have to type/enter/configure password while starting our MongoDB server.
Creating Private Key and CSR for server2:


Now we will sign the server2’s CSR with our CA certificate and we will generate the public certificate of server2.


Creating Private Key and CSR for server3:


Creating Private Key and CSR for server4:

Generating certificates of server3 and server4:

Once certificates for all the three servers are generated, we will copy the server certificate and CA certificates to each server.
We have to concatenate the private key and the public certificate of a server into a single .pem file.
The cat command concatenates the private key and the certificate into a PEM file suitable for MongoDB.

Changes in MongoDB configuration file:

In all the three mongodb servers we have to add the below highlighted config options. For PEMKeyFile and clusterFile we have to use the respective .pem files of that server.
Config file in server3:
net:
  port: 27017
  bindIp: 0.0.0.0
  ssl:
        mode: preferSSL
        PEMKeyFile: /mongodb/config/server3.pem
        CAFile: /mongodb/config/mongo-CA-cert.crt
        clusterFile: /mongodb/config/server3.pem






security:
  clusterAuthMode: x509

Explanation:
mode: preferSSL
Connections between servers use TLS/SSL. For incoming connections, the server accepts both TLS/SSL and non-TLS/non-SSL. For details please read https://docs.mongodb.org/manual/reference/configuration-options/#net.ssl.mode
For the timebeing we are configuring SSL connection between our servers. The clients can connect to MongoDB server without TLS/SSL.

PEMKeyFile: /mongodb/config/server3.pemThe .pem file that we created using the private key and certificate of that particular server.
Note: We created unencrypted server’s private key, so we do not have to specify the net.ssl.PEMKeyPassword option. If the private key is encrypted and if we do not specify net.ssl.PEMKeyPassword option then mongod or mongos will prompt for a passphrase.

CAFile: /mongodb/config/mongo-CA-cert.crt
This option requires a .pem file containing root certificate chain from the Certificate Authority. In our case we have only one certificate of CA, so here the path of the .crt file is specified.

clusterFile: /mongodb/config/server3.pem
This option specifies the .pem file that contains the x.509 certificate-key file for membership authentication. If we skip this option then the cluster uses the .pem file specified in the PEMKeyFile setting. In our case this file is same, so we may skip this option as our .pem file is same as the one we specified in PEMKeyFile setting.

clusterAuthMode: x509The authentication mode used for cluster authentication. Please visit the link https://docs.mongodb.org/manual/reference/configuration-options/#security.clusterAuthMode for details.

Now we are ready to start the MongoDB servers of our replica set. After starting the mongod if we check the logs, we can see lines as shown below:
2016-02-24T10:20:39.424+0530 I ACCESS   [conn8]  authenticate db: $external { authenticate: 1, mechanism: "MONGODB-X509", user: "emailAddress=pranabksharma@gmail.com,CN=server2,OU=Technologies,O=Pe-Kay,L=Mumbai,ST=Maharashtra,C=IN" }
2016-02-24T10:20:40.254+0530 I ACCESS   [conn9]  authenticate db: $external { authenticate: 1, mechanism: "MONGODB-X509", user: "emailAddress=pranabksharma@gmail.com,CN=server4,OU=Technologies,O=Pe-Kay,L=Mumbai,ST=Maharashtra,C=IN" }
Seems the x.509 authentication for our replica set members are working .
Now login to the primary of the replica set and create our initial user administrator using the localhost exception. For details of creating users you may read http://pe-kay.blogspot.in/2016/02/update-existing-mongodb-replica-set-to.html and http://pe-kay.blogspot.in/2016/02/mongodb-enable-access-control.html , I am not going to write about that in this post.

Enabling client SSL x.509 certificate authentication

Each unique x.509 client certificate corresponds to a single MongoDB user, so we have to create client x.509 certificates for all the users that we need to connect to the database. We can not use one client certificate to authenticate more than one user. First we will create the client certificates and then we will add the corresponding mongodb users.
While generating the CSR for a client, we have to remember the following points:

• A single Certificate Authority (CA) must issue the certificates for both the client and the server.
• A client x.509 certificate’s subject, which contains the Distinguished Name (DN), must differ from the certificates that we generated for our mongodb servers server2, server3 and server4. The subjects must differ with regards to at least one of the following attributes: Organization (O), the Organizational Unit (OU) or the Domain Component (DC). If we do not change any of these attributes, we will get error ("Cannot create an x.509 user with a subjectname that would be recognized as an internal cluster member.") when we add user for that client.
  In the below screenshot I tried to create user for one of the client. In this case I created the client certificate keeping all the attributes same as the Member x.509 Certificate, except the Common Name. So I got the error: 
  

So for my client certificate I am going to change the Organizational Unit attribute. For our mongodb server server2, server3 and server4 certificates, we generated CSR with Organizational Unit as Technologies, for client certificate I am going to use Organizational Unit as Technologies-Client. Also I am keeping the Common Name as the name of the user root (which going to connect to the database). All the remaining attributes in the client certificate CSR are kept same as the server certificate.

After that using the same CA as we did for our server certificates, we signed the client certificate CSR and generated our client certificate.

Then we concatenated the user’s private key and the public certificate and created the .pem file for that user.

Our client certificate is created, we have to add user in MongoDB for this certificate. To authenticate with a client certificate, we have to add a MongoDB user using the value of subject collected from the client certificate.
The subject of a certificate can be extracted using the below command:
openssl x509 -in mongokey/rootuser.pem -inform PEM -subject -nameopt RFC2253

Using the subject value emailAddress=pranabksharma@gmail.com,CN=root,OU=Technologies-Client,O=Pe-Kay,L=Mumbai,ST=Maharashtra,C=IN we will add one user in the $external database (Users created on the $external database should have credentials stored externally to MongoDB). So using the subject I created one user with root role in the admin database:
rep1:PRIMARY> db.getSiblingDB("$external").runCommand({
... createUser: "emailAddress=pranabksharma@gmail.com,CN=root,OU=Technologies-Client,O=Pe-Kay,L=Mumbai,ST=Maharashtra,C=IN",
... roles: [{role: "root", db: "admin"}]
... })

Our client certificate is created, corresponding user is created, we are ready to connect to mongodb using SSL certificate. For that first we have to change the value of ssl mode option from existing preferSSL to requireSSL in all our mongod server config files. This will make all the servers to accept only ssl connections.
net:
  port: 27017
  bindIp: 0.0.0.0
  ssl:
        mode: requireSSL
        PEMKeyFile: /mongodb/config/server4.pem
        CAFile: /mongodb/config/mongo-CA-cert.crt
If we change the mode: preferSSL to mode: requireSSL, and try to connect to mongod in usual way, we get the following error:
Error: network error while attempting to run command 'isMaster' on host '<hostname>:<port>'

Now the client authentication needs x.509 certificates rather than username and password as we change the ssl mode to requireSSL.
Lets connect to mongod using our client certificate:
mongo --ssl --sslPEMKeyFile ./mongokey/rootuser.pem --sslCAFile ./ssl/CA/mongo-CA-cert.crt --host server2
After getting connected, if we try to run any query, we get the following error:

As the user got root role and we connected to the admin database, it should be able to get the list of collections. What is the issue ???
If we run the connection status query, we can see that the authentication information is blank.
rep1:PRIMARY> db.runCommand({'connectionStatus' : 1})
{
        "authInfo" : {
                "authenticatedUsers" : [ ],
                "authenticatedUserRoles" : [ ]
        },
        "ok" : 1
}
Well when our mongo shell connects to our mongod using ssl certificate as shown above, only the connection part is completed. To have authenticated, we have to use the db.auth() method in the $external database as shown below:

rep1:PRIMARY> db.getSiblingDB("$external").auth(
...   { mechanism: "MONGODB-X509",
... user: "emailAddress=pranabksharma@gmail.com,CN=root,OU=Technologies-Client,O=Pe-Kay,L=Mumbai,ST=Maharashtra,C=IN"
... }
... )

Once we authenticated, we can run our queries and do admin tasks .
Now say we have another client certificate with subject emailAddress=pranabksharma@gmail.com,CN=reader,OU=Technologies-Client,O=Pe-Kay,L=Mumbai,ST=Maharashtra,C=IN and we connected to the database using this certificate. After getting connected we have to authenticate, so lets try to authenticate using another user(emailAddress=pranabksharma@gmail.com,CN=root,OU=Technologies-Client,O=Pe-Kay,L=Mumbai,ST=Maharashtra,C=IN) , and we will get error:
Error: Username "emailAddress=pranabksharma@gmail.com,CN=root,OU=Technologies-Client,O=Pe-Kay,L=Mumbai,ST=Maharashtra,C=IN" does not match the provided client certificate user "emailAddress=pranabksharma@gmail.com,CN=reader,OU=Technologies-Client,O=Pe-Kay,L=Mumbai,ST=Maharashtra,C=IN"

We have to authenticate with the same user, using which we connected to MongoDB server. We can’t use a different user’s certificate to connect to mongodb and later use a different user to authenticate.