Formatting MongoDB 4.4+ Logs

MongoDB has always output log entries as plaintext.

Starting in MongoDB 4.4, mongod / mongos instances now output all log messages in structured JSON format. This includes log output sent to the file, syslog, and stdout (standard out) log destinations, as well as the output of the getLog command.

The documentation includes Log Parsing Examples using the jq command line utility, but what if we want to tail the logs and produce a similar result as to what was present prior to the introduction of structured logging?

For the following example I’ve used the m MongoDB Version Manager to install MongoDB 4.4.6:

m 4.4.6-ent
mkdir data
mongod --dbpath data --logpath data/mongodb.log --fork

Tailing the log now (tail -n 30 data/mongodb.log) will show the structured log output that is the default in MongoDB 4.4+, however using jq we can reformat (and colourize!!!) the output using one of the following:

# Windows
tail -f data\mongodb.log | jq-win64 --compact-output -r -C ".msg |= sub(\"\\n\";\"\") | .t.\"$date\"+\" \"+.c+\" [\"+.ctx+\"] \"+.msg, .attr | select(.!=null)

# Linux/OSX
tail -f data/mongodb.log | jq --compact-output -r -C '.msg |= sub("\n";"") | .t."$date"+" "+.c+" ["+.ctx+"] "+.msg, .attr | select(.!=null)'

This makes visually consuming the logs a lot easier! Some log messages, more commonly seen with increased Logging Verbosity may contain escape sequences (ex: \n and \t):

To render these escape sequences on screen while also tailing the logs in follow (-f) mode try the following:

# Linux/OSX
stdbuf -o0 tail -f mongod.log | stdbuf -o0 jq -r -C '.msg |= sub("\n";"") | .t."$date"+" "+.c+" ["+.ctx+"] "+.msg, .attr | select(.!=null)' | sed 's/\\n/\n/g; s/\\t/\t/g'

Note the stdbuf and sed commands can be installed on OSX via brew install coreutils gnu-sed.

I personally find this a lot easier when monitoring a node’s logs while troubleshooting.

Let me know if you find this useful :)