How do I find Drupal messages with the syslog module enabled?

At Code Enigma we always recommend disabling the Drupal database logging (dblog) module and enabling the system logging (syslog) module. We do this because dblog means lots of write queries to the database, all the time, and write queries are always expensive, plus you always want to keep as much traffic away from the database as possible because it's the hardest part of the LAMP stack to scale. Using syslog makes Drupal write messages to the system log instead, protecting the database from a flood of Drupal log entries and also providing much faster performance.

On Windows systems the syslog module logs directly into the Windows Server Event Log, so you don't lose much in user experience - you just have to know to go there to see Drupal log messages. For *nix-like systems (Apple OSX, Linux, BSD, etc.) things are a little trickier. The main barrier to using syslog is people not knowing how to search and manipulate Drupal messages with it enabled, and the purpose of this FAQ is to show people how to do just that.

Because this is an FAQ about Code Enigma systems, we are going to look specifically at where Drupal messages are stored in a Debian (our preferred server operating system) so you should read the syslog documentation on Drupal.org for further information about other systems. As a starting point you should already be familiar with our other more general FAQ on manipulating logs, which includes instruction on logging into servers and searching files for information.

At the moment Code Enigma mostly uses Debian "Wheezy" 7, and Drupal messages get sent to this file:

/var/log/syslog

Now this is the general system log, so there is a lot of stuff in there we don't care about. So how do we get Drupal-specific information out? Let's start by looking at a Drupal entry in syslog:

Feb  2 09:56:11 stage2 drupal: http://codeenigma_new.stage.stage2.codeenigma.com|1454406971|cron|127.0.0.1|http://codeenigma_new.stage.stage2.codeenigma.com/||0||Cron run completed.

That's the stage version of this website running its automated cron tasks. We can immediately see there are some useful patterns we can exploit here. For example, to get all Drupal messages from syslog we could do this:

sudo grep " drupal: " /var/log/syslog

And going further, to get all Drupal messages for the site with $base_url set to http://codeenigma_new.stage.stage2.codeenigma.com we could do this:

sudo grep " drupal: http://codeenigma_new.stage.stage2.codeenigma.com|" /var/log/syslog

And what about specific services? For example, to get all the cron messages with dblog I have a nice interface and I can just select it like this:

Cron in dblog

So with syslog? Still easy, I can do something like this:

sudo grep "|cron|" /var/log/syslog

"Ah ha, but I've got multiple sites on my server!" I hear you cry. "How do I do this for a single site?" Well, with grep you have all the power of regular expressions at your fingertips, so while it might be a steeper learning curve, it's actually way more powerful than the Windows Event Log or the Drupal admin page. Let's look at that specific example:

sudo grep -E "drupal:(.*)codeenigma_new\.stage\.stage2\.codeenigma\.com(.*)\|cron\|" /var/log/syslog

What we're basically saying there is match "drupal." followed by anything "(.*)" followed by codeenigma_new.stage.stage2.codeenigma.com followed again by anything (.*) followed finally by "|cron|". Note the dots and the pipes have been preceded with a forward slash, the regex escape character. Digital Ocean actually have a pretty good tutorial about regex and grep.

Following this pattern, I could go further. For example, show me all the user and cron events for this same site:

sudo grep -E "drupal:(.*)codeenigma_new\.stage\.stage2\.codeenigma\.com(.*)(\|cron\||\|user\|)" /var/log/syslog

Let's disect (\|cron\||\|user\|). You start with "(", this is an expression, then "\|cron\|" which matches the string |cron| - remember what we said about escaping pipes? - then another pipe symbol, "|" which denotes or, then "\|user\|" matching the string |user| and finally, ")", end of expression.

Start an expression, search for |cron| or |user|, end of expression. (\|cron\||\|user\|) in a nutshell!

Hopefully this FAQ gives you enough signposts and information to be confident in turning on the syslog module and moving away from dblog, knowing you can still find Drupal information if you need it. Happy grepping!