logstash + graylog2 – cant ask more for logging

Everyone knows how important logging is for a web application. It is a shelter we build for the rainy day. But the logs in all the application has enormous amount of valuable information, that can give interesting perspectives about the application. Also as applications grow in complexity and scale, logs become spread across multiple machines and files. Soon developers/support people will be overwhelmed by the sheer number of log files to search and track issues. In order to overcome this situation, and to have a clear, simplified oversight over the system, a central logging system becomes necessary.

Was part of a team which was responsible for building a highly distributed and complex system. Just to give a idea of the kind of complexity involved, let me give a brief summary of tech components. The project was developed using three languages (1 dynamic and 2 static), uses sql and a nosql db, a queuing engine, etc.. In order to get the log consolidated and have a central view, we evaluated splunk. Unfortunately it was extremely costly. So we were forced to look at open source alternatives and finally we selected logstash and graylog.

Looking back at that decision, was one of the great decisions. I will share some insights into how logstash and graylog2 could be configured.

First we need to download the logstash jar to each of the machines where log files are generated.
Then we need to configure the input file for logstash to process. its pretty simple. we need to provide the path to files and also group them under different types.

input {
  file {
    type => nginx_web
    path => ["/opt/nginx/logs/access.log", "/opt/nginx/logs/error.log"]
  }
}

Graylog uses a log format called Graylog Extended Log Format(GELF). By using this format, we can send custom fields with the log. This format also doesnot have a log size limitation. We can preprocess the logs and define custom fields based on the line logged, before they are sent to the graylog server. Below is a example of nginx log being parsed using grok patterns(regular expressions) and fields are defined with the values matched. Grok patterns are extremely powerful, so they need a separate detailed blog… but thats for later.

filter {
 grok {
   type => nginx_web
   pattern => "%{IP:clientip} (?:%{HOST:clienthost}|-) (?:%{USER:clientuser}|-) \[%{HTTPDATE:time}\] \"(?:%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}|%{DATA:unparsedrq})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QUOTEDSTRING:httpreferrer} %{QUOTEDSTRING:httpuseragent}"
 }
}

The corresponding nginx log format is:

log_format  main  '$remote_addr $host $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $ssl_cipher $request_time';

Above grok filter will parse each log entry and fill the values for the following fields clientip, clienthost, clientuser, time, verb, request, httpversion, unparsedrq, response, bytes, httpreferrer, httpuseragent.

The advantage of splitting the input log entry into multiple fields, is that custom rules to search, group can be done on each of these fields in the graylog2 web.
Sample grok pattern for the rails 3 project is:

 grok {
   type => app
   pattern => "\[%{WORD:requesttoken}\] \[%{IP:clientip}\] (?:%{DATA:logstmt}|-)"
 }
 grok {
   type => app
   pattern => "Rendered %{DATA:renderedtemplate} \(%{DATA:rendertime}\)"
 }

The above grok filter parses the rails 3 app log and defines the following fields: requesttoken, clientip, logstmt, renderedtemplate, rendertime.

Next we need to configure the output. This is where we specify the output format as gelf and the graylog server to contact.
We can also specify custom fields. Here we are sending a custom field named environment with value as ‘uat’.
Another predefined field in gelf format is ‘facility’ and here we are setting it with the value of field ‘type’.

output {
 gelf {
 host => "graylog-server-ip"
 facility => "%{@type}"
 custom_fields => ["environment", "uat"]
 }
}

A complete sample configuration for logstash agent is:

input {
  file {
    type => pg91
    path => ["/var/lib/pgsql/9.1/pgstartup.log", "/var/lib/pgsql/9.1/data/pg_log/*.log"]
  }
  file {
    type => app
    path => ["/app/employeemgmt/shared/log/uat.log"]
  }
  file {
    type => nginx_web
    path => ["/opt/nginx/logs/access.log", "/opt/nginx/logs/error.log"]
  }
}

filter {
 grok {
   type => nginx_web
   pattern => "%{IP:clientip} (?:%{HOST:clienthost}|-) (?:%{USER:clientuser}|-) \[%{HTTPDATE:time}\] \"(?:%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}|%{DATA:unparsedrq})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QUOTEDSTRING:httpreferrer} %{QUOTEDSTRING:httpuseragent}"
 }
 grok {
   type => app
   pattern => "\[%{WORD:requesttoken}\] \[%{IP:clientip}\] (?:%{DATA:logstmt}|-)"
 }
 grok {
   type => app
   pattern => "Rendered %{DATA:renderedtemplate} \(%{DATA:rendertime}\)"
 }
}

output {
 gelf {
 host => "graylog-server-ip"
 facility => "%{@type}"
 custom_fields => ["environment", "uat"]
 }
}

All these configuration need to be done in a conf file.
Once the conf file is ready, we need to install the graylog2 server. Installation is pretty simple, again download the server jar file and run it. This graylog2 wiki will help you with that.
Finally we need to start the logstash agent using:

java -jar logstash-1.1.0-monolithic.jar agent -f logstash-agent.conf

Enjoy all your logs are in central Graylog2 server. Watch out for the next post on graylog2 web interface and how we can use it.