logstash + graylog2 – cant ask more for logging


Everyone knows how important logging is for a web application. It is a shelter we build for the rainy day. But the logs in all the application has enormous amount of valuable information, that can give interesting perspectives about the application. Also as applications grow in complexity and scale, logs become spread across multiple machines and files. Soon developers/support people will be overwhelmed by the sheer number of log files to search and track issues. In order to overcome this situation, and to have a clear, simplified oversight over the system, a central logging system becomes necessary.

Was part of a team which was responsible for building a highly distributed and complex system. Just to give a idea of the kind of complexity involved, let me give a brief summary of tech components. The project was developed using three languages (1 dynamic and 2 static), uses sql and a nosql db, a queuing engine, etc.. In order to get the log consolidated and have a central view, we evaluated splunk. Unfortunately it was extremely costly. So we were forced to look at open source alternatives and finally we selected logstash and graylog.

Looking back at that decision, was one of the great decisions. I will share some insights into how logstash and graylog2 could be configured.

First we need to download the logstash jar to each of the machines where log files are generated.
Then we need to configure the input file for logstash to process. its pretty simple. we need to provide the path to files and also group them under different types.

input {
  file {
    type => nginx_web
    path => ["/opt/nginx/logs/access.log", "/opt/nginx/logs/error.log"]
  }
}

Graylog uses a log format called Graylog Extended Log Format(GELF). By using this format, we can send custom fields with the log. This format also doesnot have a log size limitation. We can preprocess the logs and define custom fields based on the line logged, before they are sent to the graylog server. Below is a example of nginx log being parsed using grok patterns(regular expressions) and fields are defined with the values matched. Grok patterns are extremely powerful, so they need a separate detailed blog… but thats for later.

filter {
 grok {
   type => nginx_web
   pattern => "%{IP:clientip} (?:%{HOST:clienthost}|-) (?:%{USER:clientuser}|-) \[%{HTTPDATE:time}\] \"(?:%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}|%{DATA:unparsedrq})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QUOTEDSTRING:httpreferrer} %{QUOTEDSTRING:httpuseragent}"
 }
}

The corresponding nginx log format is:

log_format  main  '$remote_addr $host $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $ssl_cipher $request_time';

Above grok filter will parse each log entry and fill the values for the following fields clientip, clienthost, clientuser, time, verb, request, httpversion, unparsedrq, response, bytes, httpreferrer, httpuseragent.

The advantage of splitting the input log entry into multiple fields, is that custom rules to search, group can be done on each of these fields in the graylog2 web.
Sample grok pattern for the rails 3 project is:

 grok {
   type => app
   pattern => "\[%{WORD:requesttoken}\] \[%{IP:clientip}\] (?:%{DATA:logstmt}|-)"
 }
 grok {
   type => app
   pattern => "Rendered %{DATA:renderedtemplate} \(%{DATA:rendertime}\)"
 }

The above grok filter parses the rails 3 app log and defines the following fields: requesttoken, clientip, logstmt, renderedtemplate, rendertime.

Next we need to configure the output. This is where we specify the output format as gelf and the graylog server to contact.
We can also specify custom fields. Here we are sending a custom field named environment with value as ‘uat’.
Another predefined field in gelf format is ‘facility’ and here we are setting it with the value of field ‘type’.

output {
 gelf {
 host => "graylog-server-ip"
 facility => "%{@type}"
 custom_fields => ["environment", "uat"]
 }
}

A complete sample configuration for logstash agent is:

input {
  file {
    type => pg91
    path => ["/var/lib/pgsql/9.1/pgstartup.log", "/var/lib/pgsql/9.1/data/pg_log/*.log"]
  }
  file {
    type => app
    path => ["/app/employeemgmt/shared/log/uat.log"]
  }
  file {
    type => nginx_web
    path => ["/opt/nginx/logs/access.log", "/opt/nginx/logs/error.log"]
  }
}

filter {
 grok {
   type => nginx_web
   pattern => "%{IP:clientip} (?:%{HOST:clienthost}|-) (?:%{USER:clientuser}|-) \[%{HTTPDATE:time}\] \"(?:%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}|%{DATA:unparsedrq})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QUOTEDSTRING:httpreferrer} %{QUOTEDSTRING:httpuseragent}"
 }
 grok {
   type => app
   pattern => "\[%{WORD:requesttoken}\] \[%{IP:clientip}\] (?:%{DATA:logstmt}|-)"
 }
 grok {
   type => app
   pattern => "Rendered %{DATA:renderedtemplate} \(%{DATA:rendertime}\)"
 }
}

output {
 gelf {
 host => "graylog-server-ip"
 facility => "%{@type}"
 custom_fields => ["environment", "uat"]
 }
}

All these configuration need to be done in a conf file.
Once the conf file is ready, we need to install the graylog2 server. Installation is pretty simple, again download the server jar file and run it. This graylog2 wiki will help you with that.
Finally we need to start the logstash agent using:

java -jar logstash-1.1.0-monolithic.jar agent -f logstash-agent.conf

Enjoy all your logs are in central Graylog2 server. Watch out for the next post on graylog2 web interface and how we can use it.

About these ads

About boojapathy

A software developer who is passionate about coding, language internals...
This entry was posted in Technology and tagged , , , , , , , . Bookmark the permalink.

9 Responses to logstash + graylog2 – cant ask more for logging

  1. Nice article, concisely describes how to use logstash grok stuff. I was wondering how large your log volume is. And if you had any issues setting up elasticsearch to handle the volume. We have a stupid amount of log volume and had issues getting elasticsearch to handle it without building an elasticsearch cluster dedicated to logs.

    • boojapathy says:

      Elasticsearch occupies a pretty large space in order to store these logs. unfortunately the most easy solution is to build a elasticsearch cluster. But also you can look at logstash processing and leaving out unnecessary logs, for example load balancer health checks etc…

  2. Philipp says:

    can you expand your example in order to put the ssl_chiper and request_time in additional fields too?

    my logformat looks like this:
    ‘$host $remote_addr – $remote_user [$time_local] “$request” ‘
    ‘$status $body_bytes_sent “$http_referer” ‘
    ‘”$http_user_agent” “$http_x_forwarded_for” ‘
    ‘$ssl_cipher $request_time ‘
    ‘$gzip_ratio $upstream_addr $upstream_response_time';

    I am not able to catch any field after http_user_agent :/

    This is my logstash filter
    “%{IPORHOST:host} %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \”%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\” %{NUMBER:response} (?:%{NUMBER:bytes}|-) \”(?:%{URI:referrer}|-)\” %{QS:agent} %{QS:x_forwarded_for} %{USER:ssl_chiper} %{NUMBER:request_time} %{NUMBER:gzip_rato} %{IPORHOST:upstream} %{NUMBER:upstream_request_time}”

    • boojapathy says:

      Based on my understanding from the above log format, the “%{USER:ssl_chiper}” has to be replaced by “%{QUOTEDSTRING:ssl_chiper}”. In case, you are still having issues, can you post a sample line from the log file for which it is failing.

  3. Phil says:

    I guess my question is why both? They are both more than capable independently, correct? What are the advantages?

  4. Sharmith says:

    How to create a custom field and fill it with dynamic data from the log message.
    Sample log message given below. I want to add one field for “client IP” filled with client IP address, “Event ID” filled with event ID number in the below example “675”, “Username” filled with Username, “Service Name” filled with service name from the log.
    Your help on this is highly appreciated.
    Log:
    MSWinEventLog 1 Security 15596139 Mon Aug 06 11:21:48 2012 675 Security SYSTEM User Failure Audit XXXXX1 Account Logon Pre-authentication failed: User Name: xxxxxxxxx User ID: %{S-1-5-21-1058243859-2292328430-792808483-12929} Service Name: krbtgt/XXX Pre-Authentication Type: 0x0 Failure Code: 0x19 Client Address: 10.X.X.X 15534664

  5. Amit says:

    Nice Post. One question though – Why do you need both – graylog2 and logstash? Don’t both provide the same feature set? Parsing log files and breaking the log statements into custom fields can also be done through logstash. If you can elaborate the need for each of them?

  6. Hi, we are looking for a log aggregation solution for our system. Would you still recommend logstash and greylog2 combo as the way to go?

    • boojapathy says:

      Hi

      If your predominant use is for debugging and issue resolution using the logs then you can go with logstash + graylog2. There are direct gelf sdks available but if the system pumps in a lot of log messages, we have seen impact in terms of performance. Having said that Gelf provides capability for much more contextual and understandable logging without worrying about regular expressions.

      If you are more interested in analytics and graphs to visualise patterns, then logstash + elasticsearch + kibana is another option.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s