Standard Logs and their grok patterns

Apache Log:

• Sample log: Refer Here for the whole file

83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"

• For apache log format Refer Here

83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"

83.149.9.216 => host
17/May/2015:10:05:03 +0000 => timestamp
METHOD => GET
/presentations/logstash-monitorama-2013/images/kibana-search.png => PATH
HTTP/1.1 => http version
200 => Status
203023 => size
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36 => client info (User Agent)
http://semicomplete.com/presentations/logstash-monitorama-2013/" => Referer

• Now lets build a pipeline which reads logs from stdin and shows all the necessary fields on the stdout and filter to create fields using GROK

input
{
    stdin {}
}
filter
{
    grok
    {
        match => {
            "message" => "%{COMBINEDAPACHELOG}"
        }
    }

}
output
{
   stdout {}
}

* Note: Refer Here for the logs of various applications

SSH Log

• Sample log

Dec 10 06:55:46 LabSZ sshd[24200]: reverse mapping checking getaddrinfo for ns.marryaldkfaczcz.com [173.234.31.186] failed - POSSIBLE BREAK-IN ATTEMPT!

• Refer Here for the log file

%{MONTH:month}%{SPACE}%{MONTHDAY:day}%{SPACE}%{TIME:time}%{SPACE}%{WORD:host}%{SPACE}%{WORD}(?:\[)%{NUMBER:pid}(?:...)%{GREEDYDATA:message}