Standard Logs and their grok patterns
Apache Log:
- Sample log: Refer Here for the whole file
83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
- For apache log format Refer Here
83.149.9.216 - - [17/May/2015:10:05:03 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 => host
17/May/2015:10:05:03 +0000 => timestamp
METHOD => GET
/presentations/logstash-monitorama-2013/images/kibana-search.png => PATH
HTTP/1.1 => http version
200 => Status
203023 => size
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36 => client info (User Agent)
http://semicomplete.com/presentations/logstash-monitorama-2013/" => Referer
- Now lets build a pipeline which reads logs from stdin and shows all the necessary fields on the stdout and filter to create fields using GROK
input
{
stdin {}
}
filter
{
grok
{
match => {
"message" => "%{COMBINEDAPACHELOG}"
}
}
}
output
{
stdout {}
}
* Note: Refer Here for the logs of various applications
SSH Log
- Sample log
Dec 10 06:55:46 LabSZ sshd[24200]: reverse mapping checking getaddrinfo for ns.marryaldkfaczcz.com [173.234.31.186] failed - POSSIBLE BREAK-IN ATTEMPT!
- Refer Here for the log file
%{MONTH:month}%{SPACE}%{MONTHDAY:day}%{SPACE}%{TIME:time}%{SPACE}%{WORD:host}%{SPACE}%{WORD}(?:\[)%{NUMBER:pid}(?:...)%{GREEDYDATA:message}
