Logstash Grok Patterns with Regular expressions

• Regex Cheatsheet Refer Here
• Grok Debugger Refer Here
• Sample logs github Refer Here
• Sample grok patterns
  
  
  
• Consider the following log

Jun 14 15:16:01 combo sshd(pam_unix)[19939]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4

log_date = Jun 14 15:16:01
process_name = sshd
pid=19939
log_message= autentication failure
remote_host = 218.188.2.4

• Solution

# One
(?<log_date>%{WORD}%{SPACE}%{MONTHDAY}%{SPACE}%{TIME})%{SPACE}%{WORD}%{SPACE}%{WORD:process_name}\(%{WORD}\)\[%{NUMBER:pid}\]:%{SPACE}%{GREEDYDATA:log_message};%{GREEDYDATA}%{IP:remote_host}

# Better solution
%{SYSLOGTIMESTAMP:log_date}%{SPACE}%{WORD}%{SPACE}%{WORD:process_name}\(%{WORD}\)\[%{NUMBER:pid}\]:%{SPACE}%{GREEDYDATA:log_message};%{GREEDYDATA}%{IP:remote_host}

• Consider the following log

17/06/09 20:10:40 INFO executor.CoarseGrainedExecutorBackend: Registered signal handlers for [TERM, HUP, INT]

log_date = 17/06/09 20:10:40
level = info
class = executor.CoarseGrainedExecutorBackend
message = Registered signal handlers for [TERM, HUP, INT]

• Solution

%{DATESTAMP:log_date}%{SPACE}%{LOGLEVEL:level}%{SPACE}%{JAVAFILE:class}:%{SPACE}%{GREEDYDATA:message}

Logstash

• Lets apply the grok filters to log pipelines
• Refer Here for sample pipeline
• To make above pipeline write logs to elastic search we need to change the output plugin to elastic search and also mention index
  
• For logstash and kibana while you are creating a user consider built in roles Refer Here
• For this we have executed the following curl requests

curl -X POST -u "elastic:Y+Q=JiV1EN=0lYe9hjpc" --insecure "https://localhost:9200/_security/user/qtelastic?pretty" -H 'Content-Type: application/json' -d'
{
  "password" : "india@1234",
  "roles" : [ "kibana_admin", "logstash_admin", "ingest_admin", "machine_learning_admin", "watcher_admin", "transform_admin" ],
  "full_name" : "qtelastic",
  "email" : "qtelastic@example.com",
  "metadata" : {
    "intelligence" : 7
  }
}
'

curl -X POST -u "elastic:Y+Q=JiV1EN=0lYe9hjpc" --insecure "https://localhost:9200/_security/user/qtadmin?pretty" -H 'Content-Type: application/json' -d'
{
  "password" : "india@1234",
  "roles" : [ "superuser" ],
  "full_name" : "qtadmin",
  "email" : "qtadmin@example.com",
  "metadata" : {
    "intelligence" : 7
  }
}
'

• Lets configure logstash.yaml changed elastic search default username and password
• Refer Here for the pipeline and then lets test this
• We have sent the logs from logstash to elastic search, now lets verify in kibana
  
  
  
  
  
• configuration files should be stored in /etc/logstash/conf.d
• restart the daemon and enable logstash service.