DevOps Classroomnotes 18/Dec/2022

Logstash Grok Patterns with Regular expressions

Jun 14 15:16:01 combo sshd(pam_unix)[19939]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=218.188.2.4

log_date = Jun 14 15:16:01
process_name = sshd
pid=19939
log_message= autentication failure
remote_host = 218.188.2.4
  • Solution
# One
(?<log_date>%{WORD}%{SPACE}%{MONTHDAY}%{SPACE}%{TIME})%{SPACE}%{WORD}%{SPACE}%{WORD:process_name}\(%{WORD}\)\[%{NUMBER:pid}\]:%{SPACE}%{GREEDYDATA:log_message};%{GREEDYDATA}%{IP:remote_host}

# Better solution
%{SYSLOGTIMESTAMP:log_date}%{SPACE}%{WORD}%{SPACE}%{WORD:process_name}\(%{WORD}\)\[%{NUMBER:pid}\]:%{SPACE}%{GREEDYDATA:log_message};%{GREEDYDATA}%{IP:remote_host}
  • Consider the following log
17/06/09 20:10:40 INFO executor.CoarseGrainedExecutorBackend: Registered signal handlers for [TERM, HUP, INT]

log_date = 17/06/09 20:10:40
level = info
class = executor.CoarseGrainedExecutorBackend
message = Registered signal handlers for [TERM, HUP, INT]
  • Solution
%{DATESTAMP:log_date}%{SPACE}%{LOGLEVEL:level}%{SPACE}%{JAVAFILE:class}:%{SPACE}%{GREEDYDATA:message}

Logstash

  • Lets apply the grok filters to log pipelines
  • Refer Here for sample pipeline
  • To make above pipeline write logs to elastic search we need to change the output plugin to elastic search and also mention index
    Preview
  • For logstash and kibana while you are creating a user consider built in roles Refer Here
  • For this we have executed the following curl requests
curl -X POST -u "elastic:Y+Q=JiV1EN=0lYe9hjpc" --insecure "https://localhost:9200/_security/user/qtelastic?pretty" -H 'Content-Type: application/json' -d'
{
  "password" : "india@1234",
  "roles" : [ "kibana_admin", "logstash_admin", "ingest_admin", "machine_learning_admin", "watcher_admin", "transform_admin" ],
  "full_name" : "qtelastic",
  "email" : "qtelastic@example.com",
  "metadata" : {
    "intelligence" : 7
  }
}
'

curl -X POST -u "elastic:Y+Q=JiV1EN=0lYe9hjpc" --insecure "https://localhost:9200/_security/user/qtadmin?pretty" -H 'Content-Type: application/json' -d'
{
  "password" : "india@1234",
  "roles" : [ "superuser" ],
  "full_name" : "qtadmin",
  "email" : "qtadmin@example.com",
  "metadata" : {
    "intelligence" : 7
  }
}
'

  • Lets configure logstash.yaml changed elastic search default username and password
  • Refer Here for the pipeline and then lets test this
  • We have sent the logs from logstash to elastic search, now lets verify in kibana
    Preview
    Preview
    Preview
    Preview
    Preview
  • configuration files should be stored in /etc/logstash/conf.d
  • restart the daemon and enable logstash service.

Preview

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

About continuous learner

devops & cloud enthusiastic learner