DevOps Classroomnotes 20/Jul/2022

Log Parsing with Logstash

Filter Plugins

• Lets read the input from stdin and display the output to the stdout and i want to add one field called as purpose with value learning

input 
{
    stdin {}
}
filter
{
    mutate {
        add_field => {
            "purpose" => "learning"
        }
    }

}
output
{
    stdout {}

}

• Lets search for a filter plugin which can add fields . Refer Here for all the standard filter plugins and Refer Here for mutate filter
• Now start the logstash with the above pipeline
  
• Activity 5: Split the message with commas

input 
{
    stdin {}
}
filter
{
    mutate {
        split => {
            "message" => ","
        }
    }

}
output
{
    stdout {}

}

• Activity 6: Convert the message into upper case and then split the message with ,

input 
{
    stdin {}
}
filter
{
    mutate {
        uppercase => [ "message" ]
    }
    mutate {
        split => {
            "message" => ","
        }
    }

}
output
{
    stdout {}

}

Grok filter plugin

• Refer Here for the official documentation
• Logstash is shipped with the grok patterns Refer Here
• For testing grok patterns we can use Refer Here
• Ensure you go through GROK Basics Refer Here
• By using Grok Patterns we had parsed 55.3.244.1 GET /index.html 15824 0.043 this into multiple fields by using expression %{IP:clientip}%{SPACE}(?<method>\w+)%{SPACE}%{UNIXPATH:path}%{SPACE}%{NUMBER:size}%{SPACE}%{NUMBER:time} and the result was

{
  "clientip": [
    [
      "55.3.244.1"
    ]
  ],
  "IPV6": [
    [
      null
    ]
  ],
  "IPV4": [
    [
      "55.3.244.1"
    ]
  ],
  "SPACE": [
    [
      " ",
      " ",
      " ",
      " "
    ]
  ],
  "method": [
    [
      "GET"
    ]
  ],
  "path": [
    [
      "/index.html"
    ]
  ],
  "size": [
    [
      "15824"
    ]
  ],
  "BASE10NUM": [
    [
      "15824",
      "0.043"
    ]
  ],
  "time": [
    [
      "0.043"
    ]
  ]
}