DevOps Classroomnotes 24/Sep/2023

Log Parsing using logstash

• To use logstash we need to build log pipelines which has 3 essential parts to it
  • inputs: Where we read the data from
  • filter: Transformations to the data
  • outputs: where to redirect the data
• Logstash has plugins to perform these activities
• Installtion Refer Here
• first pipeline

input 
{ 
    stdin { } 
} 
output 
{ 
    stdout {} 
}

• The command logstash -e 'input { stdin { } } output { stdout {} }' will be used to see the first logstash
  
• Now lets create a sample pipeline in /tmp/attempt.conf

input 
{ 
    stdin { } 
} 
output 
{ 
    stdout {
        codec => "json"
    } 
}

• lets evaluate the results

### input
is this json
### output 
{
    "message": "is this json",
    "event": {
        "original": "is this json"
    },
    "@version": "1",
    "@timestamp": "2023-09-24T03:34:19.111196717Z",
    "host": {
        "hostname": "ip-172-31-16-199"
    }
}

• Now lets execute the following plugin

input 
{ 
    stdin { } 
} 
filter {
    mutate {
        add_field => { "shortHostname" => "%{[hostname][0]}" }
    }
}
output 
{ 
    stdout { } 
}

• Then next

input {
    file {
        path => ["/var/log/apache2/access.log"]
    }

}
output {
    stdout {

    }
}

• Parsing single message into multiple fields Refer Here and also Refer Here
• Lets use the sample pattern

input {
    file {
        path => ["/var/log/apache2/access.log"]
    }

}
filter {
    grok {
        match => {
            "message" => "%{IP:clientip}%{SPACE}-%{SPACE}-%{SPACE}\[%{HTTPDATE:when}\]%{SPACE}\"%{WORD:method}%{SPACE}%{PATH:where}%{SPACE}%{GREEDYDATA:message}"
        }
    }
}
output {
    stdout {

    }
}

• Our ideal config file

input {
    beats {
        # apache
        port => 5045 
    }
}
filter {
    grok {

    }
    mutate {
        add_field => { "component" => "radiology" }
    }
}
output {
    elasticsearch {
        cloud.id => ""
        cloud.auth => ""
        index => "radiology-<date>"
    }
}