Log Parsing using logstash
- To use logstash we need to build log pipelines which has 3 essential parts to it
- inputs: Where we read the data from
- filter: Transformations to the data
- outputs: where to redirect the data
- Logstash has plugins to perform these activities
- Installtion Refer Here
- first pipeline
input
{
stdin { }
}
output
{
stdout {}
}
- The command
logstash -e 'input { stdin { } } output { stdout {} }'will be used to see the first logstash
- Now lets create a sample pipeline in /tmp/attempt.conf
input
{
stdin { }
}
output
{
stdout {
codec => "json"
}
}
- lets evaluate the results
### input
is this json
### output
{
"message": "is this json",
"event": {
"original": "is this json"
},
"@version": "1",
"@timestamp": "2023-09-24T03:34:19.111196717Z",
"host": {
"hostname": "ip-172-31-16-199"
}
}
- Now lets execute the following plugin
input
{
stdin { }
}
filter {
mutate {
add_field => { "shortHostname" => "%{[hostname][0]}" }
}
}
output
{
stdout { }
}
- Then next
input {
file {
path => ["/var/log/apache2/access.log"]
}
}
output {
stdout {
}
}
- Parsing single message into multiple fields Refer Here and also Refer Here
- Lets use the sample pattern
input {
file {
path => ["/var/log/apache2/access.log"]
}
}
filter {
grok {
match => {
"message" => "%{IP:clientip}%{SPACE}-%{SPACE}-%{SPACE}\[%{HTTPDATE:when}\]%{SPACE}\"%{WORD:method}%{SPACE}%{PATH:where}%{SPACE}%{GREEDYDATA:message}"
}
}
}
output {
stdout {
}
}
- Our ideal config file
input {
beats {
# apache
port => 5045
}
}
filter {
grok {
}
mutate {
add_field => { "component" => "radiology" }
}
}
output {
elasticsearch {
cloud.id => ""
cloud.auth => ""
index => "radiology-<date>"
}
}
