Log Monitoring with Elastic Stack

Scenario: Monitoring Apache Logs

• Basic Structure for Log Monitoring using Elastic Stack
  
• Lets setup the apache server

sudo apt update
sudo apt install apache2 -y

• Once the apache server is setup to generate the traffic we have created the small script

#!/bin/bash
while true
do
    SERVER_IP_ADDRESS='54.244.44.152'
    curl "http://${SERVER_IP_ADDRESS}"
    sleep 1s
    curl "http://${SERVER_IP_ADDRESS}"
    sleep 1s
    curl "http://${SERVER_IP_ADDRESS}/test.html"
    sleep 1s
done

• Setting up logstash: Refer Here
• Now we need to create the logstash pipeline to recieve the logs from beats and send it to the elastic search and also transfom the apache logs
• The untested pipeline for transforming apache logs is as shown below

input
{
    beats 
    {
        port => 5044

    }
}
filter
{
    grok 
    {
        match => { "message" => "%{COMBINEDAPACHELOG}" }
    }

}
output
{
    elasticsearch 
    {
        cloud_auth => "elastic:vu2fraeXw6bLb7DuDF6w0U3A"
        hosts => ["https://qt-elastic.es.us-central1.gcp.cloud.es.io"]
        index => "apache-%{+yyyy.MM.dd}"
    }
}

• Now save this as apache.conf and save it in /etc/logstash/conf.d/ and then enable and start logstash

sudo systemctl enable logstash
sudo systemctl start logstash

• Now we need to setup beats which reads the logs written to file /var/logs/apache2/access.log
• Elastic stack has different beats
  
• For our scenario, we use file beat
• Refer Here for the installation
• Installation using APT Refer Here
• Refer Here for configuring file beats
• Create a Dataview in kibana for the apache-*
  
• Lets search for the logs with status code 200 and 404 and save the query
  
  
  
  
• Refer Here for the changeset containing the polling script and apache config for the logstash.