DevOps Classroomnotes 01/Mar/2022

Overview of Logstash Plugins

  • Logstash has rich collection of input, filter and output plugins.
  • List of Plugins available in logstash
    Preview
  • View installed plugin list by group
    Preview
  • View installed plugin by name
    Preview
  • If you want to install additional plugins logstash-plugin install <plugin-name>
  • Codec Plugins:
    • These are used to encode or decode incoming or outgoing events from logstash

Exploring Plugins

  • File (input):
  • Lets install apache server
    sudo apt update
    sudo apt install apache2 -y
  • Now lets use logstash to read the access and error logs and redirect to stdout
    input {
    file{
    path => ["/var/log/apache2/access.log", "/var/log/apache2/error.log"]
    }
    }
    output {
    stdout {
    codec => rubydebug
    }
    }

    Preview
  • Now wait for the logstash to start and send http requests to your apache server
    Preview
  • File plugin for reading the logs is used when your organizational setup is to store logs on network file systems
  • Now as a continuation, lets try to send the output of the apache logs to elastic search. For this we need to configure
    • Refer Here for official documentation.
    • The current conf file
      input {
      file{
      path => ["/var/log/apache2/access.log", "/var/log/apache2/error.log"]
      }
      }
      output {
      stdout {
      codec => rubydebug
      }
      elasticsearch {
      hosts => "http://172.31.39.45:9200"
      index => "learningls-%{+yyyy.MM.dd}"
      }
      }
    • Now start logstash from command line
      Preview
  • At this moment we are able to send the logs from apache running locally to elastic search via Logstash. Typically logdata = timestamp + data. In this data we might have multiple fields which can be aggregated.
  • The individual access log is as shown below
223.238.77.113 - - [01/Mar/2022:03:20:20 +0000] "GET / HTTP/1.1" 200 3477 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"

49.205.103.150 - - [01/Mar/2022:03:24:30 +0000] "GET /favicon.ico HTTP/1.1" 404 491 "http://34.209.75.177/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36"


157.45.11.132 - - [01/Mar/2022:03:26:48 +0000] "GET /favicon.ico HTTP/1.1" 404 492 "http://34.209.75.177/" "Mozilla/5.0 (Linux; Android 11; RMX3360) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.101 Mobile Safari/537.36"


49.206.34.54 - - [01/Mar/2022:03:27:21 +0000] "GET /favicon.ico HTTP/1.1" 404 492 "http://34.209.75.177/" "Mozilla/5.0 (iPhone; CPU iPhone OS 15_0_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Mobile/15E148 Safari/604.1"
  • Rather than considering every thing as text if we can create multiple fields like client ip, datetime, http method, uri, status code, client-device
  • Next Step:
    • How to parse a text and create multiple fields to help in log analysis

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

About continuous learner

devops & cloud enthusiastic learner