DevOps Classroomnotes 01/Mar/2022

Overview of Logstash Plugins

• Logstash has rich collection of input, filter and output plugins.
• List of Plugins available in logstash
  
• View installed plugin list by group
  
• View installed plugin by name
  
• If you want to install additional plugins logstash-plugin install <plugin-name>
• Codec Plugins:
  • These are used to encode or decode incoming or outgoing events from logstash

Exploring Plugins

• File (input):
• Lets install apache server
  sudo apt update
sudo apt install apache2 -y
• Now lets use logstash to read the access and error logs and redirect to stdout
  input {
file{
path => ["/var/log/apache2/access.log", "/var/log/apache2/error.log"]
}
}
output {
stdout {
codec => rubydebug
}
}
  
• Now wait for the logstash to start and send http requests to your apache server
  
• File plugin for reading the logs is used when your organizational setup is to store logs on network file systems
• Now as a continuation, lets try to send the output of the apache logs to elastic search. For this we need to configure
  • Refer Here for official documentation.
  • The current conf file
    input {
file{
path => ["/var/log/apache2/access.log", "/var/log/apache2/error.log"]
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => "http://172.31.39.45:9200"
index => "learningls-%{+yyyy.MM.dd}"
}
}
  • Now start logstash from command line
    
• At this moment we are able to send the logs from apache running locally to elastic search via Logstash. Typically logdata = timestamp + data. In this data we might have multiple fields which can be aggregated.
• The individual access log is as shown below

223.238.77.113 - - [01/Mar/2022:03:20:20 +0000] "GET / HTTP/1.1" 200 3477 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"

49.205.103.150 - - [01/Mar/2022:03:24:30 +0000] "GET /favicon.ico HTTP/1.1" 404 491 "http://34.209.75.177/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36"


157.45.11.132 - - [01/Mar/2022:03:26:48 +0000] "GET /favicon.ico HTTP/1.1" 404 492 "http://34.209.75.177/" "Mozilla/5.0 (Linux; Android 11; RMX3360) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.101 Mobile Safari/537.36"


49.206.34.54 - - [01/Mar/2022:03:27:21 +0000] "GET /favicon.ico HTTP/1.1" 404 492 "http://34.209.75.177/" "Mozilla/5.0 (iPhone; CPU iPhone OS 15_0_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Mobile/15E148 Safari/604.1"

• Rather than considering every thing as text if we can create multiple fields like client ip, datetime, http method, uri, status code, client-device
• Next Step:
  • How to parse a text and create multiple fields to help in log analysis