DevOps Classroom Series – 24/Jul/2020

Codec plugins

• Codec plugins are used to encode or decode incoming or outgoing events from logstash Refer Here

Exploring Input Plugins

• File:

  • file plugin is used to stream events from file(s) line by line.
  • Refer file input plugin from here
  • Lets create a configuration file-hello.conf

  input 
  {
      file {
          path => "/home/ubuntu/logfiles/*"
      }
  }
  output
  {
      stdout {
          codec => rubydebug
      }
  }
  

  

  • The sample configuration to read the logs from /var/log folder will be

  input {
      file {
          path => "/var/log/*.log"
      }
  }
  output {
      stdout {
          codec => rubydebug
      }
  }
  

• Beats: Beats plugin allows Logstash to recievd events from elastic beats framework, Process them & send them to destinations

  • Lets write a configuration file and start logstash executable

  input {
      beats {
          port => 1234
      }
  }
  output {
      stdout {
          codec => rubydebug
      }
  }
  

  

  • In reality our output from beats is always elastic search

• JDBC: This plugin is used to import the data from the database to Logstash. JDCB configuration generally have the following important conf . Refer Here to understand connectivity terms used

  • jdbc_driver_libary: Path to Jar file which is jdbc driver
  • jdbc_driver_class: This is name of the driver
  • jdbc_connection_string: jdbc connection string
  • jdbc_user: username of db
  • jdbc_password: password of db
  • statement: this is sql statement or query to pull logs
  • parameters: query paramteres
  • schedule => when to run this query (* * * * *) (Cronjob)
  • Sample configuration

  input {
      jdbc {
          jdbc_driver_library => "/usr/share/mysql-connector-java"
          jdbc_driver_class => "com.mysql.jdbc.Driver"
          jdbc_connection_string => "jdbc:mysql://localhost:3306/sonoo"
          jdbc_user => "root"
          jdbc_password => "root@123"
          schedule => "30 * * * *"
          statement => "Select * from applogs"
  
      }
  }
  output{
      stdout {
          codec => rubydebug
      }
  }
  

• IMAP: This plugin is used to read emails from IMAP server.

• Play with Heroku Grok debugger Refer Here