DevOps Classroom Series – 24/Apr/2021

Logstash architecture

  • Logstash processing pipeline has 3 stages
    • inputs
    • filters
    • outputs
  • The basic architecture is as shown below Preview
  • The logstash pipeline is stored in a configuration file that ends with .conf and is as shown below
input
{
    #input plugins
}
filter
{
    # filter plugins
}
output
{
    # output plugins
}
  • For input plugins Refer Here and for filter plugins Refer Here and for output plugins Refer Here

  • Lets start creating logstash configurations

  • Scenario1: : Read input from standard input and write the output the standard ouput

  • Refer Here for the configuration file created

  • Run the logstash from commandline Preview Preview

  • Scenario 2: Read input from standard input and convert the message into uppercase and write the output the standard output

  • Refer Here for the conf file

  • Run the logstash from command line Preview

  • Scenario3: Read the input from standard input and split the message whenever there is | character, give the first item (message[0]) a new field name firstname and second field as lastname.

  • Refer Here for the configuration file Preview

  • Listing Plugins Preview Preview

  • we can also group the plugins by either input, output, filter and codec

  • Scenario4: Try to read the movies from movies.csv and display the contents

  • Now we can conclude given the message format, we can do some formatting/filter using logstash and send it to elastic search

  • When applications run they generate logs

  • We need to extract meanigful information from these logs to gain insights.

  • Basic Log Pipeline Preview

  • Extracting meaningful information from logs is challenging and we need to look into some additional filters offered by logstash

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

About learningthoughtsadmin