Logstash

• This is a popular open source data collection engine with real-time pipelining capabilities.
• This allows us to build a pipeline that can help in collecting logs from wide variety of input sources, and parse, enrich, unify and store it in a wide variety of destinations (elastic search)
• Logstash does the work of ETL Engine
  
• Installing Logstash Refer Here. I will be using apt based installation
• Logstash configuration files location: /etc/logstash
• Logstash installation location: /usr/share/logstash
• Refer below for logstash executable
  
• For logstash to define a pipeline we need to specify configuration. Generally configurations will be specified as .conf file and will be placedin /etc/logstash/conf.d.
• When ever logstash starts it will read all the .conf files in /etc/logstash/conf.d and creates the pipeline
• Lets start experimenting from commandline without conf files
  • input: stdin
  • output: stdout
• Lets execute the following commands

cd /usr/share/logstash
sudo ./bin/logstash -e "input { stdin {}} output { stdout{} }"

* Logstash Architecture

* The Logstash pipeline is stored in a configuration with a .conf extension. This configuration file has three sections

input
{

}
filter
{

}
output
{

}

• Now in input section we can add input plugins supported by logstash Refer Here. In addition to these built in plugins, there are lot of community/open source plugins or your can create your own plugin as well
• In the filter Section we can use filter plugins Refer Here
• In the output section we can use output plugins Refer Here

Simple Pipeline configuration

• Lets create a pipeline which reads the input from stdin and redirects the output to stdout

input {
    stdin { }
}
output {
    stdout { }
}

• Now lets store this in any folder in the vm and execute the following command

sudo /usr/share/logstash/bin/logstash -f simple.conf

* Activity 1: Lets create a pipeline which reads from std input and converts the message to uppercase and outputs to standard output
* Refer Here for the mutate filter
* The configuration is as shown below

input {
    stdin { }
}
filter {
    mutate {
        uppercase  =>  ["message"]
    }
}
output {
    stdout { }
}

• Start the logstash sudo /usr/share/logstash/bin/logstash -f activity1.conf
  
• Activity 2: Lets create a pipeline which reads from std input and converts the message to uppercase and outputs to standard output and to /tmp/logstashoutput file

input {
    stdin { }
}
filter {
    mutate {
        uppercase  =>  ["message"]
    }
}
output {
    stdout { }
    file {
        path => "/tmp/logstashoutput"
    }
}

• Now lets execute this with logstash and run some inputs and check the file /tmp/logstashoutput