LogStash

• Extract , Transform and Load (ETL):
  • Extract i.e. read the logs from source
  • Transform the log records into queryable fields
  • load i.e. store/push/index into elastic search
• Install Logstash: Refer Here
• Logstash performs extract, transform and load with the help of plugins
  • Extract => input plugins Refer Here
  • Transform => filter plugins Refer Here
  • Load => Output plugins Refer Here

Play with Logstash

• Logstash is located generally in /usr/share/logstash/bin
  
• Lets search for the options sudo /usr/share/logstash/bin/logstash --help
  
• Refer Here for the command line options
• To the logstash we need to provide pipeline as input

input 
{
    <plugin-name> 
    {
        <option-1> => <value-1>
        ...
        <option-n> => <value-n>
    }
}
filter
{
    <plugin-name> 
    {
        <option-1> => <value-1>
        ...
        <option-n> => <value-n>
    }

}
output
{
    <plugin-name> 
    {
        <option-1> => <value-1>
        ...
        <option-n> => <value-n>
    }

}

• filter is optional
• Activity 1: Lets create a pipeline which reads from command line (termianl) and shows the output on the terminal. Save this in ~/logstash-pipelines/hello-wrold.conf

input
{
    stdin 
    {

    }
}
output
{
    stdout
    {

    }

}

• Now try to run logstash with the following command sudo ./logstash -f ~/logstash-pipelines/hello-wrold.conf
  
• Now lets give some input
  
• Activity 2: Now lets try to create a logstash pipeline which reads from stdin and stores in a file and also shows the output in stdout. save it as activity2.conf

input
{
    stdin {}
}
output
{
    stdout {}
    file
    {
        path => '/tmp/activity2'
        create_if_deleted => true
        flush_interval => 0
    }

}

• Now run logstash sudo ./logstash -f ~/logstash-pipelines/activity2.conf
  
  
• Activity 3: Create a logstash pipeline which reads the inputs from a file /tmp/messages and writes the output to stdout

input
{
    file
    {
        path => '/tmp/messages'
        start_position => 'beginning'

    }

}
output
{
    stdout
    {

    }

}

• Next Steps:
  • Lets read logs from some applications