DevOps Classroomnotes 19/Jul/2022

LogStash

  • Extract , Transform and Load (ETL):
    • Extract i.e. read the logs from source
    • Transform the log records into queryable fields
    • load i.e. store/push/index into elastic search
  • Install Logstash: Refer Here
  • Logstash performs extract, transform and load with the help of plugins

Play with Logstash

  • Logstash is located generally in /usr/share/logstash/bin
    Preview
  • Lets search for the options sudo /usr/share/logstash/bin/logstash --help
    Preview
  • Refer Here for the command line options
  • To the logstash we need to provide pipeline as input
input 
{
    <plugin-name> 
    {
        <option-1> => <value-1>
        ...
        <option-n> => <value-n>
    }
}
filter
{
    <plugin-name> 
    {
        <option-1> => <value-1>
        ...
        <option-n> => <value-n>
    }

}
output
{
    <plugin-name> 
    {
        <option-1> => <value-1>
        ...
        <option-n> => <value-n>
    }

}
  • filter is optional
  • Activity 1: Lets create a pipeline which reads from command line (termianl) and shows the output on the terminal. Save this in ~/logstash-pipelines/hello-wrold.conf
input
{
    stdin 
    {

    }
}
output
{
    stdout
    {

    }

}
  • Now try to run logstash with the following command sudo ./logstash -f ~/logstash-pipelines/hello-wrold.conf
    Preview
  • Now lets give some input
    Preview
  • Activity 2: Now lets try to create a logstash pipeline which reads from stdin and stores in a file and also shows the output in stdout. save it as activity2.conf
input
{
    stdin {}
}
output
{
    stdout {}
    file
    {
        path => '/tmp/activity2'
        create_if_deleted => true
        flush_interval => 0
    }

}
  • Now run logstash sudo ./logstash -f ~/logstash-pipelines/activity2.conf
    Preview
    Preview
  • Activity 3: Create a logstash pipeline which reads the inputs from a file /tmp/messages and writes the output to stdout
input
{
    file
    {
        path => '/tmp/messages'
        start_position => 'beginning'

    }

}
output
{
    stdout
    {

    }

}
  • Next Steps:
    • Lets read logs from some applications

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

About continuous learner

devops & cloud enthusiastic learner