DevOps Classroom Series – 29/Sept/2021

Logstash Configuration

• The logstash pipeline is stored in a configuration file with .conf extension. The three sections of the configuration file are

input 
{

}
filter
{

}
output
{

}

• In Each of section, we can have one or more plugin configuration
• Lets create a simple configuration file

input
{
    stdin {}

}
filter
{
    mutate {
        uppercase => [
            "message",
            "host"
        ]
    }
}
output
{
    stdout {}

}

• So in the logstash we have plugins and plugins have properties which we can set

Overview of Logstash Plugins

• To view all the plugins installed in logstash

sudo /usr/share/logstash/bin/logstash-plugin list

• To install a plugin

sudo /usr/share/logstash/bin/logstash-plugin install <plugin-name>

• To update a plugin

sudo /usr/share/logstash/bin/logstash-plugin update <plugin-name>

• Lets Explore some of the poplular plugins
• File:
  • Refer Here for the official documentation
  • Lets look at the following configuration

    input
    {
        file 
        {
            path => ["/var/log/*", "/var/log/**/*.log"]
            start_position => "beginning"
            exclude => ["*.csv"]
            discover_interval => 20
            stat_interval => "500ms"
    
        }
    
    }
    output
    {
        stdout {}
    }
    

  • Try to create a logstash config which reads mysql logs stored /var/logs/mysql/*.log