DevOps Classroom Series – 30/Nov/2020

Analyzing log data

  • Logs contain information about the state & behavior of system or an application.
  • Logs are generally generated whenever some event occurs
  • Typically
Log = Timestamp + Data
  • Logs don’t have consistent format
    • Different applications might be creating logs in different formats
    • Different applications might store the logs in different destinations (files, database, event logs, sys logs)
    • Logs are decentralized
  • Reasons for log analysis:
    • Troubleshooting
    • To understand application behavior
    • Auditing
    • Predective Analytics

Logstash

  • This is an opensource data collection engine with real-time pipelining capabilities.
  • This allows us to easily build a pipeline that can help in collecting data from various input sources (Extract), parse, enrich, unify(Transform) and store (Load) in wide variety of destinations. Preview
  • Logstash provides sets of plugins
    • input plugins
    • filter plugins
    • output plugins
  • Logstash has a plugin based architecture
  • Installing Logstash: Refer Here
  • Logstash architecture Preview
  • Now lets build a simple logstash pipeline which reads the input from standard input and send the data to standard output Preview Preview
  • Now lets give some inputs Preview
  • Logstash pipelines are stored in configuration files with .conf extenstion. The configuration file looks as
input
{

}
filter
{

}
output
{

}
  • In Each of this section one or more plugin configurations can be written. Plugins are generally configured by providing the name of the plugin and the settings of plugins are key value pairs . Value is assigned to key by using => operator
  • Plugins links Refer Here
  • Input plugins Refer Here Output Plugins Refer Here Filter Plugins Refer Here Codec Plugins Refer Here
  • Lets try to convert the input written into upper case
input
{
  stdin {}
}
filter
{
    mutate {
        uppercase => ["message"]
    }

}
output
{
    stdout {
        codec => json
    }
    
}

Preview Preview

  • Overview of logstash plugins Preview Preview Preview
  • Use the file input plugin to read the log files from /var/log/*.log files and output that to stdout
input
{
    file
    {
        path => ["/var/log/*.log"]
        mode => "read"
    }
}
output 
{
    stdout {}
}

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

About learningthoughtsadmin