Logging

• Logging for an application or system or os is where the information about events
• Refer Here for common log formats
• Log Levels:
  • Critical
  • Error
  • Debug
  • Info
  • Warning
• Log time: When did it happened
• Log Source: Which component is executing this log
• Message: This describes the log
• We have understood logging and how it can be useful.
• Problem is no standard form of logging
• We have beat components which can send logs to elastic search but the complete line is stored in one field which will still be difficult to search so we need to break the log message into multiple fields

17.18.19.20 - - [28/Jul/2006:10:22:04 -0300] "GET /orders HTTP/1.0" 200 2216
client ip = 17.18.19.20
datetime: 28/Jul/2006:10:22:04
request type: GET
location: /orders
response: 200

• logstash can read the logs from various sources, filter or transform the logs by many mechanisms and send output to many destinations, for this we create a log pipeline

inputs {
    // plugin a
    // plugin b
}
filters {
    // plugin c
    // plugin d

}
output {
    // plugin d
    // plugin e
}