Logstash

• Logstash helps centralize event data such as logs, metrics or any other data in any format.
• It can perform transformations before sending it to the stash of your choice.
• Logstash is a server side component
• For example. consider the log 08:09 18-02-2022 [com.qt.ecommerce.web] 200 Ok response for products /api/products and from this log which is a text record we want to create multiple fields and store it in elastic search

datetime => 18/02/2022 08:09 AM
component => com.qt.ecommerce.web
status code => 200 
message => Ok response for products
service endpoint => /api/products

• The above transformation can help us with coming up with a chart on status codes for our application
  
• In Logstash we can create a log pipeline with the help of plugins to acheive this

Beats

• This is a platform of opensource light weight data shippers.
• Logstash is a server side component, whereas beats has a role on the client side.
• We install Beats on the servers from where we want to collect data (logs, metrics etc…)
• Beats can complement the Logstash and if the transformations are not require they can forward the data directly to elastic search as well.
  

Kibana

• This is the visualization tool for the Elastic Stack
• Here we can view the logs, Analyze the logs and create visualizations (Charts, tables etc) and create dashboards.

Lab Environment For understanding Elastic Search

• Initially, we will explore the concepts and features of Elastic Search and to experiement with Elastic Search, We would use Kibana
  
• Server Configuration for installation
  • Operating Sytem: Ubuntu 20.04
  • RAM => 4GB
  • Cpu Core => 2
• I will be creating a virtual machine on the cloud (AWS/Azure)
• Creating Virtual Machines on
  • Azure Refer Here
  • AWS Refer Here
• Note:
• Please ensure you shutdown your virtual machines after usage.
• Installing Necessary softwares on Windows Refer Here
• If you are windows 10/11 user, configure Windows Terminal Refer Here
• Mac users:
  • Install Homebrew Refer Here
  • Install Git: brew install git
  • Install visual studio code brew install --cask visual-studio-code

Installing Elastic Search

• Ensure you have a linux vm where you can login with above mentioned Configuration
• Login into the linux machine and execute sudo apt update
• Next Steps:
• Completing installation and configuration.

Terms

• HTTP Status Codes:
  • 1xx => information
  • 2xx => ok status
  • 3xx => redirection
  • 4xx => client side errors
  • 5xx => server side errors