- Logstash helps centralize event data such as logs, metrics or any other data in any format.
- It can perform transformations before sending it to the stash of your choice.
- Logstash is a server side component
- For example. consider the log
08:09 18-02-2022 [com.qt.ecommerce.web] 200 Ok response for products /api/productsand from this log which is a text record we want to create multiple fields and store it in elastic search
datetime => 18/02/2022 08:09 AM component => com.qt.ecommerce.web status code => 200 message => Ok response for products service endpoint => /api/products
- The above transformation can help us with coming up with a chart on status codes for our application
- In Logstash we can create a log pipeline with the help of plugins to acheive this
- This is a platform of opensource light weight data shippers.
- Logstash is a server side component, whereas beats has a role on the client side.
- We install Beats on the servers from where we want to collect data (logs, metrics etc…)
- Beats can complement the Logstash and if the transformations are not require they can forward the data directly to elastic search as well.
- This is the visualization tool for the Elastic Stack
- Here we can view the logs, Analyze the logs and create visualizations (Charts, tables etc) and create dashboards.
Lab Environment For understanding Elastic Search
- Initially, we will explore the concepts and features of Elastic Search and to experiement with Elastic Search, We would use Kibana
- Server Configuration for installation
- Operating Sytem: Ubuntu 20.04
- RAM => 4GB
- Cpu Core => 2
- I will be creating a virtual machine on the cloud (AWS/Azure)
Creating Virtual Machines on
- Azure Refer Here
- AWS Refer Here
- Please ensure you shutdown your virtual machines after usage.
- Installing Necessary softwares on Windows Refer Here
- If you are windows 10/11 user, configure Windows Terminal Refer Here
- Mac users:
- Install Homebrew Refer Here
- Install Git:
brew install git
- Install visual studio code
brew install --cask visual-studio-code
Installing Elastic Search
- Ensure you have a linux vm where you can login with above mentioned Configuration
- Login into the linux machine and execute
sudo apt update
- Next Steps:
- Completing installation and configuration.
- HTTP Status Codes:
- 1xx => information
- 2xx => ok status
- 3xx => redirection
- 4xx => client side errors
- 5xx => server side errors