Scenario
- To understand Elastic Stack, We assume the organization as AdventureWorks in this series.
- Lets see different application/db servers used by Adventure Works
Application-1 Architecture
- To make the above application observable, we need to monitor
- Logs
- Traces
- Metrics
Enabling Log Monitoring
- It is very difficult to login into each server for the logs
- We need a central location for looking into logs of each applciation/server
- To acheive this, we have the following options
- Store all the logs into Database like mysql/oracle
- Use Syslog / Event log services
Lets Examine SQL Approach
- In this Approach we send the logs of application to the Central SQL Server
- To send the logs of mysql and mongodb we might use ETL (Extract, Transform and Load) to see the logs in SQL Server
- It looks like the problem is solved.
- Problems with this approach:
- Logs in the SQL server needs to be queried. But logs are text. Text searching in SQL is not that great.
- Logs of all the different servers generally will be in different formats
Possible Solution
- We need a system which can process different log formats and do the necessary adjustments
- We also need a system which can search text quickly
- Elastic Stack comes to the rescue
Elastic Stack
- It is suite of Products
- In our Application’s Scenario we use Elastic Stack as our monitoring/logging platform.
- Elastic Stack = ELBK
Elastic Search
- Effecient Text Searching Distributed System.
LogStash
- Log Parsing and Conversions
Kibana
- Building Dashboards
Beats
- Agents which run on nodes which export logs to LogStash or ElasticSearch and/or report metrics.
1 comment