DevOps Classroom Notes 30 Oct 2019 – Elastic Stack Usecase

Scenario

• To understand Elastic Stack, We assume the organization as AdventureWorks in this series.
• Lets see different application/db servers used by Adventure Works

Application-1 Architecture

• To make the above application observable, we need to monitor
  • Logs
  • Traces
  • Metrics

Enabling Log Monitoring

• It is very difficult to login into each server for the logs
• We need a central location for looking into logs of each applciation/server
• To acheive this, we have the following options
  • Store all the logs into Database like mysql/oracle
  • Use Syslog / Event log services

Lets Examine SQL Approach

• In this Approach we send the logs of application to the Central SQL Server
• To send the logs of mysql and mongodb we might use ETL (Extract, Transform and Load) to see the logs in SQL Server
• It looks like the problem is solved.
• Problems with this approach:
  • Logs in the SQL server needs to be queried. But logs are text. Text searching in SQL is not that great.
  • Logs of all the different servers generally will be in different formats

Possible Solution

• We need a system which can process different log formats and do the necessary adjustments
• We also need a system which can search text quickly
• Elastic Stack comes to the rescue

Elastic Stack

• It is suite of Products
• In our Application’s Scenario we use Elastic Stack as our monitoring/logging platform.
• Elastic Stack = ELBK

Elastic Search

• Effecient Text Searching Distributed System.

LogStash

• Log Parsing and Conversions

Kibana

• Building Dashboards

Beats

• Agents which run on nodes which export logs to LogStash or ElasticSearch and/or report metrics.