DevOps Classroom Series – 16/Jul/2020

Applications and Centralized Log Monitoring

  • Applications running generate logs.
  • Applications run on servers which also generated logs
  • Applications store the data in database which also will have logs
  • Logs are everywhere, But the problem is logs are not standardized
  • Event viewer example
- <Event xmlns="">
- <System>
  <Provider Name="ESENT" /> 
  <EventID Qualifiers="0">455</EventID> 
  <TimeCreated SystemTime="2020-07-16T01:55:19.800855900Z" /> 
  <Security /> 
- <EventData>
  <Data>-1023 (0xfffffc01)</Data> 
  • Application Log example
Started at Wed May 06 09:29:26 IST 2020
Finished at Wed May 06 09:29:26 IST 2020. 4ms

  • DB Logs will be different format

  • Searching logs will be a tedious job

  • Consider the sample architecture below with different servers and applications Preview

  • When any thing goes to get down to root cause, we might need to search logs and this is difficult job as logs are in different formats

  • So we need a centralized log monitoring which can

    • search various formats of logs
    • Help us building dashboards
    • Help in APM (Application Performance Monitoring)
    • Help in Audits
  • Some of the tools which can help over here

    • System Center
    • Splunk
    • AppDynamics (APM)
    • Elastic Stack

Elastic Stack

  • The Elastic Stack is eco-system of components serving full search & analytics stack.

  • Main components are

    • Elastic Search: Provides storage, search and analytical capabilities
    • LogStash: Helps in getting data into Elastic search
    • Kibana: UI for elastic stack, Visualization capabilities
    • Beats: Helps in getting data into Elastic search (agent)
    • X-pack: Provides features for monitoring, alerting, security Preview
  • Lets have 5000 feet overview of elastic stack in sample architecture Preview

Elastic Search

  • Elastic search is a real-time distributed search and analytics engine that is horizontally scalable and capable of solving wide variety of use cases.
  • Elastic search is core of Elastic Stack.
  • Elastic search plays the central role of search engine and analytics engine
  • Apache has built a tool for text searching and indexing which is called as Apache Lucene
  • Elastic search is built on top of Apache Lucene
  • Elastic search’s key benefits
    • Schemaless, document-oriented
    • Searching
    • Analytics
    • Rich Client Library support and REST API
    • Near real-time
    • Lightning-fast
    • Fault-tolerant

Schemaless and document Oriented

  • Elastic Search stores the data in JSON Documents, A example document looks as shown below
    "name": "Khaja Ibrahim",
    "courses": ["AWS", "Azure", "DevOps", "Python"],
    "Organization": "QualityThought"
    "name": "Ramana",
    "courses": ["Agile", "Manaul Testing"],
    "Organization": "QualityThought",
    "Email": ""
  • These kind of documents represent faculty records
  • Use Cases of Elastic Stack
    • Log analysis
    • Product search
    • Metric analytics
    • Web Searches and website search

Our Setup for Elastic Search

  • Ubuntu VM with 4 VCPUs and 16 GB of RAM
  • Install Elastic Search and Kibana on the same server Preview
  • Installation Steps for Elastic Search Refer Here
  • Installation Steps for Kibana Refer Here

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

About learningthoughtsadmin