DevOps Classroom Series – 16/Jul/2020

Applications and Centralized Log Monitoring

• Applications running generate logs.
• Applications run on servers which also generated logs
• Applications store the data in database which also will have logs
• Logs are everywhere, But the problem is logs are not standardized
• Event viewer example

EVENT VIEWER LOGS in Windows
- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="ESENT" /> 
  <EventID Qualifiers="0">455</EventID> 
  <Level>2</Level> 
  <Task>3</Task> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2020-07-16T01:55:19.800855900Z" /> 
  <EventRecordID>31349</EventRecordID> 
  <Channel>Application</Channel> 
  <Computer>DESKTOP-HGH07L2</Computer> 
  <Security /> 
  </System>
- <EventData>
  <Data>svchost</Data> 
  <Data>13036,R,98</Data> 
  <Data>TILEREPOSITORYS-1-5-18:</Data> 
  <Data>C:\WINDOWS\system32\config\systemprofile\AppData\Local\TileDataLayer\Database\EDB.log</Data> 
  <Data>-1023 (0xfffffc01)</Data> 
  </EventData>
  </Event>

• Application Log example

Started at Wed May 06 09:29:26 IST 2020
Finished at Wed May 06 09:29:26 IST 2020. 4ms

• DB Logs will be different format

• Searching logs will be a tedious job

• Consider the sample architecture below with different servers and applications

• When any thing goes to get down to root cause, we might need to search logs and this is difficult job as logs are in different formats

• So we need a centralized log monitoring which can

  • search various formats of logs
  • Help us building dashboards
  • Help in APM (Application Performance Monitoring)
  • Help in Audits

• Some of the tools which can help over here

  • System Center
  • Splunk
  • AppDynamics (APM)
  • Elastic Stack

Elastic Stack

• The Elastic Stack is eco-system of components serving full search & analytics stack.

• Main components are

  • Elastic Search: Provides storage, search and analytical capabilities
  • LogStash: Helps in getting data into Elastic search
  • Kibana: UI for elastic stack, Visualization capabilities
  • Beats: Helps in getting data into Elastic search (agent)
  • X-pack: Provides features for monitoring, alerting, security

• Lets have 5000 feet overview of elastic stack in sample architecture

Elastic Search

• Elastic search is a real-time distributed search and analytics engine that is horizontally scalable and capable of solving wide variety of use cases.
• Elastic search is core of Elastic Stack.
• Elastic search plays the central role of search engine and analytics engine
• Apache has built a tool for text searching and indexing which is called as Apache Lucene
• Elastic search is built on top of Apache Lucene
• Elastic search’s key benefits
  • Schemaless, document-oriented
  • Searching
  • Analytics
  • Rich Client Library support and REST API
  • Near real-time
  • Lightning-fast
  • Fault-tolerant

Schemaless and document Oriented

• Elastic Search stores the data in JSON Documents, A example document looks as shown below

{
    "name": "Khaja Ibrahim",
    "courses": ["AWS", "Azure", "DevOps", "Python"],
    "Organization": "QualityThought"
}
{
    "name": "Ramana",
    "courses": ["Agile", "Manaul Testing"],
    "Organization": "QualityThought",
    "Email": "qtramana@gmail.com"
}

• These kind of documents represent faculty records
• Use Cases of Elastic Stack
  • Log analysis
  • Product search
  • Metric analytics
  • Web Searches and website search

Our Setup for Elastic Search

• Ubuntu VM with 4 VCPUs and 16 GB of RAM
• Install Elastic Search and Kibana on the same server
• Installation Steps for Elastic Search Refer Here
• Installation Steps for Kibana Refer Here