Reference Architecture

• Consider the above reference architecture, we have

  • web
  • app
  • db
  • DNS
  • Mail

• All of the components should be up

• All of the servers should be running optimally

• Now we need to setup monitoring to

  • identify failures
  • Diagnosis information in case of failures

• To do the monitoring we have many solution

  • Prometheus
  • Zabbix
  • AppDynamics
  • Splunk
  • Datadog
  • Stackdriver (GCE)
  • Azure Monitoring stack
  • Cloud watch
  • Elastic Stack

Elastic Stack

• Set of tools available to setup monitoring, logging and tracing for almost all applications
• Stack of Products from elastic

Elastic Search

• Is used to store and index text
• In our use case, you can think of this as a database of logs, metrics and traces

Logstash

• Is used to parse the logs into meaningful text
• It does pattern matching and convert the logs collected from applications and stores in Elastic Search

kibana

• Visualizations of the logs can be observed using kibana
• Dashboards are built on kibana

Beats

• Agents installed on systems to export the logs to logstash or elastic search.
• Differnet kinds of beats are available
  • File Beat
  • Packet Beat

Adding Elastic Stack to the application