Kubernetes Classroom Series – 17/Aug/2021

An Overview of Metric Collection approaches (Push and Pull)

  • In Push Based monitoring systems, emitted metrics or events are sent directly from producing application or from a logical agent to the collecting server Preview
  • Some examples of this approach is used in Elastic Search, LogStash and Kibana (Elastic Stack)
  • In contrast, pull based monitoring systems collect metrics directly from applications or from proxy processes that makes those metrics available Preview
  • Prometheus is a pull-based monitoring system & it also provides a way of ingesting pushed metrics by using a gateway that converts from push to pull.


  • Prometheus is time series based open source monitoring system.

  • It collects data by sending HTTP requests to hosts and services on metric endpoints, which it makes available for analysis and alerting using a powerful query language

  • Refer Here for official docs

  • Prometheus has joined Cloud Native Computing Foundation (CNCF) in year 2016.

  • The Prometheus ecosystem is composed of several components

  • High level overview of main components of Prometheus eco system Preview

  • As we can see in the image above,

    • Prometheus Server collects time series data, stores it and makes it available for querying and send alerts based on it
    • The Alert Manager recieves alert triggers from Prometheus and handling routing and dispatching of events
    • The Push gatewayy handles the metrics that have been pushed from short-lived jobs (cron jobs or batch jobs)
    • Applications that support the Prometheus exposition format make internal state available through and HTTP endpoint
    • Community driven exporters expose metrics from applications that do not support prometheus natively
    • First-Party and Third-party dashboarding provide a visualization of collected
  • Prometheus when it was originally created at SoundCloud was influenced from Google’s Borgmon.

    • Scraping plain text from metrics endpoints
    • exporters as proxies for metrics collections
    • time series as multi dimensional vectors
    • use of ruleset evaluations

Exposing Internal State with exporters

  • Not all applications are built with Prometheus compatible instrumentations, Sometimes no metrics are exposed at all, In these case we can rely of exporter. Preview
  • Exporter is nothing more that a piece of software that collects data from service or application and exposes via HTTP in the Prometheus
  • Node Exporter is one of most commonly used exporters, which presents number of kernel statistics such as disk I/O, CPU, Memory, network, filesystem usage and much more.
  • We have exporters for pretty much every thing Refer Here
  • Terminology:
    • Scrape: The HTTP GET request made by the Prometheus server to the observed system for metric collection is called as scrape.
  • Guidelines:
    • If you are one writing the service, the best option is to instrument the code directly using a Prometheus client library.
    • There are official libraries for
      • Go
      • Java
      • Python
      • Ruby
    • There are community driven client libraries for almost all the programming languages Refer Here
    • If you want to develop exporters Refer Here
  • Alerting routes: There are multiple out of the box integrations available for most common use case such as
    • email
    • HipChat
    • Slack
    • OpsGenie
    • PagerDuty
  • For more info Refer Here

Visualization Data collected

  • Prometheus exposes a well defined AP where PromQL queries can provide raw data for visualizations
  • As of now the best external software for visualization used along with Prometheus is Grafana.
  • The Prometheus server also ships with two internal visualization components
    • Experssion browser: Here we can run PromQL directly to quickly query and visualize data instantly
    • Consoles: These are web pages that ar built using the Golang templating language and served by Prometheus server itself.

Local Environment

  • Windows 10:

  • Next steps:

    • Installing minikube and using helm charts to bring up prometheus.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

About learningthoughtsadmin