An Overview of Metric Collection approaches (Push and Pull)
- In Push Based monitoring systems, emitted metrics or events are sent directly from producing application or from a logical agent to the collecting server
- Some examples of this approach is used in Elastic Search, LogStash and Kibana (Elastic Stack)
- In contrast, pull based monitoring systems collect metrics directly from applications or from proxy processes that makes those metrics available
- Prometheus is a pull-based monitoring system & it also provides a way of ingesting pushed metrics by using a gateway that converts from push to pull.
Prometheus is time series based open source monitoring system.
It collects data by sending HTTP requests to hosts and services on metric endpoints, which it makes available for analysis and alerting using a powerful query language
Refer Here for official docs
Prometheus has joined Cloud Native Computing Foundation (CNCF) in year 2016.
The Prometheus ecosystem is composed of several components
High level overview of main components of Prometheus eco system
As we can see in the image above,
- Prometheus Server collects time series data, stores it and makes it available for querying and send alerts based on it
- The Alert Manager recieves alert triggers from Prometheus and handling routing and dispatching of events
- The Push gatewayy handles the metrics that have been pushed from short-lived jobs (cron jobs or batch jobs)
- Applications that support the Prometheus exposition format make internal state available through and HTTP endpoint
- Community driven exporters expose metrics from applications that do not support prometheus natively
- First-Party and Third-party dashboarding provide a visualization of collected
Prometheus when it was originally created at SoundCloud was influenced from Google’s Borgmon.
- Scraping plain text from metrics endpoints
- exporters as proxies for metrics collections
- time series as multi dimensional vectors
- use of ruleset evaluations
Exposing Internal State with exporters
- Not all applications are built with Prometheus compatible instrumentations, Sometimes no metrics are exposed at all, In these case we can rely of exporter.
- Exporter is nothing more that a piece of software that collects data from service or application and exposes via HTTP in the Prometheus
- Node Exporter is one of most commonly used exporters, which presents number of kernel statistics such as disk I/O, CPU, Memory, network, filesystem usage and much more.
- We have exporters for pretty much every thing Refer Here
- Scrape: The HTTP GET request made by the Prometheus server to the observed system for metric collection is called as scrape.
- If you are one writing the service, the best option is to instrument the code directly using a Prometheus client library.
- There are official libraries for
- There are community driven client libraries for almost all the programming languages Refer Here
- If you want to develop exporters Refer Here
- Alerting routes: There are multiple out of the box integrations available for most common use case such as
- For more info Refer Here
Visualization Data collected
- Prometheus exposes a well defined AP where PromQL queries can provide raw data for visualizations
- As of now the best external software for visualization used along with Prometheus is Grafana.
- The Prometheus server also ships with two internal visualization components
- Experssion browser: Here we can run PromQL directly to quickly query and visualize data instantly
- Consoles: These are web pages that ar built using the Golang templating language and served by Prometheus server itself.