Completek8s Classroom notes 15/Oct/2023

Monitoring kubernetes

Prometheus

• Prometheus is an open source systems monitoring and alerting toolkit that was originally built at Soundcloud
• Prometheus have paved the way in montioring distributed systems and microservices.
• EcoSystem:
  • Alert Manager: This does alert handling
  • Grafana (Visualizations)
  • exporters
• Grafana is open source platform for monitoring and observability and it has ability to produce rich, interactiv and informative visualizations of data.
• Architecture
  

Components of Prometheus

• Data Collection – Targets and Exporters:
  • Prometheus collects metrics from configured targets at given intervals. Targets are endpoints from which Prometheus scrapes data.
  • Refer Here for the client libraries thta help in exposing endpoints
  • Exporters: Exporters are necessary when working with services that do not natively support prometheus metrics. They act as translators and pull metrics from source and then convert and expose them in a from Prometheus can understand Refer Here
• Retrieving metrics – Scrape
  • Prometheus retrives metrics from the targets at regular intervals, defind by scrape interval
• Storage: Time Series Database
  • Once the metrics are scraped, they are stored in Prometheus local time series database, Prometheus also can be configure to send data to remote storage systemcs
  • Generally in most of the cases we run prometheus in k8s cluster and this storage is handled via persitent volumes
• Service Discovery:
  • This plays a crucial role in automatically finding targets to scrape metrics from. This is essential
    • Dynamic Environments
    • Scale
    • Up-to-Date Environments
  • Types of Service Discoveries
    • Static configuration
    • File based Service Discovery
    • Cloud Providers and Orchestrations
      • Kubernetes SD
      • Consul SD
      • AWS SD
      • GCE SD
      • Azure SD
      • Docker SD
• Data Processing – PromQL
  • PromQL (Prometheus Query Language): this allows users to select and aggreage time-series data in real-time
• Visualisations: Grafana
  • Even though Prometheus provides its basic expression browser for visualizations, Grafana is used for powerful visualizations which help in creating dashboards
• Alert Manager
  • Alerting Rules: Prometheus evaludes alerting rules defind using PromQL and triggers alerts when conditions are met
  • Alert Manager: When an alert is fired, its sent to Alert Manager which is responsible for grouping, silencing or routing through various channels like email, PagerDuty, Slack, Teams etc
• Push Gateway: For short lived jobs that cannot be scrapped regularly, metrics can be pushed to pushgateway from where Prometheus can scrape them

Exercise: Install prometheus into k8s cluster using helm

• Refer Here for helm chart of prometheus community

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
kubectl create namespace prometheus

• Create a values.yaml

alertmanager:
  enabled: true
server:
  resources:
    limits:
      memory: 1Gi

• install helm with custom values

helm install prometheus prometheus-community/prometheus --namespace prometheus -f values.yaml

• Refer Here for helm chart docs
• To be done: fix pvc issue in the next session

Open Questions

• Argo with kustomize
• secerts vault or manager in kubernetes (aks/eks/gke)