Setting up Prometheus and Grafana Outside Kubernetes cluster

Installing kubernetes using kube admin

• Install docker on each node in kuberentes cluster

curl -fsSL https://get.docker.com -o get-docker.sh
sudo usermod -aG docker $(whoami)
cat <<EOF | sudo tee /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF
sudo systemctl enable docker
sudo systemctl daemon-reload
sudo systemctl restart docker

• Install kubeadm, kubelet and kubectl on all the nodes Refer Here
• Now login into master and setup kubeadm

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

• Make a note of commands to setup kubectl to work with non root user and copy kubeadm join command and execute on the nodes as root user

kubeadm join --token <token> <IP>:6443

Prometheus installation on ubuntu

• Login into the ubuntu system and download prometheus from Refer Here

sudo apt update
wget https://github.com/prometheus/prometheus/releases/download/v2.29.2/prometheus-2.29.2.linux-amd64.tar.gz

• Create Prometheus users

sudo useradd --no-create-home --shell /bin/false prometheus

• Create Prometheus Directories

sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus

• Now untar prometheus

tar xvzf prometheus-2.29.2.linux-amd64.tar.gz

• Now copy prometheus and promtool binary files to /usr/local/bin

sudo cp prometheus-2.29.2.linux-amd64/{prometheus,promtool} /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/{prometheus,promtool}

• After copying binary files copy the required libararies to the /etc/prometheus directory

cd prometheus-2.29.2.linux-amd64/
sudo cp -r consoles/ /etc/prometheus/
sudo cp -r console_libraries/ /etc/prometheus/

• Now change ownership of the configuration files

sudo chown -R prometheus:prometheus /etc/prometheus

• Now lets change ownership of /var/lib/prometheus

sudo chown -R prometheus:prometheus /var/lib/prometheus

• Now lets create a systemd service for prometheus

sudo nano /etc/systemd/system/prometheus.service

• Add the following content in the file

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

• Now lets reload daemon

sudo systemctl daemon-reload

• Lets change the prometheus configuration at sudo nano /etc/prometheus/prometheus.yml and the following content

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

• Now enable and start prometheus

sudo systemctl enable prometheus
sudo systemctl start prometheus
sudo systemctl status prometheus

• Now access the prometheus server with http://ip-address:9090
• Now Lets install grafana Refer Here

sudo apt-get install -y apt-transport-https software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
sudo apt update
sudo apt-get install grafana=8.1.3 -y
sudo systemctl enable grafana-server
sudo systemctl start grafana-server
sudo systemctl status grafana-server

• Now access grafana http://<public-ip>:3000 and the username and password is admin

• Now add datasource and Select Prometheus and enter http://prometheus_server_private_ip:9090 in this case as prometheus and grafana are on same server you can use http://localhost:9090

• Click Save & Test and you should see banner says DataSource is working

• Now test the setup by Clicking on the explore icon

• Now lets enable kube-state metrics on kubernetes master

git clone https://github.com/kubernetes/kube-state-metrics.git
cd kube-state-metrics
kubectl apply -f examples/standard/

• Now create a new file kube-state-metrics-nodeport.yaml with the following content

kind: Service
apiVersion: v1
metadata:
  namespace: kube-system
  name: kube-state-nodeport
spec:
  selector:
    app.kubernetes.io/name: kube-state-metrics
  ports:
  - protocol: TCP
    port: 8080
    nodePort: 30000
  type: NodePort

• Now execute kubectl apply -f kube-state-metrics-nodeport.yaml and execute curl localhost:30000/metrics to verify if the settings are working correctly or not
• Now lets configure the prometheus to scrap metrics from kube-state-metrics
• log in into prometheus server
• edit /etc/prometheus/prometheus.yml and add the following to scrape_config, please note you need to add private ip of k8s master

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'kubernetes'
    static_configs:
      - targets: ['10.128.0.31:30000']

• now restart prometheus
• Now import the dashboard with id 13332 into grafana