Interacting with k8s API

• How kube api server processes a Request
  
• Kubernetes makes accessing the API Server a little bit more convienient for specific usecases such as a Pod to send a request to k8s api can speak with service instead of IP Address using name kubernetes.default.svc. This service lives in default namespace
  

Restricting User Permissions

• Create a private key

openssl genrsa -out qtdevops.key

• Create and approve CertificateSigningRequest

openssl req -new -key qtdevops.key -out qtdevops.csr
# copy the base64 encoded of csr
cat qtdevops.csr | base64 | tr -d "\n"

• Create a certificate Signing Request

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: qtdevops
spec: 
  request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2lqQ0NBWElDQVFBd1JURUxNQWtHQTFVRUJoTUNRVlV4RXpBUkJnTlZCQWdNQ2xOdmJXVXRVM1JoZEdVeApJVEFmQmdOVkJBb01HRWx1ZEdWeWJtVjBJRmRwWkdkcGRITWdVSFI1SUV4MFpEQ0NBU0l3RFFZSktvWklodmNOCkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFNY1paYlg5a3hGb2Fsc3B0MUFNWEZHamdLQmxaUTBtVjMzVXM2VjkKOXBpb1pmdXdiRG9rMnBTUWtLY216d3pkOHB6S2FjUU1PdjV2Z24ybW01M0tSbS9sOUpGYUNuZFJxK2xUYUZPVgpXZ2M0RDYyb09RK1JjZE1yMGg3bk82c2d3ZFJ3c25idHlSWmZRWFVISWZtQXdzekVSQXlVYkhUS3Z1VzU4UVNkCm9UbFNEREFlb01YVDJmYktOUkZjb3dtWDVVS2wwQWkyTHZGaDhzeWs0MTZreUx6MGVpSnJiV29DcmxCei96TzMKVnV2WXhqTzkzMmdJL1QzN3hZYU9DK1JMWnBvQUl1QlNxaUhvVyt5QmdVaEsvVWxIV0xJZnF1S1NDVmtjWDJ2QwpUYzVrRHNiVDcvVGVqSHJWeGhjS1R1aWg2bHdpdWNkWDNvSjFNTk1NSU9MQVZVc0NBd0VBQWFBQU1BMEdDU3FHClNJYjNEUUVCQ3dVQUE0SUJBUUJxaE85MHNqanZ0dnBDeWJOc1QxNzN5ZGdkUzkxdzhmQjBnSDdPRG10VWhSV3oKd0t5RktpcnlMb2F1MFY0Y25xdjkxYlJKOUY3dzFDb2drVG1Mb2pZd1dTUVIyekxZUzZQZDJTbG1pTVBJLzN4YgpFdk1hWFBNWjdaaTdkT2tzMDV3T2hpME5LWW9MNkc3b2xVRU5MTngyeU9BRXpNR01rSHI1SUxZR1F1eEhaMHJECmN2SlkzRzhxQnNkMnpVd0NSWWx6OHlWMk96cnZWQm4rOEp1djdRS3BmcGttaDRZY0I3bWJCeHFUczFkYjNpWkgKRVcwSms5KzdZQWI0TWZKY25tUW5mRnBxVG9XdEliZVQ1QmU4NjNKcWdYcVJ5Rmp6eHoyUHB3Ym9lSzR4Mm4wVgpQWjd3cGo0VlBoSUsvbVpmWDV3dWM3SnFTK3NMMGdCbWw1b2UwM0tFCi0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: 86400
  usages:
    - client auth

• Apply the yaml
  
• Now as a kubernetes-admin user we can approve the certificate
  
• Get the crt file

 kubectl get csr johndoe -o jsonpath={.status.certificate} | base64 -d > johndoe.crt

• Create a Role and a Rolebinding

kubectl create role developer --verb=create --verb=get --verb=list --verb=delete --verb=update --resource=pods

kubectl create rolebinding developer-johndoe --role=developer --user=johndoe

• Add the user to kubeconfig

kubectl config set-credentials johndoe --client-key=johndoe.key --client-certificate=johndoe.crt --embed-certs=true
kubectl config set-context johndoe --cluster=kubernetes --user=johndoe
kubectl config use-context johndoe

• Note: We were getting the following error when we used the user qtdevops

error: You must be logged in to the server (Unauthorized)

Understanding Audit Logs

• k8s can store records for events triggered by end user for any requests
• Entries in audit log exist in JSON format and majorly record information about
  • Which event occured
  • Who triggered the event
  • When it was triggered
  • Which k8s component handled the request
• We need to define an audit policy
• The audit backend is responsible for storing the recorded audit events as defined by audit policy. We have two configurable options for backend
  • A log backend which writes events to file
  • A webhook backend which sends events to external service via HTTP(S)
• Refer Here for configuring audit log backends and Refer Here for audit policy

Lab Setup

• Create a kubernetes cluster
• create two pods/deployments
  • nginx
  • httpd
• add nginx ingress controller to your kubernetes cluster
• In this case we are using kubeadm so nginx ingress controller with baremetal exposed on nodePort Refer Here

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.2/deploy/static/provider/baremetal/deploy.yaml

• We have create ingress with two services and acessed over http and used the default https
• We created our own certificate and secret from that and used in ingress

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minimal-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  tls:
  - hosts:
      - securedingress.com
    secretName: securedingress
  rules:
  - host: securedingress.com
    http:
      paths:
      - path: /service1
        pathType: Prefix
        backend:
          service:
            name: service1
            port:
              number: 80
      - path: /service2
        pathType: Prefix
        backend:
          service:
            name: service2
            port:
              number: 80

• The challenge with this approach is to automate certificate renewals and getting new certificates
• This is where Letsencrypt Refer Here and cert-manager Refer Here can help