CRD’s and Operators

• CRD (Custom Resource Definition): This defines the blueprint of custom resource

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # name must match the spec fields below, and be in the form: <plural>.<group>
  name: DBCluster.resources.qt.com
spec:
  # group name to use for REST API: /apis/<group>/<version>
  group: resources.qt.com
  # list of versions supported by this CustomResourceDefinition
  versions:
    - name: v1
      # Each version can be enabled/disabled by Served flag.
      served: true
      # One and only one version must be marked as the storage version.
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                replicas:
                  type: integer
                version:
                  type: string
                svc:
                  type: string
                sc:
                  type: string
  # either Namespaced or Cluster
  scope: Namespaced
  names:
    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
    plural: mysqlclusters
    # singular name to be used as an alias on the CLI and for display
    singular: mysqlcluster
    # kind is normally the CamelCased singular type. Your resource manifests use this.
    kind: DBCluster
    # shortNames allow shorter string to match your resource on the CLI
    shortNames:
    - mysql

• Custom Resource: This the instance of blueprint

---
apiVersion: resources.qt.com/v1
kind: DBCluster
metadata:
  name: nop-mysql
spec:
  replicas: 3
  sc: default
  svc: nop-mysql-svc

• Operator: This works to reconcile or maintains the state. Operators can be developed in GO lang or python

Sample operator

Here is a sample Kubernetes Operator written in Python using the kopf (Kubernetes Operator Pythonic Framework) library. This example demonstrates how to manage a custom resource (CR) called ExampleResource.

1. Define the Custom Resource Definition (CRD)

Save the following YAML as example-crd.yaml to define the CRD:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: exampleresources.example.com
spec:
  group: example.com
  names:
    plural: exampleresources
    singular: exampleresource
    kind: ExampleResource
  scope: Namespaced
  versions:
    - name: v1alpha1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                replicas:
                  type: integer
                image:
                  type: string

Apply the CRD using kubectl:

kubectl apply -f example-crd.yaml

2. Implement the Operator Logic in Python

Save the following Python code as example-operator.py:

import kopf
import kubernetes

@kopf.on.create("example.com", "v1alpha1", "exampleresources")
def create_fn(spec, **kwargs):
    # Extract spec fields from the custom resource.
    replicas = spec.get('replicas', 1)
    image = spec.get('image', 'nginx')

    # Define a Kubernetes Deployment based on the custom resource.
    deployment = {
        "apiVersion": "apps/v1",
        "kind": "Deployment",
        "metadata": {
            "name": "example-deployment",
            "namespace": kwargs['namespace'],
        },
        "spec": {
            "replicas": replicas,
            "selector": {
                "matchLabels": {"app": "example-app"}
            },
            "template": {
                "metadata": {
                    "labels": {"app": "example-app"}
                },
                "spec": {
                    "containers": [{
                        "name": "example-container",
                        "image": image,
                    }]
                }
            }
        }
    }

    # Use Kubernetes API to create the Deployment.
    api = kubernetes.client.AppsV1Api()
    api.create_namespaced_deployment(namespace=kwargs['namespace'], body=deployment)

    return {"message": f"Deployment created with {replicas} replicas and image {image}"}

3. Deploy and Run the Operator

• Install dependencies:

pip install kopf kubernetes

• Run the operator:

kopf run example-operator.py

4. Create a Custom Resource (CR)

Save this YAML as example-cr.yaml:

apiVersion: example.com/v1alpha1
kind: ExampleResource
metadata:
  name: example-resource
spec:
  replicas: 3
  image: nginx:latest

Apply the CR to trigger the operator:

kubectl apply -f example-cr.yaml

Explanation

• CRD defines a new resource type (ExampleResource) with fields like replicas and image.
• Operator Logic listens for creation events of ExampleResource and creates a corresponding Deployment.
• Reconciliation Loop ensures that changes to CRs are reflected in the cluster.

This simple operator can be extended to handle updates, deletions, and more complex logic.

Citations:
[1] https://blog.palark.com/writing-a-kubernetes-operator-in-python-without-frameworks-and-sdk/
[2] https://dev.to/ashokan/build-a-kubernetes-operator-with-python-27e1
[3] https://www.groundcover.com/blog/kubernetes-operator
[4] https://kubernetes.io/docs/concepts/extend-kubernetes/operator/
[5] https://www.cncf.io/blog/2022/06/15/kubernetes-operators-what-are-they-some-examples/
[6] https://spacelift.io/blog/kubernetes-operator
[7] https://konghq.com/blog/learning-center/what-is-a-kubernetes-operator
[8] https://www.techtarget.com/searchitoperations/tutorial/How-to-build-a-Kubernetes-operator

Service Mesh

• Service Mesh
• Popular Service mesh
  • istio
  • linkerd
  • consul
  • traefik
• Service Mesh Offerings by Cloud Provider (Managed K8s)
  • Azure:
    • Azure Service Mesh: This is based on OSM (Open Service Mesh)
    • AKS with Istio
    • Consul
  • AWS:
    • AWS App Mesh: This is written by AWS for EKS and ECS
    • EKS with istio addon
    • Consul
  • GCP:
    • Anthos: This is istio as a serivce offering
    • Consul

Using istio

• Creating an AKS Cluster with istio add on

export CLUSTER=myAksCluster
export RESOURCE_GROUP=myResourcegroup
export LOCATION=eastus
az group create --name ${RESOURCE_GROUP} --location ${LOCATION}
az aks create --resource-group ${RESOURCE_GROUP} --name ${CLUSTER} --enable-asm --generate-ssh-keys

# verify
az aks show --resource-group ${RESOURCE_GROUP} --name ${CLUSTER}  --query 'serviceMeshProfile.mode'

# get credentials
az aks get-credentials --resource-group ${RESOURCE_GROUP} --name ${CLUSTER}

# view the pods by istion namespace
kubectl get pods -n aks-istio-system

• Istio Architecture
• Istio
  • Traffic Managment
  • Security
• Create a simple http and nginx deployments with common tags and service which matches based on common tags

apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpd
spec:
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
        version: v1
    spec:
      containers:
      - name: httpd
        image: httpd:latest
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
        version: v2
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: my-web
spec:
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 80

• Goal: 80% traffic to httpd and 20% traffic to nginx
• Step 1: Create a DestinationRule

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: bookinfo-ratings
spec:
  host: ratings.prod.svc.cluster.local
  trafficPolicy:
    loadBalancer:
      simple: LEAST_REQUEST

• step 2: Create a Virtual Service with traffic splitter

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: web-virtual-service
spec:
  hosts:
  - my-web.default.svc.cluster.local
  http:
  - route:
    - destination:
        host: my-web.default.svc.cluster.local
        subset: httpd
      weight: 80
    - destination:
        host: my-web.default.svc.cluster.local
        subset: nginx
      weight: 20

• Step 3: Create a gateway

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: my-gateway
  namespace: some-config-namespace
spec:
  selector:
    app: my-gateway-controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "ns1/*"
    - "ns2/foo.bar.com"

• Enabling external gateway in azure
• Watch classroom recording for using istio