Cheatsheet Kubernetes

Summary: Kubernetes hints, tips, oneliners and best practices.
Date: 15 December 2024

cheatsheet, kubernetes

Terminology

Pods
Basic scheduling unit that holds one or more containers.

Nodes
Machines (physical or virtual) in the cluster where pods are scheduled.

Cluster
Collection of nodes and associated resources.

Kubelet
An agent running on each node, responsible for managing the node and its containers.

Kubernetes Controller Manager
Manages controllers to regulate the state of the system.

Kube Proxy
Maintains network rules to allow communication between pods and external traffic.

etcd
Consistent and highly-available key-value store used for all cluster data.

API Server
Serves the Kubernetes API and is the primary entry point for administrative tasks.

Scheduler
Assigns pods to nodes based on resource requirements and other constraints.

Controller
Maintains the desired state of the system, such as ensuring the correct number of replicas for a particular application.

Service
Provides a consistent way to access a set of pods.

Namespace
A way to divide cluster resources between multiple users.

Volumes
Kubernetes supports various types of storage volumes, providing data persistence for pods.

Secrets and ConfigMaps
Mechanisms to manage sensitive information and configuration data separately from application code.

Deployment
A higher-level resource that manages updates to applications by handling the deployment and scaling of pods.

StatefulSets
Manages stateful applications, ensuring stable network identities and persistent storage for pods.

DaemonSets
Ensures that specific pods run on all (or specific) nodes for cluster-wide tasks like logging or monitoring.

Jobs and CronJobs
Run short-lived or scheduled tasks within the cluster.

Ingress
Manages external access to services, typically HTTP.

Network Policies
Define how groups of pods can communicate with each other and other network endpoints.

Horizontal Pod Autoscaler
Automatically adjusts the number of replica pods to handle varying load.

Vertical Pod Autoscaler
Adjusts the resources allocated to individual pods based on their usage.

Operators
A way to package, deploy, and manage applications using Kubernetes APIs and controllers.

Kubectl
The command-line interface to interact with Kubernetes clusters.

Kubectl

Basic commands and information

Get cluster information
kubectl cluster-info

List all k8s objects from Cluster Control plane
kubectl get all --all-namespaces

Deploy and delete a manifest file
kubectl apply -f manifest.yaml
kubectl delete -f manifest.yaml

Deploy and delete a manifest folder

kubectl apply -f kube-manifests/
kubectl delete -f kube-manifests/
# Recursive
kubectl apply -R -f kube-manifests/
kubectl delete -R -f kube-manifests/

Deploy to a namespace
kubectl apply -f manifest.yaml -n dev1

Get the kubectl version
kubectl version

Nodes

List all nodes

kubectl get nodes
kubectl get nodes -o wide

Get detailed information about a node
kubectl describe node aks-agentpool-20417106-vmss000001

Remove taint from a node

kubectl taint nodes aks-agentpool-20417106-vmss000001 CriticalAddonsOnly=true:NoSchedule-

Get node resource performance

kubectl top nodes
kubectl top nodes --sort-by=cpu
kubectl top nodes --sort-by=memory
# Sort from low to high
kubectl top nodes --sort-by=cpu --no-headers | sort -k3 -n
kubectl top nodes --sort-by=memory --no-headers | sort -k3 -n

Connect to nodes in a private aks cluster

# Based on https://learn.microsoft.com/en-us/azure/aks/node-access where is explained on how to connect to aks nodes. However, this does not work for a private cluster as the container is not available. You can, assuming that you've added the container to your private container registry, use that one:
# Get all the nodes
kubectl get nodes -o wide
# Start the debug command to connect to the node using an image from a private container registry
kubectl debug node/aks-npuser001-34232393-vmss000001 -it --image=acreuwprd.azurecr.io/docker/docker/library/busybox:1.37

Resources

Get & Describe Limits

kubectl get limits -n dev3
kubectl describe limits default-cpu-mem-limit-range -n dev3

Get Resource Quota

kubectl get quota -n dev3
kubectl describe quota ns-resource-quota -n dev3

Check cpu and memory in pods, notice that this is actually the resources of the host
# cpu
cat /proc/cpuinfo | grep proc
# memory
free -h

Namespaces

List all namespaces and work with namespaces for other objects

kubectl get namespaces
kubectl get ns
kubectl get pods --all-namespaces

Create a namespace

kubectl create namespace dev1
kubectl create namespace dev2

Delete a namespace

kubectl delete ns dev1
kubectl delete ns dev2

Set a default namespace

# Set the default namespace and expected output
kubectl config set-context --current --namespace=ops
Context "aks-cluster" modified.

Pods

List all pods
kubectl get pods
kubectl get po
List all pods from a specific namespace
kubectl get pods -n dev1

Get logging from a pod

kubectl logs -f podname
# pods can have changing names, so you can use this command if you don't know the podname:
kubectl logs -f $(kubectl get po | egrep -o 'external-dns[A-Za-z0-9-]+')

Get detailed information about a pod

kubectl describe pod podname
kubectl describe pod myapp1-deployment-5bc58f6848-7vm2v

Get pod specifications like cpu and memory
kubectl get pod <pod-name> -o yaml

Get pod resource performance

kubectl top pods
kubectl top pods --sort-by=cpu
kubectl top pods --sort-by=memory

Deployments

List all deployments
kubectl get deployments
kubectl get deploy

Restart a deployment

kubectl rollout restart deployment/kube-prometheus-stack-grafana

Restart all deployments in a namespace

kubectl rollout restart deployment -n shared

Apply a manifest

kubectl apply -f kube-prometheus-stack.yaml --server-side -n shared
kubectl apply -f kube-prometheus-stack.yaml --server-side --force-conflicts -n shared

Services

List all services
kubectl get services
kubectl get svc

List all services from all namespaces

kubectl get services --all-namespaces
# Sorted on name
kubectl get services --all-namespaces --sort-by=.metadata.name
# Sorted on type
kubectl get services --all-namespaces --sort-by=.spec.type
# Get all services of type LoadBalancer
kubectl get services --all-namespaces | grep LoadBalancer

List services with a specific label

kubectl get service -l app.kubernetes.io/name=ingress-nginx --namespace ingress-basic

Describe a service

kubectl describe svc proxy-public --namespace dev1

StatefulSets

List all StatefulSets
kubectl get statefulsets
kubectl get sts

Kill all the pods in a statefulSet by setting the number of replicas to 0
kubectl scale statefulset myapp1 --replicas=0

Restart a statefulset
kubectl rollout restart statefulset/myapp1

Restart all statefulsets in a namespace
kubectl rollout restart statefulset -n shared

Storage

List all storage classes
kubectl get storageclasses
kubectl get sc

List all persistent volumes claims
kubectl get pvc

List all persistent volumes (the actual storage)
kubectl get pv

Delete a persistent volume
kubectl delete pv my-pv

List all storage information at once
kubectl get sc,pvc,pv

Networking

Get all ingress
kubectl get ingress

Secrets

List all secrets
kubectl get secrets

Create a secret

kubectl create secret generic azure-config-file --from-file=azure.json

Decode a secret

echo "cGxhY2Vob2xkZXJwYXNzd29yZA==" | base64 --decode

Decode a secret with powershell

kubectl get secret argocd-initial-admin-secret --namespace ops -o json | ConvertFrom-Json | select -ExpandProperty data | % { $_.PSObject.Properties | % { $_.Name + [System.Environment]::NewLine + [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($_.Value)) + [System.Environment]::NewLine + [System.Environment]::NewLine } }

MISC

Connect to MySQL using Kubectl by installing a sql client pod

kubectl run -it --rm --image=mysql:8.0 --restart=Never mysql-client -- mysql -h akswebappdb201.mysql.database.azure.com -u dbadmin -p<password>

Reset Grafana admin password from within the pod

# Start k8s vscode extension -> vtxops -> configuration -> secrets -> kube-prometheus-stack-grafana -> Note down the admin-password
# opsnamespace -> workloads -> pods -> kube-prometheus-stack-grafana-xxxx
# Open the terminal (click terminal icon next to the name)
grafana cli admin reset-admin-password <admin-password>

Check for kafka topics

# open a terminal on one of the kafka brokers (eg kafka-kafka-0)
./bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Get all resource kinds with their name from a manifest:
cat opentelemetry-operator.yaml | grep -i '^kind\|^  name:'

Loop commands with kubectl

Here are some examples I use to perform actions on multiple resources at once.

Remove all finalizers for kafka topics

kubectl get kafkatopic.kafka.strimzi.io -n shared -o name |
while read topic; do
    echo "Removing finalizer from $topic"
    kubectl patch $topic -n shared -p '{"metadata":{"finalizers":[]}}' --type=merge
done

Delete all jobs that start with “backup”

export NAMESPACE=shared
kubectl get jobs -n $NAMESPACE -o name | grep backup- |
while read job; do
    echo "Deleting $job"
    kubectl delete $job -n $NAMESPACE
done

AKS

Cloud Shell

Connect to AKS

az aks get-credentials --resource-group myResourceGroup --name myAKSCluster
# overwrite the existing context
az aks get-credentials --resource-group myResourceGroup --name myAKSCluster --overwrite-existing

Local