start

This is an old revision of the document!


SHIFT-WIKI - Sjoerd Hooft's InFormation Technology

This WIKI is my personal documentation blog. Please enjoy it and feel free to reach out through blue sky if you have a question, remark, improvement or observation.


Using VS Code and Lens to connect to a private AKS cluster

Summary: This wiki page explains how to use VS Code and Lens to connect to a private AKS cluster.
Date: 15 September 2025

There are many ways to connect to a private AKS cluster. This page assumes you have a jumpbox VM in the same virtual network as the AKS cluster, and that you can connect to the jumpbox using Azure Bastion. Using VS Code with the extensions explained below is the easiest way to connect to a private AKS cluster, but also has it's limitations. For that reason, I'll also explain how to use Lens to connect to the private AKS cluster.

Design

See here the design of the network setup to connect to a private AKS cluster using a jumpbox through Azure Bastion. For simplicity, setup and requirements that are not relevant for this page, like private DNS, are left out:

Network overview of an private AKS cluster


VS Code

To use VS Code to connect to a private AKS cluster, make sure you have the 'Remote Explorer' and 'Kubernetes' extensions installed. Then, follow these steps:

  • Optional: If applicable, activate the required Privileged Identity Management (PIM) groups to assign permissions for your account. You need at least 'Reader' permissions on the resource group where the bastion is located, and, depending on what you'll need to do, 'Reader' or 'Contributor' permissions on the resource group where the jumpbox and AKS cluster are located.
  • Use the following Azure CLI commands to connect to Azure and set the subscribtion to the one with the bastion:
    # We need to use the azure cli as creating the tunnel to the jumpbox will only work with the azure cli
    az logout
    # Disable the subscription selector feature as well as sign in with WAM
    az config set core.login_experience_v2=off
    az config set core.enable_broker_on_windows=false
    az login --tenant 7e4an71d-a123-a123-a123-abcd12345678
    # Set the subscription to the bastion subscription
    az account set --subscription aa123456-a123-a123-a123-abcd12345678
  • Use the following Azure CLI command to create a ssh tunnel to the jumpbox through the bastion:
    az network bastion tunnel --name bas-001 --resource-group rg-bastion --target-resource-id /subscriptions/aa123456-a123-a123-a123-abcd12345678/resourceGroups/rg-cluster/providers/Microsoft.Compute/virtualMachines/vm-jumpbox --resource-port 22 --port 50022
  • To use the 'Remote Explorer' extension, you need to add an entry to your ssh config file (usually located at `C:\Users\<your-username>\.ssh\config`):
    # Read more about SSH config files: https://linux.die.net/man/5/ssh_config
    Host vm-jumpbox
        HostName 127.0.0.1
        User azadmin
        Port 50022
        IdentityFile C:/Users/sjoer/.ssh/private_cluster.pem
    • Note that the `IdentityFile` must point to the private key that matches the public key configured on the jumpbox VM.
  • In VS Code, open the Remote Explorer extension from the activity bar → SSH Targets → Open 127.0.0.1 (linux) in a new window
    • This will open a new VS Code window connected to the jumpbox VM. You can now open a terminal in this window and run commands on the jumpbox VM. To be able to connect to the AKS cluster, you need to install various tools on the jumpbox first. Note that all of the command below can be obtained by going to your cluster in the Azure portal and then to 'Connect':
# Install the Azure CLI: https://learn.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest
# Download and install the Microsoft signing key
sudo mkdir -p /etc/apt/keyrings
curl -sLS https://packages.microsoft.com/keys/microsoft.asc |
    gpg --dearmor |
    sudo tee /etc/apt/keyrings/microsoft.gpg > /dev/null
sudo chmod go+r /etc/apt/keyrings/microsoft.gpg
 
# Add the Azure CLI software repository
AZ_REPO=$(lsb_release -cs)
echo "deb [arch=`dpkg --print-architecture` signed-by=/etc/apt/keyrings/microsoft.gpg] https://packages.microsoft.com/repos/azure-cli/ $AZ_REPO main" |
    sudo tee /etc/apt/sources.list.d/azure-cli.list
 
# Update repository information and install the azure-cli package
sudo apt-get update
sudo apt-get install azure-cli
 
# Install kubectl and kubelogin: https://learn.microsoft.com/en-us/cli/azure/aks?view=azure-cli-latest#az-aks-install-cli
sudo az aks install-cli
 
# Login to Azure
az login
 
# Set subscription to the one with the AKS cluster and login to the AKS cluster
az account set --subscription aa123456-a123-a123-a123-abcd12345679
az aks get-credentials --admin --name aks-privatecluster --resource-group rg-privatecluster --overwrite-existing
 
# Use kubelogin plugin for authentication
kubelogin convert-kubeconfig -l azurecli
 
# Verify your aks connection
kubectl cluster-info
  • You can now use the 'Kubernetes' extension to view and manage the AKS cluster. Open the 'Kubernetes' extension from the activity bar → Click on the refresh button in the 'Clusters' section to load the clusters from your kubeconfig file. You should see your AKS cluster listed there. You can now expand the cluster to view its resources and perform various actions like viewing logs, executing commands in pods and more.
Note that some of the above steps can be automated using VS Code Tasks.

Lens

Lens is a popular Kubernetes IDE that makes it easy to manage and monitor Kubernetes clusters. You can download it from here. To connect to a private AKS cluster using Lens, follow these steps:

  • Lens automatically uses your local kube config to connect to clusters. To use Lens to connect to a privatecluster first connect to the privatecluster using the jumpbox as explained above. Then, copy the kubeconfig file from the jumpbox to your local machine. From your home directory on the jumpbox, use cat .kube/config to display the content of the kubeconfig file. Copy the content and save it to a file on your local machine, for example C:\Users\sjoer\.kube\privatecluster-config
  • Open the file in your VS Code and make the following changes:
    • Change server: https://aks-privatecluster-m8k6low2.privatelink.westeurope.azmk8s.io:443 to server: https://localhost:6443
    • Add the line insecure-skip-tls-verify: true under the cluster section:
      apiVersion: v1
      clusters:
      - cluster:
          certificate-authority-data: LS1QLSJBXXX
          server: https://localhost:6443
        insecure-skip-tls-verify: true
        name: aks-privatecluster
      contexts:
      - context:
          cluster: aks-privatecluster
          namespace: appops
          user: clusterAdmin_rg-privatecluster_aks-privatecluster
        name: aks-privatecluster-admin
      current-context: aks-privatecluster-admin
      kind: Config
      preferences: {}
      users:
      - name: clusterAdmin_rg-privatecluster_aks-privatecluster
        user:
          client-certificate-data: LS1QLSJBNMXXXLS0tCg==
          client-key-data: LS1QLSJBNMXXX
          token: xxx
  • In Lens, go to File → Add Cluster → Select kubeconfig file and select the file you just created. You should now see the AKS cluster in Lens.
  • Before you can manage it, you first need to create a ssh tunnel to the jumpbox through the bastion:
    az network bastion ssh --name bas-001 --resource-group "rg-bastion" --target-resource-id /subscriptions/aa123456-a123-a123-a123-abcd12345678/resourceGroups/rg-cluster/providers/Microsoft.Compute/virtualMachines/vm-jumpbox --auth-type AAD -- -L 6443:aks-privatecluster-m8k6low2.privatelink.westeurope.azmk8s.io:443
Note: The command above is different from the previous command as it used a ssh tunnel. Note that it also uses AAD authentication which is optional. A privatekey as setup before in the ssh config will work just as well.

This wiki has been made possible by:

2025/09/21 10:35

How to Clone a PVC in Kubernetes

Summary: This wiki page shows how to clone a Persistent Volume Claim (PVC) in Kubernetes.
Date: 14 September 2025

When doing an Argo CD sync I got an error on one of our PVCs. For one of our applications we upgraded the storage class but hadn't had the time yet to convert the actual PVC:

one or more objects failed to apply, reason: PersistentVolumeClaim "seq" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims   core.PersistentVolumeClaimSpec{    ... // 2 identical fields    Resources: {Requests: {s"storage": {i: {...}, Format: "BinarySI"}}},    VolumeName: "pvc-2bc75dce-01e6-4f8b-ae06-3fc6c6657dac", -  StorageClassName: &"default", +  StorageClassName: &"managed-premium",    VolumeMode: &"Filesystem",    DataSource: nil,    ... // 2 identical fields   }

As you can see, the storage class 'default' is changed to 'managed-premium', but unfortunately, this cannot be done online in Kubernetes, as that setting is immutable. Follow the procedure below to use 'korb' to quickly clone the PVC to a new one with the correct storage class.

Current Situation

This is the current pvc manifest:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: seq
  namespace: appops
  uid: 33dd11a4-e97e-4ce2-85e5-94efbad9e087
  resourceVersion: '737538059'
  creationTimestamp: '2024-08-07T14:57:02Z'
  labels:
    app: seq
    chart: seq-2024.3.1
    heritage: Helm
    k8slens-edit-resource-version: v1
    release: seq
  annotations:
    argocd.argoproj.io/tracking-id: appops:/PersistentVolumeClaim:appops/seq
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    volume.beta.kubernetes.io/storage-provisioner: disk.csi.azure.com
    volume.kubernetes.io/selected-node: aks-system-12344567-vmss000000
    volume.kubernetes.io/storage-provisioner: disk.csi.azure.com
  finalizers:
    - kubernetes.io/pvc-protection
  selfLink: /api/v1/namespaces/appops/persistentvolumeclaims/seq
status:
  phase: Bound
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Ti
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Ti
  volumeName: pvc-2bc75dce-01e6-4f8b-ae06-3fc6c6657dac
  storageClassName: default
  volumeMode: Filesystem

Note that the disk is 1TB in size. From the monitoring we see that it is 82% full.

Approach

There are several ways to clone a PVC, but I decided to create a quick clone using korb. Korb is a command-line tool to clone Kubernetes Persistent Volume Claims (PVCs) and migrate data between different storage classes. It supports various strategies for copying data, including snapshot-based and copy-twice methods.

Follow the steps below:

  • Check the latest release from: the korb release page
  • Use the following commands to install the korb binary on your linux system, and make sure to replace the version number if a newer one is available:
    # Download the latest release and make it executable
    curl -LO https://github.com/BeryJu/korb/releases/download/v2.3.4/korb_2.3.4_linux_amd64.tar.gz
    tar -xvzf korb_2.3.4_linux_amd64.tar.gz
    sudo mv korb /usr/local/bin/korb
    # Scale the application with the pvc to 0 replicas
    kubectl scale deployment seq --replicas=0 -n appops
    # Start the clone using korb. Note that the container image parameter is only necessary when working on a private cluster
    korb --new-pvc-storage-class=managed-premium --strategy=copy-twice-name --new-pvc-namespace=appops --source-namespace=appops --container-image=acreuwprd.azurecr.io/docker/beryju/korb-mover:v2 seq
    # Once the clone is ready, scale the application back to 1 replica
    kubectl scale deployment seq --replicas=1 -n appops

Note the following:

  • The korb command above was done on a private (Azure Kubernetes) cluster, so I had to add the image to the container registry before the command worked. You can do that with the following commands (replace the acr name with your own):
    az login
    docker pull ghcr.io/beryju/korb-mover:v2
    docker tag ghcr.io/beryju/korb-mover:v2 acreuwprd.azurecr.io/docker/beryju/korb-mover:v2
    docker push acreuwprd.azurecr.io/docker/beryju/korb-mover:v2
  • The strategy 'copy-twice-name' means that the pvc will first be cloned to a temporary pvc with a temporary name, and then right away to the final pvc with the original name. This works best in an environment with Argo CD which tracks the pvc by name.
  • The clone process from the standard storage class to the managed-premium storage class took 3 hours and 20 minutes for a 1 TB pvc that was 82% full. The second clone (to the final pvc with the original name) was some faster, but still took a long time. Keep that in mind when planning a maintenance window.

Result

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: seq
  namespace: appops
  uid: 1946a79c-37aa-4f01-8ce0-ed090d2b9b67
  resourceVersion: '743857155'
  creationTimestamp: '2025-09-10T22:24:19Z'
  labels:
    app: seq
    chart: seq-2024.3.1
    heritage: Helm
    k8slens-edit-resource-version: v1
    release: seq
  annotations:
    argocd.argoproj.io/tracking-id: appops:/PersistentVolumeClaim:appops/seq
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    volume.beta.kubernetes.io/storage-provisioner: disk.csi.azure.com
    volume.kubernetes.io/selected-node: aks-system-12344567-vmss000000
    volume.kubernetes.io/storage-provisioner: disk.csi.azure.com
  finalizers:
    - kubernetes.io/pvc-protection
  selfLink: /api/v1/namespaces/appops/persistentvolumeclaims/seq
status:
  phase: Bound
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 1Ti
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Ti
  volumeName: pvc-1946a79c-37aa-4f01-8ce0-ed090d2b9b67
  storageClassName: managed-premium
  volumeMode: Filesystem
2025/09/14 14:29

How do Kubernetes Probes Work

Summary: This wiki page explains the usage of kubernetes probes.
Date: 16 August 2025

Even though they are pretty good described in the documentation and a availability of a lot of additional resources, I found it hard to find a simple clear overview without going too much in depth. This wiki page aims to do just that.

Probes

Kubernetes probes are Kubernetes capabilities that allow containerised applications to be more reliable and robust. There are two main probes:

  • Liveness Probe: This probe checks if the application is alive. If the liveness probe fails, Kubernetes will restart the pod.
  • Readiness Probe: This probe checks if the application is ready to serve traffic. If the readiness probe fails, Kubernetes will stop sending traffic to the pod until it is ready again.

Additionally, there is a third type of probe:

  • Startup Probe: This probe checks if the application has started successfully. If the startup probe fails, Kubernetes will kill the pod and start a new one. This is useful for applications that may take a long time to start up. The startup probe is only run once, at the start of the pod, and is not run again after that.

Probe Parameters

All probes share the same parameters:

  • initialDelaySeconds: Number of seconds after the container has started before the probe is scheduled. The probe will first fire in a time period between the initialDelaySeconds value and (initialDelaySeconds + periodSeconds). For example if the initialDelaySeconds is 30 and the period seconds is 100 seconds then the first probe will fire at some point between 30 and 130 seconds.
  • periodSeconds: The delay between performing probes.
  • timeoutSeconds: Number of seconds of inactivity after which the probe times-out and the application is assumed to be failing.
  • failureThreshold: The number of times that the probe is allowed to fail before the liveness probe restarts the container (or in the case of a readiness probe marks the pod as unavailable). A startupProbe should have a higher failure threshold to account for longer startup times.
  • successThreshold: The number of times that the probe must report success after it begins to fail in order to reset the probe process. The successThreshold parameter has no impact on a liveness probe.

In yaml, the configuration of a probe looks like this, with the default values:

livenessProbe:
  httpGet:
    path: /healthz
    port: liveness-port
  initialDelaySeconds: 0
  periodSeconds: 10
  timeoutSeconds: 1
  failureThreshold: 3
  successThreshold: 1
Note that the configuration for readiness and startup probes is the same, except for the name of the readinessProbe or startupProbe key.

The Schedule

I've found the graphical representation of the probe schedule below very helpful in understanding how the probes work and interact:

Graphical representation of probe schedules


Useful Links

2025/08/16 15:02

How to manually install Grafana Plugins into the Kubernetes Helm Chart Pods

Summary: This wiki page shows how to manually install Grafana plugins into the Kubernetes Helm Chart Pods.
Date: 10 August 2025

In this wiki page I'll show you the steps to manually install Grafana plugins into the Kubernetes Helm Chart Pods. This includes the following steps:

  • Update the helm values file to allow for unsigned plugins.
  • Use kubectl to copy the plugin files into the pod.
  • Restart the Grafana pod.

Helm Values File

We'll be using the helm chart from the Grafana Community Kubernetes Helm Charts repository. The default values file can be found here.

The value file should get updates for persistence and the name of the plugin to allow loading unsigned plugins:

```yaml persistence:

enabled: true

grafana.ini:

plugins:
  allow_loading_unsigned_plugins: timeseries

```

Note that the plugin we're installing is a custom plugin called 'timeseries'.

Use kubectl to copy the plugin files

First of all we need to know the location of the plugin files in the Grafana pod. This is done by checking the Grafana settings for the plugin path:

  • Go to grafana → administration → general → settings → paths → plugins
    • Note that the path, /var/lib/grafana/plugins, is located at the PVC, which is mounted at /var/lib/grafana. This is important as otherwise the data would be lost on every pod restart.
  • Use the following command to copy the plugin zip file into the pod: kubectl cp /mnt/c/Users/sjoerd/plugin/time-series-datasource_V1.2.zip grafananamespace/grafana-pod-5678fgh89-1a2bcd:/var/lib/grafana/plugins -c grafana
    • Note:
      • Make sure to replace `grafananamespace` and `grafana-pod-5678fgh89-1a2bcd` with the actual namespace and pod name.
      • -c grafana specifies the container name in the pod, which is necessary if there are multiple containers in the pod.
  • Now use kubectl to enter the pod and unzip the plugin file:
    • kubectl exec -it grafananamespace/grafana-pod-5678fgh89-1a2bcd -c grafana -- /bin/sh
    • cd /var/lib/grafana/plugins/
    • unzip time-series-datasource_V1.2.zip -d timeseries
      • This will create a directory called 'timeseries' with the plugin files

If you would do the kubectl cp command from a Windows powershell or cmdline you could run into errors with kubectl not recognizing the file path. To solve this, you can change your current directory to where the plugin is located and run the command with just the filename without the path.

Restart the Grafana Pod

After installing the plugin, you need to restart the Grafana pod for the changes to take effect. You can do this by scaling the deployment down to 0 and then back up to 1:

```bash kubectl scale deployment grafana –replicas=0 -n grafananamespace kubectl scale deployment grafana –replicas=1 -n grafananamespace ```

2025/08/16 15:02
start.1748779162.txt.gz · Last modified: by 127.0.0.1