Understanding the New AKS Deployment Safeguards (2024)

Introduction

Last week at Build, we've introduced a new feature in Public Preview for Azure Kubernetes Service (AKS) called Deployment Safeguards.

Deployment Safeguards, as a part of Azure Policy for AKS, provides a configurable and fast way to make sure your Kubernetes deployment follow through best practices and limits that are set beforehand. In this article, we will explore how it works in real time and how can we use it to tailor AKS to your needs.

Playground Setup

For the sake of this article, I'll create a new cluster from scratch.

There are a few things we need to set up first.

For my test environment, I'm running these commands on WSL/Ubuntu with local Azure CLI.

If you're not logged in, executeaz loginand then choose the right subscription,

If you're using Azure CLI with Login Experience v2, you can just choose the subscription from the drop-down, and disregard the second command:

az loginaz account set -s "your-subscription-id"

AKS Deployment Safeguard is currently a preview feature, so we'll need to make sure our AKS extension is up-to-date:

az extension add --name aks-previewaz extension update --name aks-preview

Next, register the feature flag of Deployment Safeguards -

az feature register --namespace Microsoft.ContainerService --name SafeguardsPreview

This will take a couple of minutes; the end result should show asRegistered:

Next, refresh theMicrosoft.ContainerServiceresource provider so changes will be applied:

az provider register --namespace Microsoft.ContainerService

Create a new test resource group and AKS cluster -

az group create --name safeguard-test --location eastusaz aks create --name safeaks --resource-group safeguard-test --node-count 2 --location eastus --enable-addons azure-policy --safeguards-level Warning --safeguards-version v2.0.0 

This will create a new Azure Kubernetes Service (AKS) cluster, with 2 nodes.

This cluster will have theAzure Policy for AKS add-on enabled with Safeguard level set toWarningand version set to 2.0.0.

Node count is set to 2 to allow for faster creation and a bit of redundancy.

I have created 2 clusters, one with Safeguards level set toWarningand one withEnforcement.

I have set the safeguard level to Warning for the first cluster as we wish to experiment with it, Warning will notify us that a resource/yaml is out of policy but won't block it.

Setting safeguard level toEnforcementwill automatically block resource files that do not adhere to the safeguards that were set, and will change the ones it can change to adhere to those policies, instead of blocking them.

You can enable Deployment Safeguards on an existing cluster usingaz aks update:

az aks update --name clustername --resource-group resourcegroup --safeguards-level Warning --safeguards-version v2.0.0

You can change a cluster's safeguards level fromWarningto Enforcementand vice-versa also usingaz aks update:

az aks update --name safeaks --resource-group safeguard-test --safeguards-level Enforcement

If you wish to turn off Deployment Safeguards completely:

az aks update --name safeaks --resource-group safeguard-test --safeguards-level Off

That should wrap it up for the prerequisites.

Deployment Safeguards in Action

After the cluster is created, please allow at least 30 minutes for Deployment Safeguards and Azure Policy for AKS to successfully sync.

If you've followed with the new cluster creation, set kubectl to the new cluster context by using:

az aks get-credentials --name safeaks --resource-group safeguard-test

Let's runkubectl get nodes -o widejust to verify connectvitiy -

kubectl get nodes -o wide

Output should look like this:

Testing Deployment Safeguards

While the entirety of available safeguard policies is listed here,

We will focus onResource Limits Enforcement, together with a few others which I'll explain below.

Testing Deployment Safeguards

Let's create a normal pod that runs an Nginx image, without any special configuration, and save it as no-limits.yaml:

apiVersion: v1kind: Podmetadata: name: no-limits-herespec: containers: - name: nginx image: nginx

Let's apply it to ourWarninglevel cluster and see what happens, using kubectl apply:

kubectl apply -f no-limits.yaml

We're immediately presented with the following output:

Let's break it down:

Deployment Safeguards expects a liveness and a readiness probe,resource limits, and an image pull secret.

But, since it's set onWarning, it allows the manifest to go through.

In a cluster where safeguards level is set toEnforcement,the pod is blocked from being scheduled:

Let's "fix" our pod to adhere to some of the policies, but let's keep it withoutresource limits:

apiVersion: v1kind: Secretmetadata: name: registrykey namespace: defaultdata: .dockerconfigjson: >- eyJhdXRocyI6eyJodHRwczovL215LXNlY3VyZS1yZWdpc3RyeS5jb20iOnsidXNlcm5hbWUiOiJkb2NrZXIt dXNlciIsInBhc3N3b3JrIjoic2VjdXJlLXBhc3N3b3JkIiwiZW1haWwiOiJ1c2VyQGV4YW1wbGUuY29tIn19fQ==type: kubernetes.io/dockerconfigjson---apiVersion: v1kind: Podmetadata: name: no-limits-herespec: containers: - name: nginx image: my-awesome-registry.com/nginx:latest readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 5 periodSeconds: 5 livenessProbe: httpGet: path: / port: 80 initialDelaySeconds: 10 periodSeconds: 10 imagePullSecrets: - name: registrykey

This "fixed" pod now adheres to the readiness and liveness probe safeguards, adds a pseudo pullsecret, but does not adhere to the resource limits safeguard.

Important Note - This is a pseudo dockerconfigjson, key and of course, container registry. The container will not run. It's on purpose.

Let's save this in a new file called no-limits-updated.yaml,and apply it to the Enforcementcluster:

kubectl apply -f no-limits-updated.yaml

We're presented with the following output:

Kubernetes is not happy with our dummy secret. That's fine. Let's explore and see what happened to our pod.

Our pod did not run [as implied above] but Deployment Safeguards has made changes to it, specifically on the Limits and Requests part.

Let's query it and see what happened:

kubectl get pod no-limits-here -o=jsonpath='{.spec.containers[*].resources}'

You should see the following:

{"limits":{"cpu":"500m","memory":"500Mi"},"requests":{"cpu":"500m","memory":"500Mi"}}

Deployment Safeguards has made our pod adhere to the Limits and Requests section, even without us specifying it.

This is done on the Enforcement level to make sure your workload is aligned with the limits and requests safeguard.

The change happened because Limits and Requests areeligible for mutation.

Other policies that are currently available with mutations are:

• Reserved System Pool Taints
• Pod Disruption Budget
• ReadOnlyRootFileSystem
• RootFilesystemInitContainers

Deployment Safeguards will edit and change your workload to align with these safeguards.

On all other safeguards that are not eligible for mutation, the workload will be rejected on anEnforcement cluster.

You can also exclude a certain namespace from being enforced by Deployment Safeguards using:

az aks update --name safeaks --resource-group safeguard-test --safeguards-level Warning --safeguards-version v2.0.0 --safeguards-excluded-ns myawesomenamespace

Clean up the resources:

az aks delete --name safeaks --resource-group safeguard-test --yesaz group delete --name safeguard-test

Conclusion

Azure Kubernetes Service's Deployment Safeguards feature is a robust tool that ensures Kubernetes deployments adhere to best practices and predefined limits. With options for both Warning and Enforcement levels, users can either be alerted of non-compliance or have their deployments automatically adjusted to meet the required standards. This feature enhances security and operational efficiency, making AKS an even more reliable and user-friendly platform for managing containerized applications.