kubernetes

Using system environment variables with Kustomize

After a general overview of what Kustomize allows to do, this blog post is more about giving few tips about how we can leverage system environment variables to parameterize Kustomize files.

rm3l

Jul 1, 2020 — 5 min read

UPDATE (2022/07/28): The trick used here was previously documented officially, but the documentation has been reverted (because it is supposed to be a bug). Instead, a warning message now shows up: This Kustomization is relying on a bug that loads values from the environment when they are omitted from an env file. This behaviour will be removed in the next major release of Kustomize. See https://github.com/kubernetes/website/issues/35669 and https://github.com/kubernetes-sigs/kustomize/issues/4731

UPDATE(2022/01/07): The trick used here to set values from local environment variables has finally been documented in the official documentation: kubernetes/website#30348

UPDATE(2020/07/01): ~~This uses an~~ ~~eschewed (but undocumented) feature of Kustomize. This trick may change, break, or inexplicably disappear at any time. So use this at your own risk!~~

On my journey with Kubernetes, I played a little bit with Kustomize, which is a great tool for adjusting Kubernetes YAML resources to various deployment environments. I was actually surprised to see how Kustomize enforces the use of files (versioned) to build the Kubernetes manifests. But I also stumbled upon an undocumented feature in its code, allowing to use runtime environment variables.

After a general overview of what Kustomize allows to do, we will walk through leveraging runtime system environment variables to parameterize Kustomize files.

A typical use case for this might be in Continuous Deployment (CD) contexts, where, rather than generating configuration files, we could easily leverage existing environment variables for deployment.

For reference, the whole Kustomize project used throughout this blog post is available at GitHub://rm3l/kustomize_envvar

Overview

In a nutshell, using Kustomize, we would be able to:

start from a set of base (and general-purpose) YAML files, which we do not want to alter
apply patches to such base YAML files, resulting in customized YAML files which will get submitted to a given Kubernetes cluster

Since version 1.14, the kubectl command comes bundled with Kustomize, allowing us to use commands such as:

kubectl kustomize /path/to/kustomize/overlay | kubectl apply -f -

kubectl apply -k /path/to/kustomize/overlay

Anatomy of a Kustomize project

Let's walk through a simple scenario to better understand what we want to do.

Say we have the following project structure for Kustomize:

❯ tree     
.
├── base
│   ├── deployment.yaml
│   └── kustomization.yaml
└── overlays
    └── staging
        ├── config.properties
        ├── deployment.yaml
        └── kustomization.yaml

Base

base is the folder containing the set of raw Kubernetes files which we do not want to alter at all.

base/deployment.yaml is a typical YAML Deployment descriptor for Kubernetes:. For example, it deploys 3 replicas of a Pod configured via a ConfigMap that needs to be provided. Here we will configure the ENABLE_RISKY environment variable flag.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: the-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      deployment: hello
  template:
    metadata:
      labels:
        deployment: hello
    spec:
      containers:
      - name: the-container
        image: monopole/hello:1
        command: ["/hello",
                  "--port=8080",
                  "--enableRiskyFeature=$(ENABLE_RISKY)"]
        ports:
        - containerPort: 8080
        env:
        - name: ALT_GREETING
          valueFrom:
            configMapKeyRef:
              name: the-map
              key: ALT_GREETING
        - name: ENABLE_RISKY
          valueFrom:
            configMapKeyRef:
              name: the-map
              key: ENABLE_RISKY

And base/kustomization.yaml is a descriptor needed for Kustomize. In this case, it simply declares all resources that should be included by Kustomize using the resources field:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - deployment.yaml

Overlays

Overlays contain a set of variants off of a same base Kustomization configuration, and allows to apply patches to cater to various environment needs.

As the name suggests here, overlays/staging contains the variant for a staging environment. It will allow us to provide the the-map ConfigMap with the ENABLE_RISKY key configurable dynamically at runtime via a system environment variable.

overlays/staging/kustomization.yaml looks like what follows. It declares the base directory, along with the way to generate the the-map ConfigMap from a given key-value properties file, along with a patch to apply to :

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namePrefix: staging-

bases:
  - ../../base

configMapGenerator:
  - name: the-map
    env: config.properties

patchesStrategicMerge:
  - deployment.yaml

overlays/staging/deployment.yaml file is a simple deployment which once merged with the base one, will change the number of replicas from 3 to 2 for the staging variant. As you can see below, we do not need to provide a complete valid Kubernetes Deployment resource. But using the patchesStrategicMerge strategy, Kustomize is able to find the resource using its name (the-deployment here) and merge the two files.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: the-deployment
spec:
  replicas: 2

overlays/staging/config.properties is a simple key-value properties file used by Kustomize to generate the the-map ConfigMap:

ALT_GREETING=Hiya
ENABLE_RISKY=false

Building the Kustomize overlay

Now that all the structure of our Kustomize project is defined, we can test it by generating Kubernetes YAML descriptors for our staging overlay:

❯ kubectl kustomize overlays/staging


apiVersion: v1
data:
  ALT_GREETING: Hiya
  ENABLE_RISKY: "false"
kind: ConfigMap
metadata:
  name: staging-the-map-7c88gg7h68
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: staging-the-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      deployment: hello
  template:
    metadata:
      labels:
        deployment: hello
    spec:
      containers:
      - command:
        - /hello
        - --port=8080
        - --enableRiskyFeature=$(ENABLE_RISKY)
        env:
        - name: ALT_GREETING
          valueFrom:
            configMapKeyRef:
              key: ALT_GREETING
              name: staging-the-map-7c88gg7h68
        - name: ENABLE_RISKY
          valueFrom:
            configMapKeyRef:
              key: ENABLE_RISKY
              name: staging-the-map-7c88gg7h68
        image: monopole/hello:2
        name: the-container
        ports:
        - containerPort: 8080

The command above outputs the resulting Kubernetes resources which we can then pipe and apply against out Kubernetes cluster:

❯ kubectl kustomize overlays/staging  | kubectl apply -f -

configmap/staging-the-map-5mfm8kmm8t created
deployment.apps/staging-the-deployment created

Using system environment variables

As we can see in the sample project below, the ConfigMap keys and values are pretty static, and cannot be overridden easily. For this, we could simply copy the overlays/staging folder and change the new overlay config.properties accordingly, but this is cumbersome to me just for changing values in the ConfigMap.

The kustomize command exposes an edit command, which edits the kustomization.yaml file, and can be called with environment variables if needed.

What I wanted to do instead is use the same overlays/staging variant, but alter the ENABLE_RISKY property at runtime from environment variables, without editing any kustomization.yaml files.

To do so, the trick is basically to change 2 things:

Declare the alter-able key in the overlays/staging/config.properties file, i.e., with no value, e.g.:

ALT_GREETING=Hiya
ENABLE_RISKY

Export the environment variable prior to calling Kustomize:

❯ ENABLE_RISKY="true" kubectl kustomize overlays/staging

apiVersion: v1
data:
  ALT_GREETING: Hiya
  ENABLE_RISKY: "true"
kind: ConfigMap
metadata:
  name: staging-the-map-5mfm8kmm8t
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: staging-the-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      deployment: hello
  template:
    metadata:
      labels:
        deployment: hello
##### The rest of the output has been omitted for brevity ####

As we can see in the resulting output, the ENABLE_RISKY variable has been successfully changed to "true".

Explanations

After digging a little bit into Kustomize source code, I found out that, when keys are set but without values, the key-value loaders in Kustomize generators default to the context environment when parsing env files. See this code block:

    if len(data) == 2 {
        kv.Value = data[1]
    } else {
        // No value (no `=` in the line) is a signal to obtain the value
        // from the environment.
        kv.Value = os.Getenv(key)
    }
    kv.Key = key
    return kv, nil

This is not documented at all, but good to keep in mind, until this behavior eventually changes in the future. I don't know whether this is intentional, but for now, I thought it was worth sharing to people with similar needs.

EDIT