This topic has its origin in how we manage Kubernetes Secrets at my workplace. We use Helm for deployments, and we must support several environments with their connection strings, passwords, and other settings. As a result, some things are a bit more complicated, and one of them is the access to Kubernetes Secrets from a Spring Boot application running in a Pod.

This blog post covers the following:

1. How do you generally get Secrets into a Pod?
2. How do we currently do it using Helm?
3. How can it be improved with less configuration?
4. Any gotchas? Of course, it is software.

I will explain a lot of rationales, so expect a substantial amount of prose between the (code) snippets.

Get Kubernetes to reveal its Secrets

The first revelation was the fact that what we do in Helm is actually nothing Helm-specific.

Let me take a step back. You have three options to get the Secrets.

1. Access the Kubernetes API.
2. Define environment variables that reference Secrets.
3. Mount Secrets to the filesystem.

Option #1 is discouraged, mainly for security reasons. A Pod that has the permission to read secrets can be abused to extract information it is not supposed to have. From what I know, this is disabled by default and requires fiddling around with ServiceAccount and RoleBinding.

Option #2 is what we have used so far and works the following way.

You create the secret.

apiVersion: v1
kind: Secret
metadata:
  name: the-gunslinger-secret
  namespace: platform
type: Opaque
data:
  password: VGhlIG1hbiBpbiBibGFjayBmbGVkIGFjcm9zcyB0aGUgZGVzZXJ0LCBhbmQgdGhlIGd1bnNsaW5nZXIgZm9sbG93ZWQu

You refer to the Secret in a Deployment (or Pod).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: the-gunslinger
spec:
  replicas: all
  template:
    metadata:
      spec:
        containers:
          - name: the-gunslinger
            ...
            env:
              - name: AAD_LOGIN_PASSWORD
                valueFrom:
                  secretKeyRef:
                    name: the-gunslinger-secret
                    key: password

The important bits are secretKeyRef.name and secretKeyRef.key. The first refers to the name of the Secret object and the second to the key-value-pair in the Secret.

This is a simple enough approach when looking at it in isolation. Imagine your service communicates with ten or more platform services in a Cloud environment like Azure or AWS. It suddenly becomes tedious. There is a slight twist coming, though, which I will explain in a follow-up blog post (it is already written, I swear!).

Option #3 injects the contents of a Secret as files into a Pod. Every key-value pair in a Secret is represented as a single file in the Pod’s filesystem. The content is the plain-text value ready to be consumed by anybody interested in it (within the bounds of the Pod). I will explain this in more detail later by combining it with Spring Cloud Kubernetes.

How to make it complicated with Helm

It took a very long time for me to “warm up” to it, but Helm is a decent tool when it works and when you use it efficiently. The following example is not that.

We still have the Secret, as shown earlier. The deployment file is similar, but it is a Helm template now. Therefore, we can template all the values and make them configurable.

- name: AAD_LOGIN_PASSWORD
  valueFrom:
    secretKeyRef:
      name: {{ .Values.azureSecrets.login.secretName }}
      key: {{ .Values.azureSecrets.login.secretKey }}

Here we give us the option to change the name of Secret and its value if so desired. This way, it can be different for every environment. If you require it depends on your needs. Next, we must provide the values to this template somewhere. You do that in the values.yaml file.

azureSecrets:
  login:
    secretName: the-gunslinger-secret
    secretKey: password

The last piece of the puzzle is to use the environment variable in the Spring Boot application.yaml, so Spring Boot picks it up. We utilize a nifty trick to edit the contents of the application properties in the values YAML file and template that into the ConfigMap generated on deployment (not shown here).

applicationProperties:
  aad:
    user: Roland
    password: "${AAD_LOGIN_PASSWORD}"

Putting it all together, our values YAML looks like this.

azureSecrets:
  login:
    secretName: the-gunslinger-secret
    secretKey: password

applicationProperties:
  aad: 
    user: Roland
    password: "${AAD_LOGIN_PASSWORD}"

(Some naming inconsistencies are on purpose to emphasize the different uses.)

This is highly flexible. This is also complicated in a way that it could be designated as art. We manage four different locations to add a new password or connection string if you think about it.

1. The Kubernetes Secret.
2. The deployment YAML file to generate environment variables.
3. The values YAML to configure the name of the Secret and the key-value pair.
4. The values YAML to configure the environment variable in the application properties.

Here is a visualization of all these steps.

Unsure how we got there, probably one of those “for historical reasons” situations, we agreed that we should find out if there is a more elegant way.

Reading a file is not the worst idea after all

Well, it depends 😉. Here is how you can do it nicely.

The first order of business is mounting the Secret into the Pod. You do that instead of using environment variables. Depending on how you manage your secrets, this can reduce your configuration efforts significantly. Let us entertain the idea that every microservice gets only a single Secret containing everything it needs. This way, you configure one Secret volume mount and only add new key-value pairs to that service’s Secret. Kubernetes takes care of mounting them into your Pod as individual files.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: the-gunslinger
spec:
  replicas: all
  template:
    metadata:
      spec:
        containers:
          - name: the-gunslinger
            ...
            volumeMounts:
              - name: the-gunslinger-secret
                mountPath: "/opt/application/secrets"
                readOnly: true
        volumes:
          - name: the-gunslinger-secret
            secret:
              secretName: the-gunslinger-secret

What does that look like, you ask?

$ pwd
/opt/application
$ ls
secrets
$ ls secrets
password
$ cat secrets/password
The man in black fled across the desert, and the gunslinger followed.

Before I continue, let me make a minor adjustment to the Secret. This is a crucial part that the documentation does not explicitly mention and requires you to pick up implicitly (or I was blind).

apiVersion: v1
kind: Secret
metadata:
  name: the-gunslinger-secret
  namespace: platform
type: Opaque
data:
  aad.password: VGhlIG1hbiBpbiBibGFjayBmbGVkIGFjcm9zcyB0aGUgZGVzZXJ0LCBhbmQgdGhlIGd1bnNsaW5nZXIgZm9sbG93ZWQu

Did you notice the change? I renamed “password” to “aad.password”. Consequently, the file will be called /opt/application/secrets/aad.password. Bear with me for a minute, and it will make sense shortly.

Now it is time to introduce the Spring Cloud Kubernetes dependency. Therein lies a big gotcha to which I will get to at the end. It is enough to include the “-config” starter for our purposes. It provides support for ConfigMap and Secret.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-kubernetes-fabric8-config</artifactId>
   <version>2.1.0</version>
</dependency>

There is one more required configuration. It is a bit of an oddball because you must store it in a bootstrap.yaml file. I was unsuccessful in getting it to work in the application properties file.

spring:
  cloud:
    kubernetes:
      enabled: true

      # Theoretically, Spring Cloud Kubernetes supports reloading.
      # It may require code changes to make full use of that.
      reload:
        enabled: false

      # Config requires the service account of the Pod to have access 
      # to ConfigMaps.
      config:
        enabled: false

      # Enable scanning for secrets; does not require special 
      # ServiceAccount access like 'config'. 'paths' may contain a list.
      secrets:
        paths: /opt/application/secrets
        enabled: true

Note how spring.cloud.kubernetes.secrets.path refers to the location where we mounted the Secret into the Pod.

Word tells me that I am way over 1000 words into this blog post, and I am yet to show any source code. I should be ashamed. How dare I call myself a programmer? I am glad you asked. Let me go back a few years and… No. Code! The beauty of Spring Cloud Kubernetes Config is how it automatically reads several locations for data. One is the filesystem at the location set in the bootstrap YAML.

package com.thecodeslinger.externalconfig.config;

import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;

@Data
@ConfigurationProperties(prefix = "aad")
public class AddConfigProperties {

    private String password;
}

Do you recognize the “prefix”? Exactly! It matches the earlier change in the Secret. You could use prefixes to segment your Secret based on context and create dedicated @ConfigurationProperties classes. From this point forward, you can utilize this properties class like any other of its kind. The cool thing about it is that it not only works for explicitly created @ConfigurationProperties classes. If you rely on some of Spring Boot’s auto-configuration magic, e.g., for a database connection, you can still store the database password in the Secret. It is all about the name of the key-value pair.

Spring uses a configuration like the following to connect to a PostgreSQL server.

spring:
  datasource:
    url: jdbc:postgresql://localhost:5432/the-gunslinger
    username: Roland
    password: ...

The Secret would look like this.

apiVersion: v1
kind: Secret
metadata:
  name: the-gunslinger-secret
  namespace: platform
type: Opaque
data:
  aad.password: VGhlIG1hbiBpbiBibGFjayBmbGVkIGFjcm9zcyB0aGUgZGVzZXJ0LCBhbmQgdGhlIGd1bnNsaW5nZXIgZm9sbG93ZWQu
  spring.datasource.password: SSBkbyBub3QgYWltIHdpdGggbXkgaGFuZDsgaGUgd2hvIGFpbXMgd2l0aCBoaXMgaGFuZCBoYXMgZm9yZ290dGVuIHRoZSBmYWNlIG9mIGhpcyBmYXRoZXIu

It is like writing in a *.properties file instead of YAML. You can keep the configuration as is and only remove “password”, which will be read from the mounted Secret file.

Let me review what we have done so far (apart from writing almost no code).

1. We created the Kubernetes Secret.
2. We mounted the Secret into the Pod.
3. We added the Spring Cloud Kubernetes Config Starter (rolls right off the tongue).
4. We configured the location where secrets can be found.
5. We created the @ConfigurationProperties class.

That sounds like an awful lot! However, consider that steps 2, 3, and 4 are a one-time effort. All you need now going forward is to extend your Secret and your Spring configuration classes. These are tasks that are required no matter how you slice it. The most significant outcome of these changes is that a bunch of configurations can now enjoy the agony of the nine plains of hell.

1. No more environment variables in the deployment.
2. No more environment variables in the application properties.
3. No more configuration of the configuration of environment variables in the deployment.
4. Literally no more configuration-ception.

Remember that those had been steps to take every time a new secret value was introduced.

There is no such thing as free lunch

Welcome to the “gotcha” Section. Here are a few of the issues I discovered while researching this topic.

Compatibility

My initial tests began with the spring-cloud-starter-kubernetes-config dependency. It seemed like the most legit version. At the time of writing, the Spring Cloud Starter package 1.1.10.RELEASE is not compatible with the latest Spring Boot v2.6.x. For testing and verifying the functionality, it was good enough to downgrade to an earlier Spring Boot version. For production, it was not an option as there are a lot of security vulnerabilities that would not get past our automated CVSS scan.

As a result, I switched to the Fabric8 variant. There is another difference besides the use of more modern dependencies. See the documentation for details to determine if the differences are relevant for you.

Access Error to Kubernetes API

I have disabled the ConfigMap support in the bootstrap.yaml file, as you could see earlier. If you leave it enabled, you will find friendly reminders about access permissions in your log files. You can get around that when you configure the proper Kubernetes ServiceAccount and RoleBinding, of course.

2022-02-04 06:23:09.643  WARN 19 --- [nio-8080-exec-5] o.s.cloud.kubernetes.StandardPodUtils    : Failed to get pod with name:[the-gunslinger-77f8459b77-5h9wg]. 
You should look into this if things aren't working as you expect. Are you missing serviceaccount permissions?

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.0.0.1/api/v1/namespaces/platform/pods/the-gunslinger-77f8459b77-5h9wg. 
Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. 
pods "the-gunslinger-77f8459b77-5h9wg" is forbidden: User "system:serviceaccount:platform:default" cannot get resource "pods" in API group "" in the namespace "platform".

Local Debugging

The Spring Cloud Starter package 1.1.10.RELEASE allows parsing secrets from local directories with a JVM parameter or via the bootstrap.yaml. It automatically works.

-Dspring.cloud.kubernetes.secrets.paths=/Users/the-codeslinger/secrets

When using the fabric8-starter, this is not the case. It detects that, for example, IntelliJ is not a Kubernetes server and deactivates its components. The joys of automagic configuration.

(Such audacity.)

You can trick the library into believing it is running in a Kubernetes container with two environment variables.

export KUBERNETES_SERVICE_HOST=localhost
export KUBERNETES_SERVICE_PORT=1234

(Who is the smart one now, Spring?)

The values do not matter as long as the variables exist. This hack probably won’t work with all Spring Cloud Kubernetes features, but it does the trick for our purposes.

I found this by wading through the sources of the Spring Cloud code.

Epilogue

Spring Cloud Kubernetes is a viable option. It comes with its own set of problems, though, as I have shown. If it is only the Secrets you are after, make sure to read part 2 of this journey. There is an even simpler way that does not require an additional dependency. Unless you have use for Spring Cloud Kubernetes’ other functionality, there is no reason to utilize it just for the Secrets.

Thank you for reading all the way to the end. I hope this was helpful.