Injecting Secrets from AWS Secrets Manager into Kubernetes

Secrets Manager is an AWS service that lets you securely store passwords, keys, tokens, and other sensitive data. By using Secrets Manager, you can level up your secrets management game with scheduled rotation, fine-grained access control through IAM, and auditing.

If you’re running Kubernetes, your applications need a way to retrieve these secrets from Secrets Manager. Unfortunately, there is currently no native integration between AWS Secrets Manager and EKS (or Kubernetes in general).

So, here are a few ways to get your application access to the secrets it needs. This list is not exhaustive by any means, but should give you a few decent options.

Bash Retrieve and Inject

While deployments can become unwieldy with a breathtaking constellation of shell scripts, they get the job done and are easy for most engineers to reason about. So don’t be bashful about being bashful.

With this method, you use a simple shell script to:

• Grab a list of secrets using the aws secretsmanager cli
• Convert each secret’s name format (hey/ima/secret) to a k8s-friendly format (hey-ima-secret)
• Use kubectl to inject the list of secrets into the cluster as native k8s secrets

Here is a script I’ve written that should be generic enough for anyone to use: https://github.com/snobear/inject-k8s-secrets

Once you’ve injected your secrets into the cluster, you can reference them in your helm charts as usual, e.g.:

# mount the secret as a file
spec:
  containers:
  - name: mypod
    image: api-service
    volumeMounts:
    - name: dbcreds
      mountPath: "/etc/dbcreds"
      readOnly: true
  volumes:
  - name: dbcreds
    secret:
      secretName: api-db-dbpass
# alternatively, expose secret as an environment var
spec:
  containers:
  - name: mypod
    image: api-service
    env:
    - name: DBPASS
      valueFrom:
        secretKeyRef:
          name: api-db-dbpass
          key: password

See the kubernetes docs on secrets for full yaml options.

The downside of this approach is that if a secret is changed, you’ll need to reinject + redeploy your pods. So out of the box you can’t leverage the secret rotation that Secrets Manager offers without some additional automation.

Have the Application Grab Secrets Directly

If you have the luxury of tweaking the application code to query AWS Secrets Manager via an SDK, then this is a very secure approach. There is no special script or service in the middle injecting the secrets into the cluster and no secrets being persisted in etcd.

You’ll want to create some IAM policies that ensure only certain pods can access certain secrets. If you’ve associated IAM roles with your Kubernetes service accounts, then you can create policies to restrict access. For example, restricting a role (service account) to read-only access of a given secret namespace:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kms:DescribeKey",
                "kms:Decrypt"
            ],
            "Resource": "arn:aws:kms:us-west-2:123456789012:key/a87daa22-0000-01ac-8b89-1357591ab39"
        },
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": "arn:aws:secretsmanager:us-west-2:123456789012:secret:myapp/dev/api/*"
        }
    ]
}

Note that you need to allow the service access to the KMS key that encrypted the secret so it can decrypt it.

The downside here is having cloud vendor-specific code in your application, i.e. the AWS secrets manager SDK calls, but you could wrap that in a secret-lookup function that could be easily replaced down the road.

Kubernetes External Secrets

The GoDaddy engineering team has an open source solution called Kubernetes External Secrets. It’s simple to deploy and use, and introduces two things in your cluster:

1. An ExternalSecret object
2. An ExternalSecret controller

Per their blog post which details the design and architecture:

…users declare a desired state in an ExternalSecret object and the controller creates or updates a complementary Secret object to reach that state. The controller monitors ExternalSecret objects, fetches secret data from the specified external secret management system, and automatically creates native Secret objects that hold the secret data.

From a developer’s point of view, you have one extra step which is to define an ExternalSecret, e.g.:

apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: web-db-pass
secretDescriptor:
  backendType: secretsManager
  data:
    - key: myproduct/web-api/db-password
      name: password

where key is the alias of your secret in Secrets Manager. Then justreference that metadata name in your chart like a regular k8s Secret:

apiVersion: v1
kind: Pod
metadata:
  name: web-api
spec:
  containers:
  - name: web-api
    image: web-api
    env:
      - name: DBPASS
        valueFrom:
          secretKeyRef:
            name: web-db-pass
            key: password

Oh and External Secrets is extendable to other secrets providers/backends. It currently supports AWS Secrets Manager, AWS System Manager, Hashicorp Vault, Azure Key Vault, Google Secret Manager and Alibaba Cloud KMS Secret Manager. Thanks to the GoDaddy Engineering team for taking the time and effort to open source this.

That’s just a short list of a few approaches I’ve dug into. If you have any other clever approaches to getting secrets into your Kubernetes-based applications from an external secrets source, please share in the comments!