kadet-hub logokadet-hub

External Secrets Operator with Vault AppRole and Kapitan refs

Which secret goes in a Kapitan ref and which in Vault: the AppRole secret ID as a gpg ref, an admin token synced from Vault and rotated, tested on kind.

A Kapitan project can keep every secret as a ref, encrypted in Git and revealed at deploy time. External Secrets Operator takes the other route. It reads a secret manager such as Vault from inside the cluster and writes Kubernetes Secrets from it. Most clusters end up with both, and then each secret needs a place. This post puts one app on both and shows where each of its secrets goes and why.

The app is Vaultwarden, from the vaultwarden recipe 1.0.0. Its admin token comes from Vault through External Secrets. External Secrets logs in to Vault with an AppRole, and the AppRole’s secret ID is the one secret the inventory carries, as a gpg ref. The project is secrets-end-to-end in kadet-hub/examples. Its check script runs it on a new kind cluster on every change and once a week, with External Secrets 2.14.0, Vault 2.2.1 and Kapitan 0.36.3. It writes a new admin token to Vault and fails unless Vaultwarden takes it after a restart.

Which secret goes where

A ref changes with a commit. Its ciphertext lives in the repository’s refs/ directory, and a new value reaches the cluster with the next compile and deploy, which reveal it. A secret in Vault changes without the repository. External Secrets reads it again every refresh interval and rewrites the Secret, and nobody compiles or commits.

So the question for each secret is who changes it and when. A secret that whoever runs Vault rotates belongs in Vault. A secret that has to exist before External Secrets can read anything cannot come from External Secrets, and belongs in a ref. That second kind is the bootstrap secret, sometimes called secret zero.

The example has one of each:

AppRole secret ID Vaultwarden admin token
Kind bootstrap runtime
Issued by Vault, once whoever runs Vault, at every rotation
Kept in refs/, encrypted for a gpg key Vault, secret/vaultwarden
Reaches the cluster kapitan refs --reveal in the deploy External Secrets, every 10 seconds
A new value needs a new ref, a commit and a deploy vault kv put and a restart of Vaultwarden

External Secrets needs the secret ID to log in to Vault, so it cannot fetch the secret ID from Vault. The admin token, and any secret the app later keeps in Vault, comes through that login.

How External Secrets logs in to Vault

External Secrets’ Vault provider has several login methods, and they differ in what the inventory has to carry. Only AppRole is tested here.

Method Secret in the inventory Note
token the Vault token Tokens other than root tokens expire with their TTL unless renewed.
AppRole the secret ID; the role ID is not secret Each login returns a token of its own with token_ttl. The secret ID can be issued without expiry.
Kubernetes or JWT auth none Vault checks a ServiceAccount token of the pod, through the cluster’s TokenReview API or its signing keys.

Kubernetes or JWT auth leaves no secret in the inventory, and the split reduces to the right column of the first table. In return, Vault has to reach the cluster’s API server or hold its signing keys, which a Vault that another team runs outside the cluster may not offer. AppRole works from anywhere that reaches Vault, and its one secret has to get into the cluster some other way. Here that way is the ref.

What a refs backend needs

A ref is a tag such as ?{gpg:vaultwarden/vault-secret-id} in the inventory and a file of the same path under refs/. Three steps touch it, and they need different access:

Step gpg vaultkv, vaulttransit, cloud KMS
kapitan refs --write, or the first compile of a ref with a function such as ||random:str the recipients’ public keys a Vault token, or KMS credentials
kapitan compile without --reveal, for refs that exist nothing nothing
kapitan refs --reveal or kapitan compile --reveal a recipient’s private key the backend’s decrypt or read access

The middle row is the one that is easy to get wrong. Kapitan reads an existing ref file and writes its tag with a short hash, so a CI job can compile a repository with a committed refs/ directory and no credentials at all. A pull request to Kapitan’s docs, kapicorp/kapitan#1646, adds this to the refs documentation. Its author tested gpg, vaultkv, vaulttransit, gkms, awskms and azkms. check tests gpg, with an empty keyring, as below. The base64 and plain backends need nothing at any step and encrypt nothing, so their ref files are as secret as the values.

The target

inventory/targets/vaultwarden.yml:

classes:
  - recipes.vaultwarden
parameters:
  namespace: vaultwarden
  kapitan:
    vars:
      target: vaultwarden
    secrets:
      gpg:
        recipients:
          - name: secrets-end-to-end@example.com
    dependencies:
      - type: oci
        source: ghcr.io/kadet-hub/external-secrets:1.0.6@sha256:184e8a60f92c0c09eae32973ceaf909b2da53457e88233161ffb90c4b9a63e60
        output_path: lib/external-secrets
    compile:
      # The recipe's Secret vaultwarden. External Secrets writes a Secret of
      # that name from Vault.
      - input_type: remove
        input_paths:
          - compiled/${kapitan.vars.target}/manifests/vaultwarden/vaultwarden-secret.yml
        output_path: .
      - input_type: kadet
        input_paths: [lib/external-secrets]
        output_path: manifests/
        output_type: yml
  applications:
    vaultwarden:
      domain: https://vaultwarden.example.com
      # The recipe requires it. Its Secret is removed above.
      admin_token: from-vault
  generators:
    kubernetes:
      secrets:
        vault-approle:
          string_data:
            secret-id:
              value: ?{gpg:vaultwarden/vault-secret-id}
    external_secrets:
      secretstores:
        vault:
          spec:
            provider:
              vault:
                server: http://vault.vault.svc.cluster.local:8200
                path: secret
                version: v2
                auth:
                  appRole:
                    path: approle
                    roleId: vaultwarden
                    secretRef:
                      name: vault-approle
                      key: secret-id
      externalsecrets:
        vaultwarden:
          store: vault
          refresh_interval: 10s
          spec:
            data:
              - secretKey: ADMIN_TOKEN
                remoteRef:
                  key: vaultwarden
                  property: admin_token

The kubernetes generator, which the recipe pins, renders the Secret vault-approle with the ref. The external-secrets generator renders the SecretStore vault and the ExternalSecret vaultwarden. It renders External Secrets’ custom resources only, so check installs External Secrets itself from the project’s release manifest, as the cluster add-ons post does.

The recipe expects its admin token as a ref and renders a Secret vaultwarden with the key ADMIN_TOKEN, which its StatefulSet reads. Here that Secret comes from External Secrets. The target drops the recipe’s file with Kapitan’s remove input, and the ExternalSecret writes a Secret of the same name with the same key, so the StatefulSet stays as the recipe renders it. The recipe still requires the parameter admin_token, which therefore holds a placeholder that reaches no output. Without the remove entry, the output would hold the recipe’s Secret vaultwarden with the placeholder, and every deploy would apply it over the Secret that External Secrets writes.

The gpg recipient is named by its user ID. Kapitan looks the key up when it writes a ref and stores the key’s fingerprint in the ref file, so a later compile does not look anything up.

Vault

vault-server.yaml stands in for a Vault outside the cluster. It runs Vault’s dev server, which keeps everything in memory and mounts KV version 2 at secret/, behind a ClusterIP Service. Its root token comes from a Secret that check creates with a random value, and it starts with -dev-no-store-token, so it writes no token file. Nothing outside the cluster reaches this Vault. check runs the vault CLI inside the server’s container with kubectl exec. Vault is licensed BUSL-1.1.

check sets Vault up as its admin would:

vault auth enable approle
vault policy write vaultwarden - <<'EOF'
path "secret/data/vaultwarden" {
  capabilities = ["read"]
}
EOF
vault write auth/approle/role/vaultwarden token_policies=vaultwarden token_ttl=10m
vault write auth/approle/role/vaultwarden/role-id role_id=vaultwarden
secret_id=$(vault write -f -field=secret_id auth/approle/role/vaultwarden/secret-id)
printf %s "$token" | vault kv put secret/vaultwarden admin_token=-

The policy reads the one path of the admin token. The role ID is set to vaultwarden, so the SecretStore can name it, and Vault issues the secret ID. check reads the role back:

Vault: AppRole vaultwarden with secret_id_ttl 0, token_ttl 600, policies vaultwarden, admin token in secret/vaultwarden

secret_id_ttl 0 is the AppRole default and means that the secret ID does not expire. A secret ID that expires would turn the ref into a secret to rotate as well.

Writing the ref

The secret ID goes into the ref through stdin, encrypted for the recipient the target lists:

printf %s "$secret_id" | kapitan refs --write gpg:vaultwarden/vault-secret-id -t vaultwarden -f -

-t vaultwarden takes the recipients from the target. The ref file refs/vaultwarden/vault-secret-id holds the ciphertext and the recipient’s fingerprint, here from a local run with the ciphertext cut:

data: hF4DX62rRiT3KU4SAQdA...
encoding: original
recipients:
- fingerprint: FDD8F7AB904048215026F28F7AFFF9D3B29E7B57
type: gpg

check creates a gpg key in a keyring of its own for each run, so its refs go to a temporary directory. A repository commits refs/, and the private key stays with whoever deploys.

Compiling without the key

check compiles with GNUPGHOME pointing to an empty keyring. The compiled Secret holds the ref’s tag:

apiVersion: v1
kind: Secret
metadata:
  labels:
    name: vault-approle
  name: vault-approle
  namespace: vaultwarden
stringData:
  secret-id: ?{gpg:vaultwarden/vault-secret-id:75893683}
type: Opaque

The eight characters after the last colon are the start of a SHA-256 over the ref’s path and ciphertext. A new value changes them, so a diff of compiled/ shows that a ref changed without showing the value. check fails if the secret ID or its base64 encoding appears anywhere in compiled/, and if kapitan refs --reveal succeeds with the empty keyring. From the CI run:

compiled without a gpg key: 5 files
EX-26: compiled/vaultwarden/manifests/vaultwarden/vault-approle-secret.yml holds ?{gpg:vaultwarden/vault-secret-id:75893683}, not the secret ID
EX-26: reveal without the key fails: GPGError: no secret key

Deploying

The deploy reveals with the key and pipes into kubectl apply, so the revealed manifests are not written to disk:

kapitan refs --reveal -f compiled/vaultwarden | kubectl apply -f -

check applies the Secret vault-approle, the SecretStore and the ExternalSecret first and the rest once the Secret vaultwarden exists. Applied together, Vaultwarden’s pod can start before External Secrets has written its Secret and wait in CreateContainerConfigError until it has. The SecretStore and the ExternalSecret as compiled:

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  labels:
    name: vault
  name: vault
  namespace: vaultwarden
spec:
  provider:
    vault:
      auth:
        appRole:
          path: approle
          roleId: vaultwarden
          secretRef:
            key: secret-id
            name: vault-approle
      path: secret
      server: http://vault.vault.svc.cluster.local:8200
      version: v2
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  labels:
    name: vaultwarden
  name: vaultwarden
  namespace: vaultwarden
spec:
  data:
    - remoteRef:
        key: vaultwarden
        property: admin_token
      secretKey: ADMIN_TOKEN
  refreshInterval: 10s
  secretStoreRef:
    kind: SecretStore
    name: vault

key: vaultwarden is the path below the KV mount secret, and the provider adds the data/ of KV version 2 itself. From the CI run:

EX-26: External Secrets logged in with the revealed secret ID and wrote the admin token into the Secret vaultwarden in 1s
  SecretStore: Ready True, Valid: store validated
  ExternalSecret: Ready True, SecretSynced: secret synced
  Secret vaultwarden: keys ADMIN_TOKEN, owner ExternalSecret/vaultwarden
EX-26: admin login with the token from Vault: 200

The owner reference makes the ExternalSecret the Secret’s controller.

Rotating the admin token

check writes a new token to Vault with vault kv put and waits for the Secret. Then it logs in to Vaultwarden’s admin page with both tokens, restarts Vaultwarden and logs in again:

EX-26: the Secret vaultwarden holds the new token 10s after it was written to Vault
EX-26: admin login with the new token, before the restart: 401
EX-26: admin login with the old token, before the restart: 200
restarted in 4s
EX-26: admin login with the new token, after the restart: 200
EX-26: admin login with the old token, after the restart: 401

The Secret follows within the refresh interval, and Vaultwarden does not. It reads ADMIN_TOKEN from its environment, and the kubelet sets the environment from the Secret when it starts the container. Until the restart, the old token logs in and the new one gets 401. After kubectl rollout restart statefulset/vaultwarden it is the other way round.

External Secrets does not restart pods. A rotation that has to take effect without a person needs a controller that restarts a workload when its Secret changes, or an app that reads the value from a mounted file when it uses it. Neither is tested here. Settings saved on Vaultwarden’s admin page go to config.json on its volume and take precedence over the environment, so a token saved there no longer follows Vault.

Vaultwarden answers the fourth admin login from one address within 300 seconds with 429 Too Many Requests, by its defaults ADMIN_RATELIMIT_MAX_BURST 3 and ADMIN_RATELIMIT_SECONDS 300. The first local run logged in four times before the restart and failed on that. check now logs in three times per pod start.

What the CI run showed

Run 42003380730 of the examples workflow tested commit 3343fcf of kadet-hub/examples#146 on 5 November 2026, on GitHub’s ubuntu-24.04 runner, with the kind node image kindest/node:v1.37.0. The durations below are what check printed, in whole seconds, or come from the log’s timestamps.

Verifying index serial 113, the recipe and the two generators and pulling them took 14 seconds. kind had the cluster ready 38 seconds later. External Secrets and Vault were available 33 seconds after that, and setting up Vault took about one second. Creating the gpg key and writing the ref took 7 seconds, with uv installing Kapitan for the first uv run. The compile without the key took 3.

External Secrets wrote the Secret vaultwarden one second after the ExternalSecret was applied. Vaultwarden was ready and took the first login 9 seconds later. The new token reached the Secret 10 seconds after it was written to Vault, the full refresh interval. Two local runs took 3 and 7 seconds, depending on where in the interval the write fell. The restart took 4 seconds, and the whole check 125.

What it leaves out

The admin token is stored in Vault as plain text, and Vaultwarden logs a notice about it at every start. Vaultwarden also accepts an Argon2id PHC string of the token, which the recipe’s README recommends, so that the Secret holds only a hash. That is not tested here.

Also not tested: Vault’s Kubernetes and JWT auth, rotating the secret ID, ref backends other than gpg, a reveal inside Argo CD, and a Vault outside the cluster behind TLS. The Argo CD plugin post covers what a reveal in Argo CD’s repo server would expose.