External Secrets Operator with Vault AppRole and Kapitan refs
Which secret goes in a Kapitan ref and which in Vault: the AppRole secret ID as a gpg ref, an admin token synced from Vault and rotated, tested on kind.
A Kapitan project can keep every secret as a ref, encrypted in Git and revealed at deploy time. External Secrets Operator takes the other route. It reads a secret manager such as Vault from inside the cluster and writes Kubernetes Secrets from it. Most clusters end up with both, and then each secret needs a place. This post puts one app on both and shows where each of its secrets goes and why.
The app is Vaultwarden, from the vaultwarden
recipe 1.0.0. Its admin token comes from Vault
through External Secrets. External Secrets logs in to Vault with an
AppRole, and the AppRole’s secret ID is the one secret the
inventory carries, as a gpg ref. The project is
secrets-end-to-end
in kadet-hub/examples. Its check script runs it on a new
kind cluster on every change and once a week, with
External Secrets 2.14.0, Vault 2.2.1 and Kapitan 0.36.3. It writes a new
admin token to Vault and fails unless Vaultwarden takes it after a
restart.
Which secret goes where
A ref changes with a commit. Its ciphertext lives in the repository’s
refs/ directory, and a new value reaches the cluster with the next
compile and deploy, which reveal it. A secret in Vault changes without
the repository. External Secrets reads it again every refresh interval
and rewrites the Secret, and nobody compiles or commits.
So the question for each secret is who changes it and when. A secret that whoever runs Vault rotates belongs in Vault. A secret that has to exist before External Secrets can read anything cannot come from External Secrets, and belongs in a ref. That second kind is the bootstrap secret, sometimes called secret zero.
The example has one of each:
| AppRole secret ID | Vaultwarden admin token | |
|---|---|---|
| Kind | bootstrap | runtime |
| Issued by | Vault, once | whoever runs Vault, at every rotation |
| Kept in | refs/, encrypted for a gpg key |
Vault, secret/vaultwarden |
| Reaches the cluster | kapitan refs --reveal in the deploy |
External Secrets, every 10 seconds |
| A new value needs | a new ref, a commit and a deploy | vault kv put and a restart of Vaultwarden |
External Secrets needs the secret ID to log in to Vault, so it cannot fetch the secret ID from Vault. The admin token, and any secret the app later keeps in Vault, comes through that login.
How External Secrets logs in to Vault
External Secrets’ Vault provider has several login methods, and they differ in what the inventory has to carry. Only AppRole is tested here.
| Method | Secret in the inventory | Note |
|---|---|---|
| token | the Vault token | Tokens other than root tokens expire with their TTL unless renewed. |
| AppRole | the secret ID; the role ID is not secret | Each login returns a token of its own with token_ttl. The secret ID can be issued without expiry. |
| Kubernetes or JWT auth | none | Vault checks a ServiceAccount token of the pod, through the cluster’s TokenReview API or its signing keys. |
Kubernetes or JWT auth leaves no secret in the inventory, and the split reduces to the right column of the first table. In return, Vault has to reach the cluster’s API server or hold its signing keys, which a Vault that another team runs outside the cluster may not offer. AppRole works from anywhere that reaches Vault, and its one secret has to get into the cluster some other way. Here that way is the ref.
What a refs backend needs
A ref is a tag such as ?{gpg:vaultwarden/vault-secret-id} in the
inventory and a file of the same path under refs/. Three steps touch
it, and they need different access:
| Step | gpg | vaultkv, vaulttransit, cloud KMS |
|---|---|---|
kapitan refs --write, or the first compile of a ref with a function such as ||random:str |
the recipients’ public keys | a Vault token, or KMS credentials |
kapitan compile without --reveal, for refs that exist |
nothing | nothing |
kapitan refs --reveal or kapitan compile --reveal |
a recipient’s private key | the backend’s decrypt or read access |
The middle row is the one that is easy to get wrong. Kapitan reads an
existing ref file and writes its tag with a short hash, so a CI job can
compile a repository with a committed refs/ directory and no
credentials at all. A pull request to Kapitan’s docs,
kapicorp/kapitan#1646,
adds this to the refs documentation. Its author tested gpg, vaultkv,
vaulttransit, gkms, awskms and azkms. check tests gpg, with an empty
keyring, as below. The base64 and plain backends need
nothing at any step and encrypt nothing, so their ref files are as secret
as the values.
The target
inventory/targets/vaultwarden.yml:
classes:
- recipes.vaultwarden
parameters:
namespace: vaultwarden
kapitan:
vars:
target: vaultwarden
secrets:
gpg:
recipients:
- name: secrets-end-to-end@example.com
dependencies:
- type: oci
source: ghcr.io/kadet-hub/external-secrets:1.0.6@sha256:184e8a60f92c0c09eae32973ceaf909b2da53457e88233161ffb90c4b9a63e60
output_path: lib/external-secrets
compile:
# The recipe's Secret vaultwarden. External Secrets writes a Secret of
# that name from Vault.
- input_type: remove
input_paths:
- compiled/${kapitan.vars.target}/manifests/vaultwarden/vaultwarden-secret.yml
output_path: .
- input_type: kadet
input_paths: [lib/external-secrets]
output_path: manifests/
output_type: yml
applications:
vaultwarden:
domain: https://vaultwarden.example.com
# The recipe requires it. Its Secret is removed above.
admin_token: from-vault
generators:
kubernetes:
secrets:
vault-approle:
string_data:
secret-id:
value: ?{gpg:vaultwarden/vault-secret-id}
external_secrets:
secretstores:
vault:
spec:
provider:
vault:
server: http://vault.vault.svc.cluster.local:8200
path: secret
version: v2
auth:
appRole:
path: approle
roleId: vaultwarden
secretRef:
name: vault-approle
key: secret-id
externalsecrets:
vaultwarden:
store: vault
refresh_interval: 10s
spec:
data:
- secretKey: ADMIN_TOKEN
remoteRef:
key: vaultwarden
property: admin_tokenThe kubernetes generator, which the recipe
pins, renders the Secret vault-approle with the ref. The
external-secrets generator renders the
SecretStore vault and the ExternalSecret vaultwarden. It renders
External Secrets’ custom resources only, so check installs External
Secrets itself from the project’s release manifest, as the
cluster add-ons post does.
The recipe expects its admin token as a ref and renders a Secret
vaultwarden with the key ADMIN_TOKEN, which its StatefulSet reads.
Here that Secret comes from External Secrets. The target drops the
recipe’s file with Kapitan’s remove input, and the ExternalSecret
writes a Secret of the same name with the same key, so the StatefulSet
stays as the recipe renders it. The recipe still requires the parameter
admin_token, which therefore holds a placeholder that reaches no
output. Without the remove entry, the output would hold the recipe’s
Secret vaultwarden with the placeholder, and every deploy would apply
it over the Secret that External Secrets writes.
The gpg recipient is named by its user ID. Kapitan looks the key up when it writes a ref and stores the key’s fingerprint in the ref file, so a later compile does not look anything up.
Vault
vault-server.yaml stands in for a Vault outside the cluster. It runs
Vault’s dev server, which keeps everything in memory and mounts KV
version 2 at secret/, behind a ClusterIP Service. Its root token comes
from a Secret that check creates with a random value, and it starts
with -dev-no-store-token, so it writes no token file. Nothing outside
the cluster reaches this Vault. check runs the vault CLI inside the
server’s container with kubectl exec. Vault is licensed BUSL-1.1.
check sets Vault up as its admin would:
vault auth enable approle
vault policy write vaultwarden - <<'EOF'
path "secret/data/vaultwarden" {
capabilities = ["read"]
}
EOF
vault write auth/approle/role/vaultwarden token_policies=vaultwarden token_ttl=10m
vault write auth/approle/role/vaultwarden/role-id role_id=vaultwarden
secret_id=$(vault write -f -field=secret_id auth/approle/role/vaultwarden/secret-id)
printf %s "$token" | vault kv put secret/vaultwarden admin_token=-The policy reads the one path of the admin token. The role ID is set to
vaultwarden, so the SecretStore can name it, and Vault issues the
secret ID. check reads the role back:
Vault: AppRole vaultwarden with secret_id_ttl 0, token_ttl 600, policies vaultwarden, admin token in secret/vaultwarden
secret_id_ttl 0 is the AppRole default and means that the secret ID
does not expire. A secret ID that expires would turn the ref into a
secret to rotate as well.
Writing the ref
The secret ID goes into the ref through stdin, encrypted for the recipient the target lists:
printf %s "$secret_id" | kapitan refs --write gpg:vaultwarden/vault-secret-id -t vaultwarden -f --t vaultwarden takes the recipients from the target. The ref file
refs/vaultwarden/vault-secret-id holds the ciphertext and the
recipient’s fingerprint, here from a local run with the ciphertext cut:
data: hF4DX62rRiT3KU4SAQdA...
encoding: original
recipients:
- fingerprint: FDD8F7AB904048215026F28F7AFFF9D3B29E7B57
type: gpgcheck creates a gpg key in a keyring of its own for each run, so its
refs go to a temporary directory. A repository commits refs/, and the
private key stays with whoever deploys.
Compiling without the key
check compiles with GNUPGHOME pointing to an empty keyring. The
compiled Secret holds the ref’s tag:
apiVersion: v1
kind: Secret
metadata:
labels:
name: vault-approle
name: vault-approle
namespace: vaultwarden
stringData:
secret-id: ?{gpg:vaultwarden/vault-secret-id:75893683}
type: OpaqueThe eight characters after the last colon are the start of a SHA-256
over the ref’s path and ciphertext. A new value changes them, so a diff
of compiled/ shows that a ref changed without showing the value.
check fails if the secret ID or its base64 encoding appears anywhere in
compiled/, and if kapitan refs --reveal succeeds with the empty
keyring. From the CI run:
compiled without a gpg key: 5 files
EX-26: compiled/vaultwarden/manifests/vaultwarden/vault-approle-secret.yml holds ?{gpg:vaultwarden/vault-secret-id:75893683}, not the secret ID
EX-26: reveal without the key fails: GPGError: no secret key
Deploying
The deploy reveals with the key and pipes into kubectl apply, so the
revealed manifests are not written to disk:
kapitan refs --reveal -f compiled/vaultwarden | kubectl apply -f -check applies the Secret vault-approle, the SecretStore and the
ExternalSecret first and the rest once the Secret vaultwarden exists.
Applied together, Vaultwarden’s pod can start before External Secrets has
written its Secret and wait in CreateContainerConfigError until it
has. The SecretStore and the ExternalSecret as compiled:
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
labels:
name: vault
name: vault
namespace: vaultwarden
spec:
provider:
vault:
auth:
appRole:
path: approle
roleId: vaultwarden
secretRef:
key: secret-id
name: vault-approle
path: secret
server: http://vault.vault.svc.cluster.local:8200
version: v2
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
labels:
name: vaultwarden
name: vaultwarden
namespace: vaultwarden
spec:
data:
- remoteRef:
key: vaultwarden
property: admin_token
secretKey: ADMIN_TOKEN
refreshInterval: 10s
secretStoreRef:
kind: SecretStore
name: vaultkey: vaultwarden is the path below the KV mount secret, and the
provider adds the data/ of KV version 2 itself. From the CI run:
EX-26: External Secrets logged in with the revealed secret ID and wrote the admin token into the Secret vaultwarden in 1s
SecretStore: Ready True, Valid: store validated
ExternalSecret: Ready True, SecretSynced: secret synced
Secret vaultwarden: keys ADMIN_TOKEN, owner ExternalSecret/vaultwarden
EX-26: admin login with the token from Vault: 200
The owner reference makes the ExternalSecret the Secret’s controller.
Rotating the admin token
check writes a new token to Vault with vault kv put and waits for
the Secret. Then it logs in to Vaultwarden’s admin page with both tokens,
restarts Vaultwarden and logs in again:
EX-26: the Secret vaultwarden holds the new token 10s after it was written to Vault
EX-26: admin login with the new token, before the restart: 401
EX-26: admin login with the old token, before the restart: 200
restarted in 4s
EX-26: admin login with the new token, after the restart: 200
EX-26: admin login with the old token, after the restart: 401
The Secret follows within the refresh interval, and Vaultwarden does
not. It reads ADMIN_TOKEN from its environment, and the kubelet sets
the environment from the Secret when it starts the container. Until the
restart, the old token logs in and the new one gets 401. After kubectl rollout restart statefulset/vaultwarden it is the other way round.
External Secrets does not restart pods. A rotation that has to take
effect without a person needs a controller that restarts a workload when
its Secret changes, or an app that reads the value from a mounted file
when it uses it. Neither is tested here. Settings saved on Vaultwarden’s
admin page go to config.json on its volume and take precedence over the
environment, so a token saved there no longer follows Vault.
Vaultwarden answers the fourth admin login from one address within 300
seconds with 429 Too Many Requests, by its defaults
ADMIN_RATELIMIT_MAX_BURST 3 and ADMIN_RATELIMIT_SECONDS 300. The first
local run logged in four times before the restart and failed on that.
check now logs in three times per pod start.
What the CI run showed
Run 42003380730
of the examples workflow tested commit 3343fcf of
kadet-hub/examples#146
on 5 November 2026, on GitHub’s ubuntu-24.04 runner, with the kind node
image kindest/node:v1.37.0. The durations below are what check
printed, in whole seconds, or come from the log’s timestamps.
Verifying index serial 113, the recipe and the two generators and
pulling them took 14 seconds. kind had the cluster ready 38 seconds
later. External Secrets and Vault were available 33 seconds after that,
and setting up Vault took about one second. Creating the gpg key and
writing the ref took 7 seconds, with uv installing Kapitan for the first
uv run. The compile without the key took 3.
External Secrets wrote the Secret vaultwarden one second after the
ExternalSecret was applied. Vaultwarden was ready and took the first
login 9 seconds later. The new token reached the Secret 10 seconds after
it was written to Vault, the full refresh interval. Two local runs took 3
and 7 seconds, depending on where in the interval the write fell. The
restart took 4 seconds, and the whole check 125.
What it leaves out
The admin token is stored in Vault as plain text, and Vaultwarden logs a notice about it at every start. Vaultwarden also accepts an Argon2id PHC string of the token, which the recipe’s README recommends, so that the Secret holds only a hash. That is not tested here.
Also not tested: Vault’s Kubernetes and JWT auth, rotating the secret ID, ref backends other than gpg, a reveal inside Argo CD, and a Vault outside the cluster behind TLS. The Argo CD plugin post covers what a reveal in Argo CD’s repo server would expose.