kadet-hub logokadet-hub

Timoni vs Kapitan: CUE schemas and bundles pinned by digest

A Timoni bundle with its module and a Kapitan inventory render one app to identical manifests. Also tested: a misspelled value and a module tag that moved.

Timoni builds Kubernetes objects from modules written in CUE. A module declares a schema for its values, and Timoni rejects values that do not fit it. Modules are published as OCI artifacts, and a bundle lists the instances to build, each with its module by registry URL, version and digest. Kapitan reads YAML from an inventory and passes it to a generator, here the kadet-hub kubernetes generator, pinned by digest.

The example renders the app of the Helm vs Kustomize vs Kapitan guide with both tools. Its check fails unless the manifests match for each environment. It also tests two differences: a misspelled value, and a module tag that moves after the bundle pinned it. CI runs check on every change and once a week.

The app and the check

Namespace shop, apps web and api with a Deployment, a Service and a ConfigMap each, the restricted Pod Security profile, environments dev and prod, PodDisruptionBudgets in prod only. The api’s ConfigMap carries web_url, built from the web Service’s port.

check sorts the objects and their keys and drops one label, app.kubernetes.io/managed-by, which only the generator sets. Nothing else is dropped:

$ ./check
dev: timoni = kapitan (6 objects)
prod: timoni = kapitan (8 objects)
timoni: replica rejected (replica: field not allowed)
kapitan: replica compiled, web gets replicas: 1
timoni: pinned bundle rejected after the tag moved (digest doesn't match)
timoni: unpinned bundle built the new push

Module and bundle

The Timoni side has two parts. The module timoni/app holds the schema of the values and one template per kind. The schema is #Config in templates/config.cue:

#Config: {
	metadata: {
		name:      string
		namespace: string
	}

	image!:   string
	replicas: *1 | int & >0
	port: {
		container!: int & >0 & <=65535
		service:    *80 | int & >0 & <=65535
	}
	readinessPath!: string
	resources!:     corev1.#ResourceRequirements
	config: {
		mount!: string
		data: [string]: string
	}
	// A PodDisruptionBudget is rendered only when this is set.
	pdbMinAvailable?: int & >0
}

A ! marks a field that has to be set, *1 a default, ? an optional field. resources takes its type from the vendored Kubernetes schema, and each template unifies its object with the Kubernetes type, such as appsv1.#Deployment.

The bundle timoni/shop.cue names the module and holds the values both environments share. dev.cue and prod.cue add the rest, and Timoni unifies the files passed with -f:

bundle: {
	apiVersion: "v1alpha1"
	name:       "shop"
	instances: [string]: {
		module: {url: "oci://localhost:5001/modules/app", version: "1.0.0"}
		namespace: "shop"
	}
	instances: {
		web: values: {
			port: {container: 8080, service: 80}
			readinessPath: "/ready"
			config: mount: "/etc/web"
		}
		api: values: {
			port: {container: 9090, service: 80}
			readinessPath: "/healthz"
			config: {
				mount: "/etc/api"
				data: web_url: "http://web.\(instances.web.namespace).svc:\(instances.web.values.port.service)"
			}
		}
	}
}

The bundle is CUE, so web_url follows the web instance’s port, as the interpolation ${components:web:ports:http:service_port} does in the inventory. A value can refer only to what the bundle sets. The module’s default port of 80 is not visible in the bundle, so the bundle sets service: 80 for both apps, as the inventory does.

Hand-written files, without the vendored schemas and the generator:

files lines
Timoni module 8 206
Timoni bundle 3 66
Kapitan inventory and .kapitan 6 99

The module is code the project owns: templates that spell out the fields the generator writes, such as the probe thresholds and defaultMode: 420. On the Kapitan side that code is the generator, 4529 lines of Python without its tests, maintained in kadet-hub and pinned by digest. The vendored Kubernetes schemas add 104 files and 27,431 lines of generated CUE to the module.

A misspelled value

check replaces replicas: 3 with replica: 3 for web in prod, on both sides. Timoni refuses the bundle:

build failed for web:
timoni.instance.objects.deploy.#config.replica: field not allowed:
    ./values.cue:9:2

#Config is a CUE definition, and a definition is closed: a field it does not declare is an error. The position is in the values file that Timoni assembles in memory from the module’s defaults and the bundle, not in prod.cue.

Kapitan compiles the same change and exits 0. The generator does not know replica, ignores it and renders web with replicas: 1. It logs one warning, because the PodDisruptionBudget of 2 then allows no eviction:

web: pdb_min_available 2 >= 1 replicas allows no eviction, which blocks node drains

The warning comes from the generator’s rule for the budget, not from a check of the key. Kapitan 0.36.3 and the generator 1.14.0 accept a key they do not know.

Where the module comes from

Timoni publishes a module as an OCI artifact. check acts as the module’s author: it starts a registry on localhost:5001, pushes the module and pins the bundle to the digest of that push with one more bundle file:

digest=$(timoni mod push timoni/app oci://localhost:5001/modules/app -v 1.0.0 -o json | jq -er .digest)
echo "bundle: instances: [string]: module: digest: \"$digest\"" >"$out/pin.cue"
timoni bundle build -f timoni/shop.cue -f timoni/prod.cue -f "$out/pin.cue"

The digest is not committed. timoni mod push annotates the artifact with the Git commit and its time, so each commit of the example yields another digest.

A version tag can move. check pushes a second build of the module as 1.0.0, one that sets imagePullPolicy: Always. The pinned bundle then fails:

the upstream digest sha256:… of version 1.0.0 doesn't match the specified digest sha256:…

Without the digest, the same bundle builds the second push, and the Deployments pull Always.

On the Kapitan side, inventory/classes/common.yml pins the generator by version and digest:

source: ghcr.io/kadet-hub/kubernetes:1.14.0@sha256:787129e1eaed6b9bc868fe4c4aec58d40de6a89fc3cc08424a4cb5822b855080

Kapitan 0.36.3 checks neither the digest nor a signature when it fetches dependencies. check does both outside Kapitan: gh attestation verify shows that the kadet-hub release workflow built the artifact, oras pull fetches it by digest, and Kapitan compiles without --fetch. Timoni compares the digest itself. timoni mod push --sign=cosign signs a module and timoni mod pull --verify=cosign checks the signature, while timoni bundle build in 0.37.0 has no flag to verify one. The example signs nothing.

What the example leaves out

The images are placeholders, so nothing is deployed. timoni bundle apply, which applies the instances to a cluster and keeps a record of their objects there, is not tested, and neither is applying Kapitan’s output with kubectl, Argo CD or Flux. Timoni’s runtime values, which a bundle reads from the environment or the cluster, are not used. The example renders Kubernetes objects only, so Kapitan’s other outputs from the same inventory, such as Terraform, are not compared.

Running it

You need timoni, docker, curl, jq and uv, plus oras and a logged-in gh for the generator:

$ git clone https://github.com/kadet-hub/examples
$ cd examples/timoni-vs-kapitan
$ ./check