kadet-hub logokadet-hub

Writing a Helm chart vs Kapitan: three popular charts rebuilt

Writing a Helm chart vs a Kapitan inventory, measured on metrics-server, cert-manager and kube-prometheus-stack: equal objects and behaviour on kind.

If you install a third-party chart, keep the chart. Its values are the interface its maintainers support, and for what they do not expose you can patch the rendered output. This post is about the other side: the people who write and maintain a chart. What would they maintain if the same objects came from a Kapitan inventory and a generator instead of templates?

To find out, three of the most starred charts on Artifact Hub were rebuilt as one Kapitan project on the omegaconf backend: metrics-server, cert-manager and kube-prometheus-stack. The project and the script that compares them are in kadet-hub/examples/chart-vs-kapitan, and CI runs the comparison on every change.

How the two sides are compared

“Looks the same” is not enough, so the check asks the API server. Both renderings go through kubectl apply --dry-run=server on a kind cluster, which applies the defaults Kubernetes adds on its own. Then names, labels, selectors and list order are normalized, because the generator names objects by its own convention. What remains has to be equal.

RBAC is compared by what it grants, not by how it is written: kubectl auth can-i --list for every ServiceAccount of the chart, and for a user bound only to the built-in view role, where aggregated ClusterRoles land. Finally each side is installed on its own fresh cluster and has to work: kubectl top nodes for metrics-server, a Certificate issued by a self-signed ClusterIssuer for cert-manager. For kube-prometheus-stack the comparison is what both installs do, described in its section.

Results

metrics-server 3.14.0 cert-manager v1.21.2 kube-prometheus-stack 92.11.0
Objects equal (4 outside RBAC) equal (17 outside RBAC, CRDs included) 114 vs 126, compared by behaviour
Permissions equal equal not compared, see below
Works on kind kubectl top nodes Certificate Ready same scrape jobs and health, same rules, Grafana dashboards
Chart templates 761 lines 15,803 lines 24,928 lines with subcharts
Chart values.yaml 248 lines 1,758 lines 9,920 lines with subcharts
Kapitan inventory 63 lines 161 lines 406 lines, plus 101 of Jsonnet

The chart’s numbers are what its authors maintain. The inventory’s numbers are what you would maintain instead; the code that turns them into objects is the kadet-hub kubernetes and prometheus-operator generators, versioned, signed and maintained once for every project that uses them. Someone who only installs a chart writes far less: 2 lines of values for metrics-server on kind, 4 for cert-manager. That is the comparison the first paragraph of this post sets aside.

metrics-server

metrics-server is a Deployment, a Service, an APIService and RBAC. The inventory declares the port once, and the generator derives the container port, the Service port, the Service’s appProtocol from the port name and the APIService that points at the Service:

components:
  metrics-server:
    ports:
      https:
        container_port: 10250
        service_port: 443
    service:
      type: ClusterIP
    api_service: {group: metrics.k8s.io, version: v1beta1, insecure_skip_tls_verify: true}
    cluster_role:
      rules:
        - {apiGroups: [""], resources: [nodes/metrics], verbs: [get]}
        - {apiGroups: [""], resources: [pods, nodes, namespaces, configmaps], verbs: [get, list, watch]}

cluster_role on the component creates the ClusterRole and binds it to the component’s own ServiceAccount, so no name is written twice. Two differences remain by design. RBAC objects cannot be named system:metrics-server, because the generator also writes the name as a label value, where colons are not allowed. And probes need timeout_seconds: 1 to match the chart, because the generator defaults to 5 seconds where Kubernetes defaults to 1.

cert-manager

cert-manager has three Deployments, two webhook configurations, six CRDs and 13 ClusterRoles. The inventory has one cluster_role per component instead of 13: the controller’s nine roles become one list of 37 rules, and can-i shows the same permissions. Shared pod settings, such as the security context, the metrics port and the POD_NAMESPACE variable, are written once in generators.manifest.default_config. Leader election Roles in kube-system sit on their component with role.namespace. The CRDs are cert-manager’s release file, pinned by sha256, not part of either count.

The webhooks only state what is specific to them:

webhook_annotations:
  cert-manager.io/inject-ca-from-secret: cert-manager/cert-manager-webhook-ca
webhooks:
  - name: webhook.cert-manager.io
    rules: [{apiGroups: [cert-manager.io], apiVersions: [v1], operations: [CREATE], resources: [certificaterequests]}]
    timeoutSeconds: 30
    clientConfig: {service: {path: /mutate}}

The Service, admissionReviewVersions: [v1] and sideEffects: None come from the generator. One behavioural difference is intended: the generator keeps a webhook out of its own namespace, so it cannot block the pods that serve it. The chart’s mutating webhook does not. The startupapicheck Job is switched off on the chart side, because it is a Helm hook and Kapitan has no hooks.

kube-prometheus-stack

kube-prometheus-stack installs the Prometheus Operator, kube-state-metrics, node-exporter, Grafana with its dashboard sidecars, a Prometheus and an Alertmanager, 13 ServiceMonitors, 35 PrometheusRules and about 30 dashboard ConfigMaps. Most of its rules and dashboards are not written by hand: a sync script generates them from kube-prometheus and kubernetes-mixin, which are Jsonnet. Kapitan compiles Jsonnet natively, so the Kapitan side builds them from the same upstream commits the chart’s sync uses, pinned in a jsonnetfile.lock.json. The workloads come from the kubernetes generator, Prometheus, Alertmanager and the ServiceMonitors from the prometheus-operator generator, and the Services for the pods the operator and the cluster run from the kubernetes generator’s services section.

Object for object comparison stops making sense here, so each side is installed on its own cluster and asked what it does:

34 of the chart’s 35 rule groups and all 29 dashboards come out equal. The 35th, k8s.rules.container_resource, is left over from an older sync: the pinned sources produce the same eight recording rules split into four groups, which the chart ships but switches off. Permissions are not compared for this chart, because the prometheus-operator generator’s Prometheus RBAC differs from the chart’s by one verb and one path.

For the comparison both sides switch off the operator’s admission webhooks and the TLS that goes with them, because the chart creates their certificates with Helm hooks.

What the comparison changed

Rebuilding the charts showed values the inventory had to repeat because the generator could not derive them. Each became a default in the kubernetes generator, overridable where it can be wrong:

Version Change
1.3.0 appProtocol on Service ports
1.4.0 appProtocol from the port name; a component’s webhooks point at its Service and default admissionReviewVersions and sideEffects; role.namespace for leader election Roles; cert-manager’s CA injection annotations
1.5.0 api_service on a component
1.6.0 Services for pods no component renders, such as Prometheus and the control-plane pods; external_name

Where the chart stays ahead

A chart is the one artifact every Kubernetes tool consumes. Helm hooks run Jobs at install time, such as cert-manager’s startupapicheck or kube-prometheus-stack’s certificate generator, which an inventory replaces with something else, like a Job you apply first. And a chart’s users expect helm install with a values file, not a Kapitan project. The inventory pays off for the people who maintain the chart’s logic, for platform teams that run the same apps in many clusters, and wherever one value has to reach several objects: the port that is also the Service port, the probe port and the APIService port.