kadet-hub logokadet-hub

MLflow on Kubernetes without Helm, with Kapitan

MLflow 3.18.0, the tracking server with its model registry and basic auth, on the PostgreSQL of the included recipe postgresql with artifacts on a claim, rendered by the kadet-hub kubernetes generator 1.14.0 from MLflow’s Helm chart mlflow 3.18.0.

What it deploys

Required parameters

applications.mlflow.admin_password is the password of the basic auth user admin, at least 12 characters. MLflow creates admin with it on the first start against an empty user store and ignores it once the user exists, so a later change of the parameter does not reach the account. Change the password through PATCH /api/2.0/mlflow/users/update-password as admin. applications.mlflow.secret_key is MLFLOW_FLASK_SERVER_SECRET_KEY, which signs the CSRF tokens of the sign-up page. MLflow does not start with basic auth without it. Set both as refs with the backend of your choice. Without one of them the compile stops with Interpolation key '<key>' not found.

Include

A target includes the recipe and its options as classes and can override anything the recipe sets:

classes:
  - recipes.mlflow
parameters:
  namespace: mlflow
  kapitan:
    vars:
      target: mlflow
  applications:
    mlflow:
      admin_password: <ref>
      secret_key: <ref>

Fetching it

The quickstart goes through these steps on a kind cluster with the uptime-kuma recipe, from installing Kapitan to the running app.

A separate fetch inventory pins the recipe by version and digest. This is fetch/targets/recipes.yml for version 1.0.0:

parameters:
  kapitan:
    dependencies:
      - type: oci
        source: ghcr.io/kadet-hub/recipes/mlflow:1.0.0@sha256:75cdc3dd57ded69bd0b99f40ac6816df68816e02d47006a38e30d0341d573e99
        output_path: inventory/classes/recipes/mlflow

Version 1.0.0 includes postgresql 1.1.1. The consumer workflow adds the included recipes at the digests the index lists. Kapitan's --fetch reads no index, so for a local compile pull them by digest first:

tmp=$(mktemp -d)
oras pull -o "$tmp" ghcr.io/kadet-hub/recipes/postgresql@sha256:1e8465058c256cc86995dc8d6f1eb2f095df244fadcaf756935599f4d9ead6a9
mkdir -p inventory/classes/recipes/postgresql
tar -xf "$tmp/postgresql.tar" -C inventory/classes/recipes/postgresql --no-same-owner

The first command fetches the recipe into inventory/classes/recipes/mlflow, the second the generators and compiles:

kapitan compile --inventory-path fetch --fetch --output-path .
kapitan compile --fetch

The first command exits with status 1 because the fetch target compiles nothing. Kapitan's --fetch checks neither the digest nor the attestation. This verifies the attestation of the recipe and of the recipes it includes:

gh attestation verify \
  oci://ghcr.io/kadet-hub/recipes/mlflow@sha256:75cdc3dd57ded69bd0b99f40ac6816df68816e02d47006a38e30d0341d573e99 \
  --owner kadet-hub \
  --cert-identity https://github.com/kadet-hub/registry/.github/workflows/release.yml@refs/heads/main

gh attestation verify \
  oci://ghcr.io/kadet-hub/recipes/postgresql@sha256:1e8465058c256cc86995dc8d6f1eb2f095df244fadcaf756935599f4d9ead6a9 \
  --owner kadet-hub \
  --cert-identity https://github.com/kadet-hub/registry/.github/workflows/release.yml@refs/heads/main

The attested index must also list each digest under its recipe with the pinned version, and not as yanked. Verifying by hand shows how to read it.

Settings

Chart value Inventory
image.tag components.mlflow-mlflow.image, tag and digest
server.value_options.allowed_hosts, the host names clients reach MLflow by applications.mlflow.allowed_hosts, by default the Service’s names in the cluster on port 5000
server.value_options.workers, 4 components.mlflow-mlflow.command, the element --workers=4
storage.size, 10Gi components.mlflow-mlflow.persistent_volume_claims.mlflow-mlflow.storage
resources components.mlflow-mlflow.resources
env components.mlflow-mlflow.env.<variable>
the Cluster, its database and its size applications.postgresql, which postgresql’s README lists

MLflow answers only requests whose Host header is localhost, a private address or one of applications.mlflow.allowed_hosts. Add the host name of an Ingress or route of the target there. Lists set in the target are appended to the recipe’s lists, so command changes only in a class included after the recipe that sets it to ~, with the new list in the target.

The chart’s Ingress, HorizontalPodAutoscaler, NetworkPolicy, ServiceMonitor, garbage collection CronJob, TLS and External Secrets objects are off by default and not part of the recipe. The chart renders no hook. Its ServiceAccount has no role, and neither it nor the pod mounts a token.

Proof

check mlflow --level deploys with tests/values.yaml and tests/target.yml, on kind for each Kubernetes minor CI tests, with check 0.16.0 and cnpg installed first. The objects of postgresql are applied with the recipe and left out of the comparison. The remaining differences are of the classes Selector scheme, Derived name (the container mlflow-mlflow), Explicit default (the pod’s automountServiceAccountToken: false, as its ServiceAccount sets it) and Referenced Secret (the Secret mlflow-mlflow with the admin password, the Flask secret key and basic_auth.ini).

tests/smoke starts a pod of the MLflow image and talks to the Service mlflow-mlflow on port 5000. It requires 401 for a request and a sign-up without a login and for a wrong password. As admin with the password of the Secret, it logs a run with a parameter, a metric and an artifact, registers version 1 of the model smoke from the run and reads all of it back. It then deletes the pod and reads everything again from its successor.

Not tested: the web UI, users other than admin, permissions, and an Ingress or route in front of the Service.

Versions

The recipe's files are in kadet-hub/recipes. All recipes