kserve

KServe Generator

Published on kadet-hub: pin a released version by digest from its catalog page.

Kapitan kadet generator for KServe model serving resources (serving.kserve.io). Built for KServe 0.21.0.

• InferenceService (v1beta1): predictor, transformer and explainer of one model
• InferenceGraph (v1alpha1): routing between InferenceServices (Sequence, Splitter, Ensemble, Switch)
• ServingRuntime, ClusterServingRuntime (v1alpha1): model server templates
• TrainedModel (v1alpha1): a model loaded into a multi-model InferenceService
• ClusterStorageContainer (v1alpha1): storage initializer per URI scheme
• LocalModelCache, LocalModelNodeGroup (v1alpha1): models cached on node-local volumes
• LLMInferenceService, LLMInferenceServiceConfig (v1alpha2): LLM serving with llm-d

Every kind is emitted in the storage version of KServe 0.21.0. v1alpha1 of the two LLM kinds is still served, but the CRDs store v1alpha2. LocalModelNode is not rendered: the LocalModelCache controller creates and updates it. Model artifacts, Secrets and the Gateway belong elsewhere.

Usage

Kapitan Configuration

parameters:
  kapitan:
    dependencies:
      - type: oci
        source: ghcr.io/kadet-hub/kserve@sha256:<digest>
        output_path: lib/kserve
    compile:
      - output_path: manifests/kserve/
        input_type: kadet
        output_type: yml
        input_paths:
          - lib/kserve

Inventory Contract

All keys live under parameters.generators.kserve.

Every resource accepts labels, annotations and spec; namespaced kinds also accept namespace. spec is merged into the resource spec first, then the flat keys below are applied; a flat key that overrides a spec value logs a warning.

Flat keys are the snake_case spelling of the CRD spec fields:

Values under these keys pass through unchanged, except:

The controller merges an InferenceService’s predictor.model into the runtime container named kserve-container (InferenceServiceContainerName, predictor.go), and a container without a name fails the CRD schema, so the default only fills a gap the API server would reject. It applies to spec.containers too. The controller binds the node group’s PVC to its PV by volumeName (utils.go); the empty class keeps the cluster’s default StorageClass off a PVC whose PV has none, which would otherwise never bind.

protocol_version(s), scale_metric, router_type and workload_type are checked against the CRD enums at compile time. Model format names and the TrainedModel framework are free strings in the CRD and pass unchecked; the installed runtimes decide what is served. model.criticality and model.lora.adapters[].criticality of the LLM kinds no longer exist in v1alpha2 and are dropped with a warning, so inventories written for v1alpha1 still compile.

Exposure: InferenceServices are externally visible through the cluster’s ingress unless labelled networking.kserve.io/visibility: cluster-local. The generator sets no router on LLM kinds; router: {route: {}} creates an HTTPRoute on the Gateway configured in the KServe operator, which is often an external one.

Example

parameters:
  generators:
    kserve:
      namespace: ml-serving
      serving_runtimes:
        sklearn-runtime:
          containers:
            - image: kserve/sklearnserver:v0.21.0
              args:
                - --model_name={{.Name}}
                - --model_dir=/mnt/models
                - --http_port=8080
          supported_model_formats:
            - name: sklearn
              version: "1"
              autoSelect: true
          protocol_versions: [v1, v2]
      inference_services:
        sklearn-iris:
          predictor:
            min_replicas: 1
            max_replicas: 3
            model:
              model_format: sklearn
              runtime: sklearn-runtime
              protocol_version: v2
              storage_uri: s3://models/sklearn/iris/v1
        sklearn-iris-v2:
          predictor:
            min_replicas: 1
            model:
              model_format: sklearn
              runtime: sklearn-runtime
              protocol_version: v2
              storage_uri: s3://models/sklearn/iris/v2
      inference_graphs:
        iris-split:
          nodes:
            root:
              router_type: Splitter
              steps:
                - service_name: sklearn-iris
                  weight: 80
                - service_name: sklearn-iris-v2
                  weight: 20
      llm_inference_service_configs:
        vllm-gpu:
          template:
            containers:
              - name: main
                resources:
                  limits:
                    nvidia.com/gpu: "1"
      llm_inference_services:
        llama-chat:
          base_refs: [vllm-gpu]
          model:
            uri: hf://meta-llama/Llama-3.1-8B-Instruct
            name: meta-llama/Llama-3.1-8B-Instruct
          replicas: 2

tests/fixtures/inventory/targets/complete.yml covers every kind, including multi-model serving with a TrainedModel and a local model cache.

Schemas

schemas/serving.kserve.io holds the JSON Schemas of the KServe 0.21.0 CRDs (Apache-2.0), closed against unknown fields; schemas/sources.yaml names the CRD files.