Worker node upgrade chain

.gitignore

@@ -1,3 +1,6 @@
+build.env.work
+cluster.env.work
+.DS_Store
 clitools/bin
 packages/
 out/

README.md

@@ -1,68 +1,87 @@
 # monok8s
-An Alpine-based Kubernetes cluster image for Mono's Gateway Development Kit
-https://docs.mono.si/gateway-development-kit/getting-started
 
-## Features
+This is an Alpine-based Kubernetes image for Mono's Gateway Development Kit.
- * A/B deployment
- * Read-only RootFS
- * k8s style OS upgrade (see Upgrading)
 
-## DISCLAIMER
+It gives you a ready-to-boot Kubernetes control-plane image so you can get your device running first, then learn and customize from there.
 
- * This is not your everyday linux image! It is best suited for users that is already familiar
+Project/device docs: <https://docs.mono.si/gateway-development-kit/getting-started>
-   with k8s. For first-timers, you may want to try the default config that gives you a ready-to-use
-   cluster and get yourself started from there
 
- * USE AT YOUR OWN RISKS. I leverage ChatGPT heavily for this.
+---
 
-## Current Status
+## What you get
 
-### Boostrapping
+The default image boots into a small Kubernetes control-plane environment with:
- * [x] initramfs
- * [x] booting into alpine
- * [x] k8s control-plane
- * [ ] k8s worker node
 
-### Kubernetes
+- Alpine Linux
- * OSUpgrade
+- Kubernetes initialized through `kubeadm`
-   * [x] Control Plane - kubeadm upgrade apply
+- read-only root filesystem layout
-   * Upgrade chain
+- A/B rootfs layout for safer OS upgrades
-     * [x] 1.33.3 -> 1.33.10
+- a Kubernetes-style OS upgrade path through `OSUpgrade`
-     * [x] 1.33.10 -> 1.34.6
-     * [x] 1.34.6 -> 1.35.3
-   * [ ] Worker node - kubeadm upgrade node
- * CNI
-   * [x] default bridge-cni
-   * [ ] Cilium
 
-### Network Traffics
+You do **not** need to know Go or understand the internal build system to try the image.
- * VPP Pod
-   * [x] fmc - works? But no way to test it yet
-   * [ ] vpp - does not work
 
-## Table of Contents
+---
-1. Flashing
-   - [USB](docs/flashing-usb.md)
-   - [Over network (tftp)](docs/flashing-network.md)
-2. [Upgrading](docs/ota.md)
-3. Getting shell access to the host
-   - UART. The thing you did in first time flashing.
-   - [Install an ssh pod](docs/installing-ssh-pod.md) (Recommended)
 
-## Build
+## Before you start
 
-Prerequisites
+You need:
-* make
-* Docker
 
+- a Linux build machine or VM
+- Docker
+- `make`
+- basic command-line comfort
+
+If you are building on a fresh Debian machine, you can install the usual build dependencies with:
+
+```bash
+devtools/setup-build-host.sh
 ```
+
+Or install the minimum packages yourself:
+
+```bash
+sudo apt-get update
+sudo apt-get install -y docker.io make qemu-user-static binfmt-support
+```
+
+Make sure your user can run Docker, or use `sudo` where needed.
+
+---
+
+## Fast path: build an image
+
+Download the project tarball, extract it, then run:
+
+```bash
 make release
 ```
 
-The default configuration will boot as a first time control-plane.
+When the build finishes, check the `out/` directory for the generated image artifacts.
 
-For control-plane
+That is the main path most users should try first.
-```
+
+---
+
+## Flash the image
+
+After building, flash the generated image to your device.
+
+Start with one of these guides:
+
+- [Flash over USB](docs/flashing-usb.md)
+- [Flash over network / TFTP](docs/flashing-network.md)
+
+USB flashing is usually the easiest path when you are setting up the device for the first time.
+
+---
+
+## First boot
+
+The default configuration is intended to boot as a first-time Kubernetes control-plane node.
+
+Default-style control-plane configuration looks like this:
+
+```bash
 make cluster-config \
 	MKS_HOSTNAME=monok8s-master \
 	MKS_CLUSTER_ROLE=control-plane \
@@ -71,7 +90,13 @@ make cluster-config \
 	MKS_APISERVER_ADVERTISE_ADDRESS=10.0.0.10
 ```
 
-For worker
+If you are just trying the image for the first time, start with the default control-plane setup. Worker-node setup is still incomplete.
+
+For all available configuration values, see:
+
+- [configs/cluster.env.default](configs/cluster.env.default)
+
+For worker node
 ```
 make cluster-config \
 	MKS_HOSTNAME=monok8s-worker \
@@ -85,12 +110,87 @@ make cluster-config \
 	MKS_DISCOVERY_TOKEN_CA_CERT_HASH=sha256:9f1c2b3a4d5e6f7890abc1234567890abcdef1234567890abcdef1234567890ab
 ```
 
-Check inside [configs/cluster.env.default](configs/cluster.env.default) for configuration details
+---
+
+## Getting shell access
+
+For first-time setup, UART is the most direct option because it is already part of the flashing process.
+
+After the device is running, the recommended path is:
+
+- [Install an SSH pod](docs/installing-ssh-pod.md)
+
+---
+
+## Upgrading
+
+monok8s includes a Kubernetes-style OS upgrade flow using the `OSUpgrade` custom resource.
+
+See:
+
+- [OTA upgrade guide](docs/ota.md)
+
+The currently tested upgrade chain is:
+
+- `1.33.3 -> 1.33.10`
+- `1.33.10 -> 1.34.6`
+- `1.34.6 -> 1.35.3`
+
+Tested worker node upgrade chain:
+
+- `1.33.3 -> 1.34.1`
+- `1.33.1 -> 1.35.3`
+
+---
+
+## Current status
+
+This project is usable for experimenting with a single control-plane device image, but it is still a development project.
+
+Working today:
+
+- initramfs boot flow
+- Alpine boot
+- Kubernetes control-plane bootstrap
+- default bridge CNI
+- control-plane OS upgrade path
+
+Still in progress:
+
+- Kubernetes worker-node support
+- Cilium support
+- VPP/DPAA networking experiments
+
+---
+
+## Common build issue
+
+### `chroot: failed to run command '/bin/sh': Exec format error`
+
+This usually means the build host cannot run ARM64 binaries.
+
+On Debian, install ARM64 emulation support:
 
-### Making sub stages
 ```bash
-make build-base # The image that builds the kernel and everything
+sudo apt-get install -y qemu-user-static binfmt-support
-make kernel     # Builds our kernel from NXP
-make initramfs
-make itb        # Builds out/board.itb (contains the kernel and the initramfs)
 ```
+
+Then run the build again:
+
+```bash
+make release
+```
+
+---
+
+## Notes
+
+This is not a general-purpose Linux distribution. It is a device image for experimenting with Kubernetes on Mono's Gateway Development Kit.
+
+The safest path is:
+
+1. build the default image,
+2. flash it,
+3. boot the control-plane,
+4. confirm Kubernetes is running,
+5. customize only after the base image works.

alpine/build-rootfs.sh

@@ -2,9 +2,11 @@
 
 set -euo pipefail
 
+source /utils.sh
+
 /preload-k8s-images.sh || exit 1
 
-export CTL_BIN_LAYER=$( skopeo inspect docker-daemon:localhost/monok8s/control-agent:dev | jq -r '.Layers[0] | sub("^sha256:"; "")' )
+export CTL_BIN_LAYER=$( skopeo inspect docker-daemon:localhost/monok8s/node-control:$TAG | jq -r '.Layers[0] | sub("^sha256:"; "")' )
 
 mkdir -p \
   "$ROOTFS/dev" \
@@ -117,6 +119,8 @@ mkdir -p "$FAKE_DEV" "$MNT_ROOTFS_IMG" "$MNT_DATA"
 
 echo "##################################################### Packaging RootFS $(du -sh "$ROOTFS" | awk '{print $1}')"
 
+ensure_loop_ready
+
 ###############################################################################
 # 1. Build reusable rootfs ext4 image once
 ###############################################################################

alpine/preload-k8s-images.sh

@@ -26,7 +26,7 @@ FUSE_OVERLAYFS="${FUSE_OVERLAYFS:-/usr/bin/fuse-overlayfs}"
 #   )
 EXTRA_IMAGES=(
 	"${EXTRA_IMAGES[@]:-}"
-	"docker-daemon:localhost/monok8s/control-agent:$TAG"
+	"docker-daemon:localhost/monok8s/node-control:$TAG"
 )
 
 # Keep archive cache version/arch scoped so downloads do not get mixed.

alpine/rootfs-extra/etc/local.d/monok8s.start

@@ -38,4 +38,4 @@ if [ -n "$K8S_MINOR" ]; then
 	"$MIGRATION_STATE_DIR/k8s/$K8S_MINOR"
 fi
 
-/usr/local/bin/ctl init --env-file "$CONFIG_DIR/cluster.env" >>/var/log/monok8s/bootstrap.log 2>&1 &
+/usr/lib/monok8s/lib/supervised-init.sh &

alpine/rootfs-extra/usr/lib/monok8s/lib/supervised-init.sh

@@ -0,0 +1,57 @@
+#!/bin/sh
+set -eu
+
+CONFIG_DIR=/opt/monok8s/config
+LOG=/var/log/monok8s/bootstrap.log
+STATE_DIR=/run/monok8s
+FAIL_COUNT_FILE="$STATE_DIR/bootstrap-fail-count"
+LOCK_DIR="$STATE_DIR/supervised-init.lock"
+
+# For debugging
+HOLD_FILE="$CONFIG_DIR/bootstrap.hold"
+
+mkdir -p "$STATE_DIR" /var/log/monok8s
+
+if ! mkdir "$LOCK_DIR" 2>/dev/null; then
+	echo "[$(date -Is)] supervised-init already running" >> "$LOG"
+	exit 0
+fi
+
+trap 'rmdir "$LOCK_DIR"' EXIT INT TERM
+
+fail_count=0
+if [ -f "$FAIL_COUNT_FILE" ]; then
+	fail_count="$(cat "$FAIL_COUNT_FILE" 2>/dev/null || echo 0)"
+	case "$fail_count" in
+		''|*[!0-9]*) fail_count=0 ;;
+	esac
+fi
+
+while true; do
+	if [ -f "$HOLD_FILE" ]; then
+		echo "[$(date -Is)] bootstrap held by $HOLD_FILE" >> "$LOG"
+		sleep 300
+		continue
+	fi
+
+	echo "[$(date -Is)] starting ctl init" >> "$LOG"
+
+	if /usr/local/bin/ctl init --env-file "$CONFIG_DIR/cluster.env" >> "$LOG" 2>&1; then
+		echo "[$(date -Is)] ctl init succeeded" >> "$LOG"
+		rm -f "$FAIL_COUNT_FILE"
+		exit 0
+	fi
+
+	fail_count=$((fail_count + 1))
+	echo "$fail_count" > "$FAIL_COUNT_FILE"
+
+	echo "[$(date -Is)] ctl init failed, count=$fail_count" >> "$LOG"
+
+	case "$fail_count" in
+		1) sleep 10 ;;
+		2) sleep 30 ;;
+		3) sleep 60 ;;
+		4) sleep 120 ;;
+		*) sleep 300 ;;
+	esac
+done

alpine/utils.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+
+ensure_loop_ready() {
+    # The loop kernel module is host-side. This only works if the container
+    # has permission and modprobe exists; otherwise the host must load it.
+    if ! grep -qw loop /proc/modules 2>/dev/null; then
+        modprobe loop 2>/dev/null || true
+    fi
+
+    # /dev/loop-control: char device 10:237
+    if [ ! -e /dev/loop-control ]; then
+        echo "Creating missing /dev/loop-control" >&2
+        mknod /dev/loop-control c 10 237 || {
+            echo "ERROR: cannot create /dev/loop-control" >&2
+            echo "Run container with --privileged, or pass --device=/dev/loop-control and loop devices." >&2
+            exit 1
+        }
+        chmod 600 /dev/loop-control || true
+    fi
+
+    if [ ! -c /dev/loop-control ]; then
+        echo "ERROR: /dev/loop-control exists but is not a character device" >&2
+        ls -l /dev/loop-control >&2 || true
+        exit 1
+    fi
+
+    # Create a reasonable pool of loop block devices.
+    # loopN block devices are major 7, minor N.
+    for i in $(seq 0 31); do
+        if [ ! -e "/dev/loop$i" ]; then
+            echo "Creating missing /dev/loop$i" >&2
+            mknod "/dev/loop$i" b 7 "$i" || {
+                echo "ERROR: cannot create /dev/loop$i" >&2
+                echo "Run container with --privileged, or pre-create/pass loop devices." >&2
+                exit 1
+            }
+            chmod 660 "/dev/loop$i" || true
+        fi
+
+        if [ ! -b "/dev/loop$i" ]; then
+            echo "ERROR: /dev/loop$i exists but is not a block device" >&2
+            ls -l "/dev/loop$i" >&2 || true
+            exit 1
+        fi
+    done
+
+    # Smoke test: ask losetup for a free loop device.
+    if ! losetup -f >/dev/null 2>&1; then
+        echo "ERROR: losetup cannot find/use a loop device" >&2
+        echo "Debug info:" >&2
+        ls -l /dev/loop-control /dev/loop* >&2 || true
+        grep -w loop /proc/modules >&2 || true
+        echo >&2
+        echo "Docker likely needs --privileged, or at minimum CAP_SYS_ADMIN plus loop devices." >&2
+        exit 1
+    fi
+}

build.env

@@ -41,4 +41,9 @@ ALPINE_HOSTNAME=monok8s-hostname
 BUILD_TAG=MONOK8S
 
 # Optional apt cache
-APT_PROXY=apt-cacher-ng.eco-system.svc.cluster.local:3142
+#   example: apt-cacher-ng.eco-system.svc.cluster.local:3142
+APT_PROXY=
+
+# remote image repository prefix to push to
+# e.g. ghcr.io/monok8s
+IMAGE_REPOSITORY=

clitools/devtools/run.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+SCRIPT_DIR="$(dirname "${BASH_SOURCE[0]}")"
+PROJ_ROOT="$( realpath "$SCRIPT_DIR"/../ )"
+OUT_DIR="$PROJ_ROOT"/out
+
+if [ "$1" == "controller" ]; then
+	if [ -f "$OUT_DIR/tls.key" ] && [ -f "$OUT_DIR/tls.crt" ]; then
+		echo "Use existing certs"
+	else
+		echo "Generating self signed certs"
+		openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
+		  -keyout "$OUT_DIR"/tls.key -out "$OUT_DIR"/tls.crt \
+		  -subj "/CN=127.0.0.1" \
+		  -addext "subjectAltName=IP:127.0.0.1,DNS:localhost"
+	fi
+	go run "$PROJ_ROOT"/cmd/ctl $@ --tls-cert-file "$OUT_DIR"/tls.crt --tls-private-key-file "$OUT_DIR"/tls.key
+else
+	go run "$PROJ_ROOT"/cmd/ctl $@
+fi

clitools/docker/ctl-agent.Dockerfile

@@ -1,16 +1,41 @@
+ARG BASE_IMAGE=localhost/monok8s/ctl-build-base:dev
+
+FROM --platform=$BUILDPLATFORM ${BASE_IMAGE} AS build
+
+ARG VERSION=dev
+ARG TARGETOS
+ARG TARGETARCH
+
+WORKDIR /src
+
+COPY . .
+
+RUN test -f pkg/buildinfo/buildinfo_gen.go
+
+RUN mkdir -p /out && \
+    GOOS=${TARGETOS} GOARCH=${TARGETARCH} CGO_ENABLED=0 \
+    go build -trimpath -ldflags="-s -w" \
+      -o /out/ctl ./cmd/ctl
+
+
 FROM alpine:latest AS cacerts
+
+
 FROM scratch
 
 ARG VERSION
+ARG TARGETOS
+ARG TARGETARCH
 ENV VERSION=${VERSION}
 
 WORKDIR /
 
-COPY bin/ctl-linux-aarch64-${VERSION} ./ctl
+COPY --from=build /out/ctl /ctl
-COPY out/fw_printenv ./
-COPY out/fw_setenv ./
 COPY --from=cacerts /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
 
+COPY out/uboot-tools/${TARGETOS}_${TARGETARCH}/fw_printenv /fw_printenv
+COPY out/uboot-tools/${TARGETOS}_${TARGETARCH}/fw_setenv /fw_setenv
+
 ENV PATH=/
 
 ENTRYPOINT ["/ctl"]

clitools/docker/ctl-builder-local.Dockerfile

@@ -1,24 +0,0 @@
-FROM golang:1.26-alpine AS build
-
-ARG VERSION
-ARG KUBE_VERSION
-ARG GIT_REV=unknown
-
-WORKDIR /src
-
-RUN apk add --no-cache git build-base
-
-COPY go.mod go.sum ./
-RUN go mod download
-
-COPY . .
-
-RUN test -f pkg/buildinfo/buildinfo_gen.go
-
-RUN mkdir -p /out && \
-    GOOS=darwin GOARCH=arm64 CGO_ENABLED=0 \
-    go build -trimpath -ldflags="-s -w" \
-      -o /out/ctl-${VERSION} ./cmd/ctl
-
-FROM scratch
-COPY --from=build /out/ /

clitools/docker/ctl-builder.Dockerfile

@@ -1,20 +0,0 @@
-ARG BASE_IMAGE=localhost/monok8s/ctl-build-base:dev
-FROM ${BASE_IMAGE} AS build
-
-ARG VERSION=dev
-ARG TARGETOS=linux
-ARG TARGETARCH=arm64
-
-WORKDIR /src
-
-COPY . .
-
-RUN test -f pkg/buildinfo/buildinfo_gen.go
-
-RUN mkdir -p /out && \
-    GOOS=${TARGETOS} GOARCH=${TARGETARCH} CGO_ENABLED=0 \
-    go build -trimpath -ldflags="-s -w" \
-      -o /out/ctl-linux-aarch64-${VERSION} ./cmd/ctl
-
-FROM scratch
-COPY --from=build /out/ /

clitools/go.mod

@@ -3,31 +3,37 @@ module example.com/monok8s
 go 1.26.0
 
 require (
+	github.com/emicklei/go-restful/v3 v3.12.2
 	github.com/klauspost/compress v1.18.5
-	github.com/spf13/cobra v1.9.1
+	github.com/spf13/cobra v1.10.2
-	golang.org/x/sys v0.31.0
+	golang.org/x/sys v0.39.0
 	gopkg.in/yaml.v3 v3.0.1
-	k8s.io/api v0.34.0
+	k8s.io/api v0.35.0
-	k8s.io/apiextensions-apiserver v0.34.0
+	k8s.io/apiextensions-apiserver v0.35.0
-	k8s.io/apimachinery v0.34.0
+	k8s.io/apimachinery v0.35.0
+	k8s.io/apiserver v0.35.0
 	k8s.io/cli-runtime v0.34.0
-	k8s.io/client-go v0.34.0
+	k8s.io/client-go v0.35.0
+	k8s.io/code-generator v0.35.0
 	k8s.io/klog/v2 v2.130.1
+	sigs.k8s.io/controller-tools v0.20.1
 	sigs.k8s.io/yaml v1.6.0
 )
 
 require (
 	github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect
+	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/blang/semver/v4 v4.0.0 // indirect
+	github.com/cespare/xxhash/v2 v2.3.0 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
-	github.com/emicklei/go-restful/v3 v3.12.2 // indirect
+	github.com/fatih/color v1.18.0 // indirect
 	github.com/fxamacker/cbor/v2 v2.9.0 // indirect
 	github.com/go-errors/errors v1.4.2 // indirect
-	github.com/go-logr/logr v1.4.2 // indirect
+	github.com/go-logr/logr v1.4.3 // indirect
 	github.com/go-openapi/jsonpointer v0.21.0 // indirect
 	github.com/go-openapi/jsonreference v0.20.2 // indirect
 	github.com/go-openapi/swag v0.23.0 // indirect
-	github.com/gogo/protobuf v1.3.2 // indirect
+	github.com/gobuffalo/flect v1.0.3 // indirect
 	github.com/google/btree v1.1.3 // indirect
 	github.com/google/gnostic-models v0.7.0 // indirect
 	github.com/google/uuid v1.6.0 // indirect
@@ -37,30 +43,42 @@ require (
 	github.com/json-iterator/go v1.1.12 // indirect
 	github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect
 	github.com/mailru/easyjson v0.7.7 // indirect
+	github.com/mattn/go-colorable v0.1.13 // indirect
+	github.com/mattn/go-isatty v0.0.20 // indirect
 	github.com/moby/term v0.5.0 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect
 	github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
 	github.com/peterbourgon/diskv v2.0.1+incompatible // indirect
-	github.com/pkg/errors v0.9.1 // indirect
+	github.com/prometheus/client_golang v1.23.2 // indirect
-	github.com/spf13/pflag v1.0.6 // indirect
+	github.com/prometheus/client_model v0.6.2 // indirect
+	github.com/prometheus/common v0.66.1 // indirect
+	github.com/prometheus/procfs v0.16.1 // indirect
+	github.com/spf13/pflag v1.0.10 // indirect
 	github.com/x448/float16 v0.8.4 // indirect
 	github.com/xlab/treeprint v1.2.0 // indirect
-	go.yaml.in/yaml/v2 v2.4.2 // indirect
+	go.opentelemetry.io/otel v1.36.0 // indirect
+	go.opentelemetry.io/otel/trace v1.36.0 // indirect
+	go.yaml.in/yaml/v2 v2.4.3 // indirect
 	go.yaml.in/yaml/v3 v3.0.4 // indirect
-	golang.org/x/net v0.38.0 // indirect
+	golang.org/x/mod v0.31.0 // indirect
-	golang.org/x/oauth2 v0.27.0 // indirect
+	golang.org/x/net v0.48.0 // indirect
-	golang.org/x/sync v0.12.0 // indirect
+	golang.org/x/oauth2 v0.30.0 // indirect
-	golang.org/x/term v0.30.0 // indirect
+	golang.org/x/sync v0.19.0 // indirect
-	golang.org/x/text v0.23.0 // indirect
+	golang.org/x/term v0.38.0 // indirect
+	golang.org/x/text v0.32.0 // indirect
 	golang.org/x/time v0.9.0 // indirect
-	google.golang.org/protobuf v1.36.5 // indirect
+	golang.org/x/tools v0.40.0 // indirect
-	gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
+	google.golang.org/protobuf v1.36.8 // indirect
+	gopkg.in/evanphx/json-patch.v4 v4.13.0 // indirect
 	gopkg.in/inf.v0 v0.9.1 // indirect
-	k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b // indirect
+	gopkg.in/yaml.v2 v2.4.0 // indirect
-	k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 // indirect
+	k8s.io/component-base v0.35.0 // indirect
-	sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 // indirect
+	k8s.io/gengo/v2 v2.0.0-20250922181213-ec3ebc5fd46b // indirect
+	k8s.io/kube-openapi v0.0.0-20250910181357-589584f1c912 // indirect
+	k8s.io/utils v0.0.0-20251002143259-bc988d571ff4 // indirect
+	sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 // indirect
 	sigs.k8s.io/kustomize/api v0.20.1 // indirect
 	sigs.k8s.io/kustomize/kyaml v0.20.1 // indirect
 	sigs.k8s.io/randfill v1.0.0 // indirect

clitools/go.sum

@@ -1,7 +1,19 @@
+cel.dev/expr v0.24.0 h1:56OvJKSH3hDGL0ml5uSxZmz3/3Pq4tJ+fb1unVLAFcY=
+cel.dev/expr v0.24.0/go.mod h1:hLPLo1W4QUmuYdA72RBX06QTs6MXw941piREPl3Yfiw=
 github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 h1:L/gRVlceqvL25UVaW/CKtUDjefjrs0SPonmDGUVOYP0=
 github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
+github.com/Masterminds/semver/v3 v3.4.0 h1:Zog+i5UMtVoCU8oKka5P7i9q9HgrJeGzI9SA1Xbatp0=
+github.com/Masterminds/semver/v3 v3.4.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM=
+github.com/antlr4-go/antlr/v4 v4.13.0 h1:lxCg3LAv+EUK6t1i0y1V6/SLeUi0eKEKdhQAlS8TVTI=
+github.com/antlr4-go/antlr/v4 v4.13.0/go.mod h1:pfChB/xh/Unjila75QW7+VU4TSnWnnk9UTnmpPaOR2g=
+github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
+github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
 github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM=
 github.com/blang/semver/v4 v4.0.0/go.mod h1:IbckMUScFkM3pff0VJDNKRiT6TG/YpiHIM2yvyW5YoQ=
+github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
+github.com/cenkalti/backoff/v4 v4.3.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
+github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
+github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
 github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
 github.com/creack/pty v1.1.18 h1:n56/Zwd5o6whRC5PMGretI4IdRLlmBXYNjScPaBgsbY=
@@ -11,12 +23,20 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/emicklei/go-restful/v3 v3.12.2 h1:DhwDP0vY3k8ZzE0RunuJy8GhNpPL6zqLkDf9B/a0/xU=
 github.com/emicklei/go-restful/v3 v3.12.2/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
+github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM=
+github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU=
+github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
+github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
+github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
+github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
 github.com/fxamacker/cbor/v2 v2.9.0 h1:NpKPmjDBgUfBms6tr6JZkTHtfFGcMKsw3eGcmD/sapM=
 github.com/fxamacker/cbor/v2 v2.9.0/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ=
 github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA=
 github.com/go-errors/errors v1.4.2/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og=
-github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
+github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
-github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
+github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
+github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
+github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
 github.com/go-openapi/jsonpointer v0.19.6/go.mod h1:osyAmYz/mB/C3I+WsTTSgw1ONzaLJoLCyoi6/zppojs=
 github.com/go-openapi/jsonpointer v0.21.0 h1:YgdVicSA9vH5RiHs9TZW5oyafXZFc6+2Vc1rr/O9oNQ=
 github.com/go-openapi/jsonpointer v0.21.0/go.mod h1:IUyH9l/+uyhIYQ/PXVA41Rexl+kOkAPDdXEYns6fzUY=
@@ -27,29 +47,31 @@ github.com/go-openapi/swag v0.23.0 h1:vsEVJDUo2hPJ2tu0/Xc+4noaxyEffXNIs3cOULZ+Gr
 github.com/go-openapi/swag v0.23.0/go.mod h1:esZ8ITTYEsH1V2trKHjAN8Ai7xHb8RV+YSZ577vPjgQ=
 github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI=
 github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8=
-github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
+github.com/gobuffalo/flect v1.0.3 h1:xeWBM2nui+qnVvNM4S3foBhCAL2XgPU+a7FdpelbTq4=
-github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
+github.com/gobuffalo/flect v1.0.3/go.mod h1:A5msMlrHtLqh9umBSnvabjsMrCcCpAyzglnDvkbYKHs=
 github.com/google/btree v1.1.3 h1:CVpQJjYgC4VbzxeGVHfvZrv1ctoYCAI8vbl07Fcxlyg=
 github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4=
+github.com/google/cel-go v0.26.0 h1:DPGjXackMpJWH680oGY4lZhYjIameYmR+/6RBdDGmaI=
+github.com/google/cel-go v0.26.0/go.mod h1:A9O8OU9rdvrK5MQyrqfIxo1a0u4g3sF8KB6PUIaryMM=
 github.com/google/gnostic-models v0.7.0 h1:qwTtogB15McXDaNqTZdzPJRHvaVJlAl+HVQnLmJEJxo=
 github.com/google/gnostic-models v0.7.0/go.mod h1:whL5G0m6dmc5cPxKc5bdKdEN3UjI7OUGxBlw57miDrQ=
 github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
 github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
 github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
-github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db h1:097atOisP2aRj7vFgYQBbFN4U4JNXUNYpxael3UzMyo=
+github.com/google/pprof v0.0.0-20250403155104-27863c87afa6 h1:BHT72Gu3keYf3ZEu2J0b1vyeLSOYI8bm5wbJM/8yDe8=
-github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db/go.mod h1:vavhavw2zAxS5dIdcRluK6cSGGPlZynqzFM8NdvU144=
+github.com/google/pprof v0.0.0-20250403155104-27863c87afa6/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA=
 github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
 github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79 h1:+ngKgrYPPJrOjhax5N+uePQ0Fh1Z7PheYoUI/0nzkPA=
 github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.3 h1:5ZPtiqj0JL5oKWmcsq4VMaAW5ukBEgSGXEN89zeH1Jo=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.3/go.mod h1:ndYquD05frm2vACXE1nsccT4oJzjhw2arTS2cpUD1PI=
 github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
 github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
 github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
 github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
 github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
 github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
-github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
-github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
 github.com/klauspost/compress v1.18.5 h1:/h1gH5Ce+VWNLSWqPzOVn6XBO+vJbCNGvjoaGBFW2IE=
 github.com/klauspost/compress v1.18.5/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
 github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
@@ -59,10 +81,17 @@ github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
 github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
 github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
+github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
+github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
 github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de h1:9TO3cAIGXtEhnIaL+V+BEER86oLrvS+kWobKpbJuye0=
 github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de/go.mod h1:zAbeS9B/r2mtpb6U+EI2rYA5OAXxsYw6wTamcNW+zcE=
 github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0=
 github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
+github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
+github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
+github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
+github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
+github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
 github.com/moby/term v0.5.0 h1:xt8Q1nalod/v7BqbG21f8mQPqH+xAaC9C3N3wfWbVP0=
 github.com/moby/term v0.5.0/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y=
 github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
@@ -75,25 +104,38 @@ github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 h1:n6/
 github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00/go.mod h1:Pm3mSP3c5uWn86xMLZ5Sa7JB9GsEZySvHYXCTK4E9q4=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
-github.com/onsi/ginkgo/v2 v2.21.0 h1:7rg/4f3rB88pb5obDgNZrNHrQ4e6WpjonchcpuBRnZM=
+github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE=
-github.com/onsi/ginkgo/v2 v2.21.0/go.mod h1:7Du3c42kxCUegi0IImZ1wUQzMBVecgIHjR1C+NkhLQo=
+github.com/nxadm/tail v1.4.8/go.mod h1:+ncqLTQzXmGhMZNUePPaPqPvBxHAIsmXswZKocGu+AU=
-github.com/onsi/gomega v1.35.1 h1:Cwbd75ZBPxFSuZ6T+rN/WCb/gOc6YgFBXLlZLhC7Ds4=
+github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE=
-github.com/onsi/gomega v1.35.1/go.mod h1:PvZbdDc8J6XJEpDK4HCuRBm8a6Fzp9/DmhC9C7yFlog=
+github.com/onsi/ginkgo v1.16.5/go.mod h1:+E8gABHa3K6zRBolWtd+ROzc/U5bkGt0FwiG042wbpU=
+github.com/onsi/ginkgo/v2 v2.27.2 h1:LzwLj0b89qtIy6SSASkzlNvX6WktqurSHwkk2ipF/Ns=
+github.com/onsi/ginkgo/v2 v2.27.2/go.mod h1:ArE1D/XhNXBXCBkKOLkbsb2c81dQHCRcF5zwn/ykDRo=
+github.com/onsi/gomega v1.38.3 h1:eTX+W6dobAYfFeGC2PV6RwXRu/MyT+cQguijutvkpSM=
+github.com/onsi/gomega v1.38.3/go.mod h1:ZCU1pkQcXDO5Sl9/VVEGlDyp+zm0m1cmeG5TOzLgdh4=
 github.com/peterbourgon/diskv v2.0.1+incompatible h1:UBdAOUP5p4RWqPBg048CAvpKN+vxiaj6gdUUzhl4XmI=
 github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
-github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
-github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
-github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII=
+github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o=
-github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o=
+github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg=
+github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
+github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
+github.com/prometheus/common v0.66.1 h1:h5E0h5/Y8niHc5DlaLlWLArTQI7tMrsfQjHV+d9ZoGs=
+github.com/prometheus/common v0.66.1/go.mod h1:gcaUsgf3KfRSwHY4dIMXLPV0K/Wg1oZ8+SbZk/HH/dA=
+github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg=
+github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is=
+github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
+github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
 github.com/sergi/go-diff v1.2.0 h1:XU+rvMAioB0UC3q1MFrIQy4Vo5/4VsRDQQXHsEya6xQ=
 github.com/sergi/go-diff v1.2.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM=
-github.com/spf13/cobra v1.9.1 h1:CXSaggrXdbHK9CF+8ywj8Amf7PBRmPCOJugH954Nnlo=
+github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
-github.com/spf13/cobra v1.9.1/go.mod h1:nDyEzZ8ogv936Cinf6g1RU9MRY64Ir93oCnqb9wxYW0=
+github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
-github.com/spf13/pflag v1.0.6 h1:jFzHGLGAlb3ruxLB8MhbI6A8+AQX/2eW4qeyNZXNp2o=
+github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
-github.com/spf13/pflag v1.0.6/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
+github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+github.com/stoewer/go-strcase v1.3.0 h1:g0eASXYtp+yvN9fK8sH94oCIk0fau9uV1/ZdJ0AVEzs=
+github.com/stoewer/go-strcase v1.3.0/go.mod h1:fAH5hQ5pehh+j3nZfvwdk2RgEgQjAoM8wodgtPmh1xo=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
 github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
@@ -104,92 +146,115 @@ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
 github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
 github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
-github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
+github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
-github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
+github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
 github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
 github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg=
 github.com/xlab/treeprint v1.2.0 h1:HzHnuAF1plUN2zGlAFHbSQP2qJ0ZAD3XF5XD7OesXRQ=
 github.com/xlab/treeprint v1.2.0/go.mod h1:gj5Gd3gPdKtR1ikdDK6fnFLdmIS0X30kTTuNd/WEJu0=
-github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
+go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=
-github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
+go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
+go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0 h1:F7Jx+6hwnZ41NSFTO5q4LYDtJRXBf2PD0rNBkeB/lus=
+go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0/go.mod h1:UHB22Z8QsdRDrnAtX4PntOl36ajSxcdUMt1sF7Y6E7Q=
+go.opentelemetry.io/otel v1.36.0 h1:UumtzIklRBY6cI/lllNZlALOF5nNIzJVb16APdvgTXg=
+go.opentelemetry.io/otel v1.36.0/go.mod h1:/TcFMXYjyRNh8khOAO9ybYkqaDBb/70aVwkNML4pP8E=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.34.0 h1:OeNbIYk/2C15ckl7glBlOBp5+WlYsOElzTNmiPW/x60=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.34.0/go.mod h1:7Bept48yIeqxP2OZ9/AqIpYS94h2or0aB4FypJTc8ZM=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.34.0 h1:tgJ0uaNS4c98WRNUEx5U3aDlrDOI5Rs+1Vifcw4DJ8U=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.34.0/go.mod h1:U7HYyW0zt/a9x5J1Kjs+r1f/d4ZHnYFclhYY2+YbeoE=
+go.opentelemetry.io/otel/metric v1.36.0 h1:MoWPKVhQvJ+eeXWHFBOPoBOi20jh6Iq2CcCREuTYufE=
+go.opentelemetry.io/otel/metric v1.36.0/go.mod h1:zC7Ks+yeyJt4xig9DEw9kuUFe5C3zLbVjV2PzT6qzbs=
+go.opentelemetry.io/otel/sdk v1.36.0 h1:b6SYIuLRs88ztox4EyrvRti80uXIFy+Sqzoh9kFULbs=
+go.opentelemetry.io/otel/sdk v1.36.0/go.mod h1:+lC+mTgD+MUWfjJubi2vvXWcVxyr9rmlshZni72pXeY=
+go.opentelemetry.io/otel/trace v1.36.0 h1:ahxWNuqZjpdiFAyrIoQ4GIiAIhxAunQR6MUoKrsNd4w=
+go.opentelemetry.io/otel/trace v1.36.0/go.mod h1:gQ+OnDZzrybY4k4seLzPAWNwVBBVlF2szhehOBB/tGA=
+go.opentelemetry.io/proto/otlp v1.5.0 h1:xJvq7gMzB31/d406fB8U5CBdyQGw4P399D1aQWU/3i4=
+go.opentelemetry.io/proto/otlp v1.5.0/go.mod h1:keN8WnHxOy8PG0rQZjJJ5A2ebUoafqWp0eVQ4yIXvJ4=
 go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
 go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
-go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI=
+go.yaml.in/yaml/v2 v2.4.3 h1:6gvOSjQoTB3vt1l+CU+tSyi/HOjfOjRLJ4YwYZGwRO0=
-go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU=
+go.yaml.in/yaml/v2 v2.4.3/go.mod h1:zSxWcmIDjOzPXpjlTTbAsKokqkDNAVtZO0WOMiT90s8=
 go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
 go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
-golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 h1:2dVuKD2vS7b0QIHQbpyTISPd0LeHDbnYEryqj5Q1ug8=
-golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56/go.mod h1:M4RDyNAINzryxdtnbRXRL/OHtkFuWGRjvuhBJpk2IlY=
-golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
+golang.org/x/mod v0.31.0 h1:HaW9xtz0+kOcWKwli0ZXy79Ix+UW/vOfmWI5QVd2tgI=
-golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
+golang.org/x/mod v0.31.0/go.mod h1:43JraMp9cGx1Rx3AqioxrbrhNsLl2l/iNAvuBkrezpg=
-golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
+golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
-golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
-golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/oauth2 v0.30.0 h1:dnDm7JmhM45NNpd8FDDeLhK6FwqbOf4MLCM9zb1BOHI=
-golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/oauth2 v0.30.0/go.mod h1:B++QgG3ZKulg6sRPGD/mqlHQs5rB3Ml9erfeDY7xKlU=
-golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
+golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
-golang.org/x/net v0.38.0 h1:vRMAPTMaeGqVhG5QyLJHqNDwecKTomGeqbnfZyKlBI8=
+golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
-golang.org/x/net v0.38.0/go.mod h1:ivrbrMbzFq5J41QOQh0siUuly180yBYtLp+CKbEaFx8=
-golang.org/x/oauth2 v0.27.0 h1:da9Vo7/tDv5RH/7nZDz1eMGS/q1Vv1N/7FCrBhI9I3M=
-golang.org/x/oauth2 v0.27.0/go.mod h1:onh5ek6nERTohokkhCD/y2cV4Do3fxFHFuAejCkRWT8=
-golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.12.0 h1:MHc5BpPuC30uJk597Ri8TV3CNZcTLu6B6z4lJy+g6Jw=
-golang.org/x/sync v0.12.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
-golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
-golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
+golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
+golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/term v0.30.0 h1:PQ39fJZ+mfadBm0y5WlL4vlM7Sx1Hgf13sMIY2+QS9Y=
+golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
-golang.org/x/term v0.30.0/go.mod h1:NYYFdzHoI5wRh/h5tDMdMqCqPJZEuNqVR5xJLd/n67g=
+golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
-golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/term v0.38.0 h1:PQ5pkm/rLO6HnxFR7N2lJHOZX6Kez5Y1gDSJla6jo7Q=
-golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/term v0.38.0/go.mod h1:bSEAKrOT1W+VSu9TSCMtoGEOUcKxOKgl3LE5QEF/xVg=
-golang.org/x/text v0.23.0 h1:D71I7dUrlY+VX0gQShAThNGHFxZ13dGLBHQLVl1mJlY=
+golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
-golang.org/x/text v0.23.0/go.mod h1:/BLNzu4aZCJ1+kcD0DNRotWKage4q2rGVAg4o22unh4=
+golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
 golang.org/x/time v0.9.0 h1:EsRrnYcQiGH+5FfbgvV4AP7qEZstoyrHB0DzarOQ4ZY=
 golang.org/x/time v0.9.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
-golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.40.0 h1:yLkxfA+Qnul4cs9QA3KnlFu0lVmd8JJfoq+E41uSutA=
-golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.40.0/go.mod h1:Ik/tzLRlbscWpqqMRjyWYDisX8bG13FrdXp3o4Sr9lc=
-golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
+golang.org/x/tools/go/expect v0.1.1-deprecated h1:jpBZDwmgPhXsKZC6WhL20P4b/wmnpsEAGHaNy0n/rJM=
-golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
+golang.org/x/tools/go/expect v0.1.1-deprecated/go.mod h1:eihoPOH+FgIqa3FpoTwguz/bVUSGBlGQU67vpBeOrBY=
-golang.org/x/tools v0.26.0 h1:v/60pFQmzmT9ExmjDv2gGIfi3OqfKoEP6I5+umXlbnQ=
+golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated h1:1h2MnaIAIXISqTFKdENegdpAgUXz6NrPEsbIeWaBRvM=
-golang.org/x/tools v0.26.0/go.mod h1:TPVVj70c7JJ3WCazhD8OdXcZg/og+b9+tH/KxylGwH0=
+golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated/go.mod h1:RVAQXBGNv1ib0J382/DPCRS/BPnsGebyM1Gj5VSDpG8=
-golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+google.golang.org/genproto/googleapis/api v0.0.0-20250303144028-a0af3efb3deb h1:p31xT4yrYrSM/G4Sn2+TNUkVhFCbG9y8itM2S6Th950=
-golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+google.golang.org/genproto/googleapis/api v0.0.0-20250303144028-a0af3efb3deb/go.mod h1:jbe3Bkdp+Dh2IrslsFCklNhweNTBgSYanP1UXhJDhKg=
-golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20250528174236-200df99c418a h1:v2PbRU4K3llS09c7zodFpNePeamkAwG3mPrAery9VeE=
-golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20250528174236-200df99c418a/go.mod h1:qQ0YXyHHx3XkvlzUtpXDkS29lDSafHMZBAZDc03LQ3A=
-google.golang.org/protobuf v1.36.5 h1:tPhr+woSbjfYvY6/GPufUoYizxw1cF/yFoxJ2fmpwlM=
+google.golang.org/grpc v1.72.2 h1:TdbGzwb82ty4OusHWepvFWGLgIbNo1/SUynEN0ssqv8=
-google.golang.org/protobuf v1.36.5/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
+google.golang.org/grpc v1.72.2/go.mod h1:wH5Aktxcg25y1I3w7H69nHfXdOG3UiadoBtjh3izSDM=
+google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc=
+google.golang.org/protobuf v1.36.8/go.mod h1:fuxRtAxBytpl4zzqUh6/eyUujkJdNiuEkXntxiD/uRU=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
-gopkg.in/evanphx/json-patch.v4 v4.12.0 h1:n6jtcsulIzXPJaxegRbvFNNrZDjbij7ny3gmSPG+6V4=
+gopkg.in/evanphx/json-patch.v4 v4.13.0 h1:czT3CmqEaQ1aanPc5SdlgQrrEIb8w/wwCvWWnfEbYzo=
-gopkg.in/evanphx/json-patch.v4 v4.12.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWMG4EsWvDvM72M=
+gopkg.in/evanphx/json-patch.v4 v4.13.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWMG4EsWvDvM72M=
 gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
 gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
+gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
+gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
+gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
+gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
-k8s.io/api v0.34.0 h1:L+JtP2wDbEYPUeNGbeSa/5GwFtIA662EmT2YSLOkAVE=
+k8s.io/api v0.35.0 h1:iBAU5LTyBI9vw3L5glmat1njFK34srdLmktWwLTprlY=
-k8s.io/api v0.34.0/go.mod h1:YzgkIzOOlhl9uwWCZNqpw6RJy9L2FK4dlJeayUoydug=
+k8s.io/api v0.35.0/go.mod h1:AQ0SNTzm4ZAczM03QH42c7l3bih1TbAXYo0DkF8ktnA=
-k8s.io/apiextensions-apiserver v0.34.0 h1:B3hiB32jV7BcyKcMU5fDaDxk882YrJ1KU+ZSkA9Qxoc=
+k8s.io/apiextensions-apiserver v0.35.0 h1:3xHk2rTOdWXXJM+RDQZJvdx0yEOgC0FgQ1PlJatA5T4=
-k8s.io/apiextensions-apiserver v0.34.0/go.mod h1:hLI4GxE1BDBy9adJKxUxCEHBGZtGfIg98Q+JmTD7+g0=
+k8s.io/apiextensions-apiserver v0.35.0/go.mod h1:E1Ahk9SADaLQ4qtzYFkwUqusXTcaV2uw3l14aqpL2LU=
-k8s.io/apimachinery v0.34.0 h1:eR1WO5fo0HyoQZt1wdISpFDffnWOvFLOOeJ7MgIv4z0=
+k8s.io/apimachinery v0.35.0 h1:Z2L3IHvPVv/MJ7xRxHEtk6GoJElaAqDCCU0S6ncYok8=
-k8s.io/apimachinery v0.34.0/go.mod h1:/GwIlEcWuTX9zKIg2mbw0LRFIsXwrfoVxn+ef0X13lw=
+k8s.io/apimachinery v0.35.0/go.mod h1:jQCgFZFR1F4Ik7hvr2g84RTJSZegBc8yHgFWKn//hns=
+k8s.io/apiserver v0.35.0 h1:CUGo5o+7hW9GcAEF3x3usT3fX4f9r8xmgQeCBDaOgX4=
+k8s.io/apiserver v0.35.0/go.mod h1:QUy1U4+PrzbJaM3XGu2tQ7U9A4udRRo5cyxkFX0GEds=
 k8s.io/cli-runtime v0.34.0 h1:N2/rUlJg6TMEBgtQ3SDRJwa8XyKUizwjlOknT1mB2Cw=
 k8s.io/cli-runtime v0.34.0/go.mod h1:t/skRecS73Piv+J+FmWIQA2N2/rDjdYSQzEE67LUUs8=
-k8s.io/client-go v0.34.0 h1:YoWv5r7bsBfb0Hs2jh8SOvFbKzzxyNo0nSb0zC19KZo=
+k8s.io/client-go v0.35.0 h1:IAW0ifFbfQQwQmga0UdoH0yvdqrbwMdq9vIFEhRpxBE=
-k8s.io/client-go v0.34.0/go.mod h1:ozgMnEKXkRjeMvBZdV1AijMHLTh3pbACPvK7zFR+QQY=
+k8s.io/client-go v0.35.0/go.mod h1:q2E5AAyqcbeLGPdoRB+Nxe3KYTfPce1Dnu1myQdqz9o=
+k8s.io/code-generator v0.35.0 h1:TvrtfKYZTm9oDF2z+veFKSCcgZE3Igv0svY+ehCmjHQ=
+k8s.io/code-generator v0.35.0/go.mod h1:iS1gvVf3c/T71N5DOGYO+Gt3PdJ6B9LYSvIyQ4FHzgc=
+k8s.io/component-base v0.35.0 h1:+yBrOhzri2S1BVqyVSvcM3PtPyx5GUxCK2tinZz1G94=
+k8s.io/component-base v0.35.0/go.mod h1:85SCX4UCa6SCFt6p3IKAPej7jSnF3L8EbfSyMZayJR0=
+k8s.io/gengo/v2 v2.0.0-20250922181213-ec3ebc5fd46b h1:gMplByicHV/TJBizHd9aVEsTYoJBnnUAT5MHlTkbjhQ=
+k8s.io/gengo/v2 v2.0.0-20250922181213-ec3ebc5fd46b/go.mod h1:CgujABENc3KuTrcsdpGmrrASjtQsWCT7R99mEV4U/fM=
 k8s.io/klog/v2 v2.130.1 h1:n9Xl7H1Xvksem4KFG4PYbdQCQxqc/tTUyrgXaOhHSzk=
 k8s.io/klog/v2 v2.130.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE=
-k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b h1:MloQ9/bdJyIu9lb1PzujOPolHyvO06MXG5TUIj2mNAA=
+k8s.io/kube-openapi v0.0.0-20250910181357-589584f1c912 h1:Y3gxNAuB0OBLImH611+UDZcmKS3g6CthxToOb37KgwE=
-k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b/go.mod h1:UZ2yyWbFTpuhSbFhv24aGNOdoRdJZgsIObGBUaYVsts=
+k8s.io/kube-openapi v0.0.0-20250910181357-589584f1c912/go.mod h1:kdmbQkyfwUagLfXIad1y2TdrjPFWp2Q89B3qkRwf/pQ=
-k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 h1:hwvWFiBzdWw1FhfY1FooPn3kzWuJ8tmbZBHi4zVsl1Y=
+k8s.io/utils v0.0.0-20251002143259-bc988d571ff4 h1:SjGebBtkBqHFOli+05xYbK8YF1Dzkbzn+gDM4X9T4Ck=
-k8s.io/utils v0.0.0-20250604170112-4c0f3b243397/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
+k8s.io/utils v0.0.0-20251002143259-bc988d571ff4/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
-sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 h1:gBQPwqORJ8d8/YNZWEjoZs7npUVDpVXUUOFfW6CgAqE=
+sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.31.2 h1:jpcvIRr3GLoUoEKRkHKSmGjxb6lWwrBlJsXc+eUYQHM=
-sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8/go.mod h1:mdzfpAEoE6DHQEN0uh9ZbOCuHbLK5wOm7dK4ctXE9Tg=
+sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.31.2/go.mod h1:Ve9uj1L+deCXFrPOk1LpFXqTg7LCFzFso6PA48q/XZw=
+sigs.k8s.io/controller-tools v0.20.1 h1:gkfMt9YodI0K85oT8rVi80NTXO/kDmabKR5Ajn5GYxs=
+sigs.k8s.io/controller-tools v0.20.1/go.mod h1:b4qPmjGU3iZwqn34alUU5tILhNa9+VXK+J3QV0fT/uU=
+sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 h1:IpInykpT6ceI+QxKBbEflcR5EXP7sU1kvOlxwZh5txg=
+sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730/go.mod h1:mdzfpAEoE6DHQEN0uh9ZbOCuHbLK5wOm7dK4ctXE9Tg=
 sigs.k8s.io/kustomize/api v0.20.1 h1:iWP1Ydh3/lmldBnH/S5RXgT98vWYMaTUL1ADcr+Sv7I=
 sigs.k8s.io/kustomize/api v0.20.1/go.mod h1:t6hUFxO+Ph0VxIk1sKp1WS0dOjbPCtLJ4p8aADLwqjM=
 sigs.k8s.io/kustomize/kyaml v0.20.1 h1:PCMnA2mrVbRP3NIB6v9kYCAc38uvFLVs8j/CD567A78=

clitools/hack/boilerplate.go.txt

@@ -0,0 +1 @@
+/* MIT License */

clitools/hack/tool.go

@@ -0,0 +1,9 @@
+//go:build tools
+// +build tools
+
+package tools
+
+import (
+	_ "k8s.io/code-generator"
+	_ "sigs.k8s.io/controller-tools/cmd/controller-gen"
+)

clitools/hack/update-codegen.sh

@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+MODULE_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+echo "MODULE ROOT: ${MODULE_ROOT}"
+
+cd "${MODULE_ROOT}"
+mkdir -p \
+	"${MODULE_ROOT}/pkg/generated/clientset" \
+	"${MODULE_ROOT}/pkg/generated/listers" \
+	"${MODULE_ROOT}/pkg/generated/informers"
+
+controller-gen \
+  object:headerFile=hack/boilerplate.go.txt \
+  paths=./pkg/apis/...
+
+MODULE="$(go list -m -f '{{.Path}}')"
+echo "MODULE: ${MODULE}"
+
+CODEGEN_PKG="$(go list -f '{{.Dir}}' -m k8s.io/code-generator)"
+echo "CODEGEN PKG: ${CODEGEN_PKG}"
+
+source "${CODEGEN_PKG}/kube_codegen.sh"
+
+mkdir -p "${MODULE_ROOT}/pkg/generated"
+
+kube::codegen::gen_helpers \
+  --boilerplate "${MODULE_ROOT}/hack/boilerplate.go.txt" \
+  "${MODULE_ROOT}/pkg/apis"
+
+kube::codegen::gen_client \
+  --with-watch \
+  --output-dir "${MODULE_ROOT}/pkg/generated" \
+  --output-pkg "${MODULE}/pkg/generated" \
+  --boilerplate "${MODULE_ROOT}/hack/boilerplate.go.txt" \
+  "${MODULE_ROOT}/pkg/apis"

clitools/makefile

@@ -1,4 +1,6 @@
 include ../build.env
+-include ../build.env.work
+export
 
 BUILD_PLATFORM ?= linux/amd64
 
@@ -12,29 +14,28 @@ KUBE_VERSION ?= v1.33.3
 GIT_REV := $(shell git rev-parse HEAD)
 
 PACKAGES_DIR    := packages
-BIN_DIR      := bin
 OUT_DIR         := out
+UBOOT_TOOLS_OUT := $(OUT_DIR)/uboot-tools
 
 UBOOT_TAR      := $(PACKAGES_DIR)/uboot-$(UBOOT_VERSION).tar.gz
 BUILDINFO_FILE := pkg/buildinfo/buildinfo_gen.go
-CRD_PATHS      := ./pkg/apis/...
+ASSETS_PATH      := ./pkg/assets
-
 
 BUILDX_BUILDER      := container-builder
 LOCAL_REGISTRY      := registry
 LOCAL_REGISTRY_PORT := 5000
 
-CTL_BUILD_BASE_IMAGE := localhost:5000/monok8s/ctl-build-base:$(VERSION)
+CTL_BUILD_BASE_REPO  := localhost:5000/monok8s/ctl-build-base
-CTL_BINARY           := ctl-linux-aarch64-$(VERSION)
+CTL_IMAGE_REPO       := localhost:5000/monok8s/node-control
+
+CTL_BUILD_BASE_IMAGE := $(CTL_BUILD_BASE_REPO):$(VERSION)
+CTL_IMAGE            := $(CTL_IMAGE_REPO):$(VERSION)
 
 DOWNLOAD_PACKAGES_STAMP := $(PACKAGES_DIR)/.download-packages.stamp
 
 $(PACKAGES_DIR):
 	mkdir -p $@
 
-$(BIN_DIR):
-	mkdir -p $@
-
 $(OUT_DIR):
 	mkdir -p $@
 
@@ -88,11 +89,14 @@ $(DOWNLOAD_PACKAGES_STAMP): docker/download-packages.Dockerfile makefile | $(PAC
 	@touch $@
 
 uboot-tools: $(DOWNLOAD_PACKAGES_STAMP)
-	docker buildx build --platform linux/arm64 \
+	rm -rf "$(UBOOT_TOOLS_OUT)"
+	mkdir -p "$(UBOOT_TOOLS_OUT)"
+	docker buildx build \
+		--platform linux/amd64,linux/arm64 \
 		-f docker/uboot-tools.Dockerfile \
 		--build-arg UBOOT_VERSION=$(UBOOT_VERSION) \
 		--build-arg UBOOT_TAR=$(UBOOT_TAR) \
-		--output type=local,dest=./$(OUT_DIR) .
+		--output type=local,dest=./$(UBOOT_TOOLS_OUT),platform-split=true .
 
 ctl-build-base: ensure-buildx ensure-registry
 	docker buildx build \
@@ -101,16 +105,6 @@ ctl-build-base: ensure-buildx ensure-registry
 		-t $(CTL_BUILD_BASE_IMAGE) \
 		--output type=image,push=true,registry.insecure=true .
 
-build-bin: .buildinfo ctl-build-base | $(BIN_DIR)
-	docker buildx build \
-		--platform $(BUILD_PLATFORM) \
-		-f docker/ctl-builder.Dockerfile \
-		--build-arg BASE_IMAGE=$(CTL_BUILD_BASE_IMAGE) \
-		--build-arg VERSION=$(VERSION) \
-		--build-arg TARGETOS=linux \
-		--build-arg TARGETARCH=arm64 \
-		--output type=local,dest=./$(BIN_DIR) .
-
 build-crds: ctl-build-base | $(OUT_DIR)
 	mkdir -p "$(OUT_DIR)/crds"
 	docker buildx build \
@@ -118,35 +112,47 @@ build-crds: ctl-build-base | $(OUT_DIR)
 		-f docker/crdgen.Dockerfile \
 		--build-arg BASE_IMAGE=$(CTL_BUILD_BASE_IMAGE) \
 		--output type=local,dest=./$(OUT_DIR)/crds .
+	rm -rf "$(ASSETS_PATH)/crds"
+	mkdir -p "$(ASSETS_PATH)/crds"
+	cp -R "$(OUT_DIR)/crds/." "$(ASSETS_PATH)/crds/"
 
-build-agent: build uboot-tools
+build-agent: .buildinfo build-crds uboot-tools
+	docker buildx build \
+		--platform linux/amd64,linux/arm64 \
+		-f docker/ctl-agent.Dockerfile \
+		--build-arg BASE_IMAGE=$(CTL_BUILD_BASE_IMAGE) \
+		--build-arg VERSION=$(VERSION) \
+		-t $(CTL_IMAGE) \
+		--output type=image,push=true,registry.insecure=true .
+
+build-local: .buildinfo build-crds uboot-tools
 	docker buildx build \
 		--platform linux/arm64 \
 		-f docker/ctl-agent.Dockerfile \
+		--build-arg BASE_IMAGE=$(CTL_BUILD_BASE_IMAGE) \
 		--build-arg VERSION=$(VERSION) \
 		--load \
-		-t localhost/monok8s/control-agent:$(VERSION) .
+		-t localhost/monok8s/node-control:$(VERSION) .
 
-build-local: .buildinfo | $(BIN_DIR)
+push-agent: .buildinfo build-crds uboot-tools
+	test -n "$(IMAGE_REPOSITORY)"
 	docker buildx build \
-		-f docker/ctl-builder-local.Dockerfile \
+		--platform linux/amd64,linux/arm64 \
+		-f docker/ctl-agent.Dockerfile \
+		--build-arg BASE_IMAGE=$(CTL_BUILD_BASE_IMAGE) \
 		--build-arg VERSION=$(VERSION) \
-		--build-arg KUBE_VERSION=$(KUBE_VERSION) \
+		-t $(IMAGE_REPOSITORY)/node-control:$(VERSION) \
-		--build-arg GIT_REV=$(GIT_REV) \
+		--push .
-		--output type=local,dest=./$(BIN_DIR) .
 
 run-agent:
 	docker run --rm \
 		-v "$$(pwd)/out:/work/out" \
-		localhost/monok8s/control-agent:$(VERSION) \
+		$(CTL_IMAGE) \
 		agent --env-file /work/out/cluster.env
 
-build: build-bin build-crds
-
 clean:
-	-docker image rm localhost/monok8s/control-agent:$(VERSION) >/dev/null 2>&1 || true
+	-docker image rm localhost/monok8s/node-control:$(VERSION) >/dev/null 2>&1 || true
 	rm -rf \
-		$(BIN_DIR) \
 		$(OUT_DIR)/crds \
 		$(BUILDINFO_FILE)
 
@@ -157,8 +163,7 @@ dockerclean:
 	@echo "Removing tagged images..."
 	- docker rmi \
 		localhost/monok8s/ctl-build-base:$(VERSION) \
-		localhost/monok8s/control-agent:$(VERSION) \
+		localhost/monok8s/node-control:$(VERSION) \
-		localhost/monok8s/ctl-builder:$(VERSION) \
 		localhost/monok8s/crdgen:$(VERSION) \
 		2>/dev/null || true
 
@@ -169,10 +174,10 @@ dockerclean:
 pkgclean:
 	rm -rf $(PACKAGES_DIR)
 
-all: build build-agent build-local
+all: build-agent build-local
 
 .PHONY: \
 	all clean dockerclean \
 	.buildinfo ensure-buildx ensure-registry \
-	build build-bin build-crds build-local build-agent \
+	build-crds build-local build-agent build-agent-local push-agent \
-	uboot-tools run-agent
+	uboot-tools run-agent run-agent-local

clitools/pkg/apis/monok8s/v1alpha1/groupversion_info.go

@@ -14,10 +14,12 @@ var (
 	APIVersion = "monok8s.io/v1alpha1"
 
 	AltPartDeviceLink = "/dev/mksaltpart"
-	Annotation        = "monok8s.io/annotation"
 	BootStateFile     = "/run/monok8s/boot-state.env"
 	CatalogURL        = "https://example.com/monok8s.io/v1alpha1/catalog.yaml"
-	ControlAgentKey   = "monok8s.io/control-agent"
+	NodeControlKey    = "monok8s.io/node-control"
+	NodeControlName   = "node-control"
+	ControllerName    = "node-controller"
+	NodeAgentName     = "node-agent"
 	EnvConfigDir      = "/opt/monok8s/config"
 	Label             = "monok8s.io/label"
 	MonoKSConfigCRD   = "monoksconfigs.monok8s.io"
@@ -25,18 +27,29 @@ var (
 )
 
 var (
-	GroupVersion  = schema.GroupVersion{Group: Group, Version: Version}
+	SchemeGroupVersion = schema.GroupVersion{Group: Group, Version: Version}
 	SchemeBuilder      = runtime.NewSchemeBuilder(addKnownTypes)
 	AddToScheme        = SchemeBuilder.AddToScheme
 )
 
 func addKnownTypes(scheme *runtime.Scheme) error {
-	scheme.AddKnownTypes(GroupVersion,
+	scheme.AddKnownTypes(SchemeGroupVersion,
 		&MonoKSConfig{},
 		&MonoKSConfigList{},
 		&OSUpgrade{},
 		&OSUpgradeList{},
+		&OSUpgradeProgress{},
+		&OSUpgradeProgressList{},
 	)
-	metav1.AddToGroupVersion(scheme, GroupVersion)
+	metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
 	return nil
 }
+
+func NodeAgentLabels() map[string]string {
+	return map[string]string{
+		"app.kubernetes.io/name":       NodeAgentName,
+		"app.kubernetes.io/component":  "agent",
+		"app.kubernetes.io/part-of":    "monok8s",
+		"app.kubernetes.io/managed-by": NodeControlName,
+	}
+}

clitools/pkg/apis/monok8s/v1alpha1/monoksconfig.go

@@ -25,7 +25,7 @@ type MonoKSConfigSpec struct {
 	ClusterDomain                 string            `json:"clusterDomain,omitempty" yaml:"clusterDomain,omitempty"`
 	ClusterRole                   string            `json:"clusterRole,omitempty" yaml:"clusterRole,omitempty"`
 	InitControlPlane              bool              `json:"initControlPlane,omitempty" yaml:"initControlPlane,omitempty"`
-	EnableControlAgent            bool              `json:"enableControlAgent,omitempty" yaml:"enableControlAgent,omitempty"`
+	EnableNodeControl            bool              `json:"enableNodeControl,omitempty" yaml:"enableNodeControl,omitempty"`
 	PodSubnet                     string            `json:"podSubnet,omitempty" yaml:"podSubnet,omitempty"`
 	ServiceSubnet                 string            `json:"serviceSubnet,omitempty" yaml:"serviceSubnet,omitempty"`
 	APIServerAdvertiseAddress     string            `json:"apiServerAdvertiseAddress,omitempty" yaml:"apiServerAdvertiseAddress,omitempty"`
@@ -40,7 +40,6 @@ type MonoKSConfigSpec struct {
 	KubeProxyNodePortAddresses    []string          `json:"kubeProxyNodePortAddresses,omitempty" yaml:"kubeProxyNodePortAddresses,omitempty"`
 	SubjectAltNames               []string          `json:"subjectAltNames,omitempty" yaml:"subjectAltNames,omitempty"`
 	NodeLabels                    map[string]string `json:"nodeLabels,omitempty" yaml:"nodeLabels,omitempty"`
-	NodeAnnotations               map[string]string `json:"nodeAnnotations,omitempty" yaml:"nodeAnnotations,omitempty"`
 	Network                       NetworkSpec       `json:"network,omitempty" yaml:"network,omitempty"`
 }
 

clitools/pkg/apis/monok8s/v1alpha1/osupgrade.go

@@ -2,7 +2,6 @@ package v1alpha1
 
 import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/runtime"
 )
 
 type OSUpgradePhase string
@@ -10,8 +9,6 @@ type OSUpgradePhase string
 const (
 	OSUpgradePhasePending  OSUpgradePhase = "Pending"
 	OSUpgradePhaseAccepted OSUpgradePhase = "Accepted"
-	OSUpgradePhaseRollingOut OSUpgradePhase = "RollingOut"
-	OSUpgradePhaseCompleted  OSUpgradePhase = "Completed"
 	OSUpgradePhaseRejected OSUpgradePhase = "Rejected"
 )
 
@@ -21,27 +18,35 @@ const (
 	OSUpgradeProgressPhasePending     OSUpgradeProgressPhase = "pending"
 	OSUpgradeProgressPhaseDownloading OSUpgradeProgressPhase = "downloading"
 	OSUpgradeProgressPhaseWriting     OSUpgradeProgressPhase = "writing"
-	OSUpgradeProgressPhaseRebooting   OSUpgradeProgressPhase = "rebooting"
 	OSUpgradeProgressPhaseVerifying   OSUpgradeProgressPhase = "verifying"
 	OSUpgradeProgressPhaseCompleted   OSUpgradeProgressPhase = "completed"
 	OSUpgradeProgressPhaseFailed      OSUpgradeProgressPhase = "failed"
 	OSUpgradeProgressPhaseRejected    OSUpgradeProgressPhase = "rejected"
+
+	// Rebooting is the point-of-no-return phase.
+	//
+	// Once a node reaches Rebooting, the agent may have already changed the boot
+	// environment and requested a reboot. The controller must not supersede,
+	// retry, retarget, or otherwise mutate this progress object until the node
+	// comes back and the agent reports Completed or Failed.
+	OSUpgradeProgressPhaseRebooting OSUpgradeProgressPhase = "rebooting"
 )
 
+// +genclient
 // +kubebuilder:object:root=true
 // +kubebuilder:subresource:status
 // +kubebuilder:resource:scope=Namespaced,shortName=osu
 // +kubebuilder:printcolumn:name="Desired",type=string,JSONPath=`.spec.desiredVersion`
 // +kubebuilder:printcolumn:name="Resolved",type=string,JSONPath=`.status.resolvedVersion`
 // +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
-// +kubebuilder:printcolumn:name="Targets",type=integer,JSONPath=`.status.summary.targetedNodes`
-// +kubebuilder:printcolumn:name="OK",type=integer,JSONPath=`.status.summary.succeededNodes`
-// +kubebuilder:printcolumn:name="Fail",type=integer,JSONPath=`.status.summary.failedNodes`
 type OSUpgrade struct {
 	metav1.TypeMeta   `json:",inline" yaml:",inline"`
 	metav1.ObjectMeta `json:"metadata,omitempty" yaml:"metadata,omitempty"`
 
+	// Specification of the desired behavior of the OSUpgrade.
 	Spec OSUpgradeSpec `json:"spec,omitempty" yaml:"spec,omitempty"`
+
+	// Most recently observed status of the OSUpgrade.
 	Status *OSUpgradeStatus `json:"status,omitempty" yaml:"status,omitempty"`
 }
 
@@ -52,68 +57,38 @@ type OSUpgradeList struct {
 	Items           []OSUpgrade `json:"items" yaml:"items"`
 }
 
-// OSUpgradeSpec defines the desired state of an OSUpgrade.
 type OSUpgradeSpec struct {
-	// DesiredVersion is the requested target version.
-	//
-	// It may be either:
-	// - "stable" to use the catalog's current stable version
-	// - an explicit version such as "v1.35.3"
-	//
-	// The resolved target version is reported in status.
 	// +kubebuilder:validation:MinLength=1
 	DesiredVersion string `json:"desiredVersion,omitempty" yaml:"desiredVersion,omitempty"`
 
-	// FlashProfile controls how aggressively the image is written to disk.
-	//
-	// Supported values are:
-	// - "fast": prioritize speed
-	// - "balanced": default tradeoff between speed and system impact
-	// - "safe": minimize I/O pressure on the node
-	//
-	// If unset, the controller should use a default profile.
 	// +kubebuilder:validation:Enum=fast;balanced;safe
 	// +kubebuilder:default=balanced
-	FlashProfile string `json:"flashProfile,omitempty"`
+	// Profiles (TODO)
+	//    safe     - api-server can be responsive most of the time
+	//    balanced - api-server can sometimes be unresponsive
+	//    fast     - disable throttling. Good for worker node.
+	FlashProfile string `json:"flashProfile,omitempty" yaml:"flashProfile,omitempty"`
 
-	// Catalog specifies how available versions and images are resolved.
+	Catalog      *VersionCatalogSource `json:"catalog,omitempty" yaml:"catalog,omitempty"`
-	//
-	// This may point to a remote catalog, inline catalog data, or another source,
-	// depending on the VersionCatalogSource definition.
-	Catalog *VersionCatalogSource `json:"catalog,omitempty"`
-
-	// NodeSelector limits the upgrade to nodes whose labels match this selector.
-	//
-	// If unset, the upgrade applies to all eligible nodes.
 	NodeSelector *metav1.LabelSelector `json:"nodeSelector,omitempty" yaml:"nodeSelector,omitempty"`
 }
 
 type VersionCatalogSource struct {
-	URL       string `json:"url,omitempty"`
+	URL       string `json:"url,omitempty" yaml:"url,omitempty"`
-	Inline    string `json:"inline,omitempty"`
+	Inline    string `json:"inline,omitempty" yaml:"inline,omitempty"`
-	ConfigMap string `json:"configMapRef,omitempty"`
+	ConfigMap string `json:"configMapRef,omitempty" yaml:"configMapRef,omitempty"`
 }
 
 type OSUpgradeStatus struct {
 	Phase              OSUpgradePhase     `json:"phase,omitempty" yaml:"phase,omitempty"`
 	ResolvedVersion    string             `json:"resolvedVersion,omitempty" yaml:"resolvedVersion,omitempty"`
 	ObservedGeneration int64              `json:"observedGeneration,omitempty" yaml:"observedGeneration,omitempty"`
-	Summary            OSUpgradeSummary   `json:"summary,omitempty" yaml:"summary,omitempty"`
 	Conditions         []metav1.Condition `json:"conditions,omitempty" yaml:"conditions,omitempty"`
-
-	// Optional, useful when rejected.
 	Reason             string             `json:"reason,omitempty" yaml:"reason,omitempty"`
 	Message            string             `json:"message,omitempty" yaml:"message,omitempty"`
 }
 
-type OSUpgradeSummary struct {
+// +genclient
-	TargetedNodes  int32 `json:"targetedNodes,omitempty" yaml:"targetedNodes,omitempty"`
-	PendingNodes   int32 `json:"pendingNodes,omitempty" yaml:"pendingNodes,omitempty"`
-	RunningNodes   int32 `json:"runningNodes,omitempty" yaml:"runningNodes,omitempty"`
-	SucceededNodes int32 `json:"succeededNodes,omitempty" yaml:"succeededNodes,omitempty"`
-	FailedNodes    int32 `json:"failedNodes,omitempty" yaml:"failedNodes,omitempty"`
-}
-
 // +kubebuilder:object:root=true
 // +kubebuilder:subresource:status
 // +kubebuilder:resource:scope=Namespaced,shortName=osup
@@ -121,12 +96,15 @@ type OSUpgradeSummary struct {
 // +kubebuilder:printcolumn:name="Source",type=string,JSONPath=`.spec.sourceRef.name`
 // +kubebuilder:printcolumn:name="Current",type=string,JSONPath=`.status.currentVersion`
 // +kubebuilder:printcolumn:name="Target",type=string,JSONPath=`.status.targetVersion`
-// +kubebuilder:printcolumn:name="Status",type=string,JSONPath=`.status.phase`
+// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
 type OSUpgradeProgress struct {
 	metav1.TypeMeta   `json:",inline" yaml:",inline"`
 	metav1.ObjectMeta `json:"metadata,omitempty" yaml:"metadata,omitempty"`
 
+	// Specification of the desired behavior of the OSUpgradeProgress.
 	Spec OSUpgradeProgressSpec `json:"spec,omitempty" yaml:"spec,omitempty"`
+
+	// Most recently observed status of the OSUpgradeProgress.
 	Status *OSUpgradeProgressStatus `json:"status,omitempty" yaml:"status,omitempty"`
 }
 
@@ -139,18 +117,24 @@ type OSUpgradeProgressList struct {
 
 type OSUpgradeProgressSpec struct {
 	SourceRef OSUpgradeSourceRef `json:"sourceRef,omitempty" yaml:"sourceRef,omitempty"`
+
+	// RetryNonce triggers a retry when its value changes.
+	// Users can update this field (for example, set it to the current time)
+	// to request a retry of a failed OS upgrade.
+	RetryNonce string `json:"retryNonce,omitempty" yaml:"retryNonce,omitempty"`
+
 	NodeName string `json:"nodeName,omitempty" yaml:"nodeName,omitempty"`
 }
 
 type OSUpgradeSourceRef struct {
 	Name      string `json:"name,omitempty" yaml:"name,omitempty"`
+	Namespace string `json:"namespace,omitempty" yaml:"namespace,omitempty"`
 }
 
 type OSUpgradeProgressStatus struct {
 	CurrentVersion    string                 `json:"currentVersion,omitempty" yaml:"currentVersion,omitempty"`
 	TargetVersion     string                 `json:"targetVersion,omitempty" yaml:"targetVersion,omitempty"`
 	Phase             OSUpgradeProgressPhase `json:"phase,omitempty" yaml:"phase,omitempty"`
-
 	StartedAt         *metav1.Time           `json:"startedAt,omitempty" yaml:"startedAt,omitempty"`
 	CompletedAt       *metav1.Time           `json:"completedAt,omitempty" yaml:"completedAt,omitempty"`
 	LastUpdatedAt     *metav1.Time           `json:"lastUpdatedAt,omitempty" yaml:"lastUpdatedAt,omitempty"`
@@ -158,41 +142,30 @@ type OSUpgradeProgressStatus struct {
 	InactivePartition string                 `json:"inactivePartition,omitempty" yaml:"inactivePartition,omitempty"`
 	FailureReason     string                 `json:"failureReason,omitempty" yaml:"failureReason,omitempty"`
 	Message           string                 `json:"message,omitempty" yaml:"message,omitempty"`
+	PlannedPath       []string               `json:"plannedPath,omitempty" yaml:"plannedPath,omitempty"`
+	CurrentStep       int32                  `json:"currentStep,omitempty" yaml:"currentStep,omitempty"`
+	CurrentFrom       string                 `json:"currentFrom,omitempty" yaml:"currentFrom,omitempty"`
+	CurrentTo         string                 `json:"currentTo,omitempty" yaml:"currentTo,omitempty"`
 
-	PlannedPath []string `json:"plannedPath,omitempty"`
+	// ObservedRetryNonce records the last retryNonce value the agent accepted.
-	CurrentStep int32    `json:"currentStep,omitempty"`
+	// When spec.retryNonce is changed by the user and differs from this value,
-	CurrentFrom string   `json:"currentFrom,omitempty"`
+	// the agent may retry a failed upgrade.
-	CurrentTo   string   `json:"currentTo,omitempty"`
+	// +optional
+	ObservedRetryNonce string `json:"observedRetryNonce,omitempty"`
 }
 
-func (in *OSUpgrade) DeepCopyObject() runtime.Object {
+func (osu OSUpgrade) StatusPhase() string {
-	if in == nil {
+	phase := ""
-		return nil
+	if osu.Status != nil {
+		phase = string(osu.Status.Phase)
 	}
-	out := *in
+	return phase
-	return &out
 }
 
-func (in *OSUpgradeList) DeepCopyObject() runtime.Object {
+func (osup OSUpgradeProgress) StatusPhase() string {
-	if in == nil {
+	phase := ""
-		return nil
+	if osup.Status != nil {
+		phase = string(osup.Status.Phase)
 	}
-	out := *in
+	return phase
-	return &out
-}
-
-func (in *OSUpgradeProgress) DeepCopyObject() runtime.Object {
-	if in == nil {
-		return nil
-	}
-	out := *in
-	return &out
-}
-
-func (in *OSUpgradeProgressList) DeepCopyObject() runtime.Object {
-	if in == nil {
-		return nil
-	}
-	out := *in
-	return &out
 }

clitools/pkg/apis/monok8s/v1alpha1/zz_generated.deepcopy.go

@@ -0,0 +1,395 @@
+//go:build !ignore_autogenerated
+
+/* MIT License */
+
+// Code generated by controller-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	"k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+)
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *MonoKSConfig) DeepCopyInto(out *MonoKSConfig) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	in.Spec.DeepCopyInto(&out.Spec)
+	if in.Status != nil {
+		in, out := &in.Status, &out.Status
+		*out = new(MonoKSConfigStatus)
+		(*in).DeepCopyInto(*out)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MonoKSConfig.
+func (in *MonoKSConfig) DeepCopy() *MonoKSConfig {
+	if in == nil {
+		return nil
+	}
+	out := new(MonoKSConfig)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *MonoKSConfigList) DeepCopyInto(out *MonoKSConfigList) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ListMeta.DeepCopyInto(&out.ListMeta)
+	if in.Items != nil {
+		in, out := &in.Items, &out.Items
+		*out = make([]MonoKSConfig, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MonoKSConfigList.
+func (in *MonoKSConfigList) DeepCopy() *MonoKSConfigList {
+	if in == nil {
+		return nil
+	}
+	out := new(MonoKSConfigList)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *MonoKSConfigSpec) DeepCopyInto(out *MonoKSConfigSpec) {
+	*out = *in
+	if in.KubeProxyNodePortAddresses != nil {
+		in, out := &in.KubeProxyNodePortAddresses, &out.KubeProxyNodePortAddresses
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+	if in.SubjectAltNames != nil {
+		in, out := &in.SubjectAltNames, &out.SubjectAltNames
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+	if in.NodeLabels != nil {
+		in, out := &in.NodeLabels, &out.NodeLabels
+		*out = make(map[string]string, len(*in))
+		for key, val := range *in {
+			(*out)[key] = val
+		}
+	}
+	in.Network.DeepCopyInto(&out.Network)
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MonoKSConfigSpec.
+func (in *MonoKSConfigSpec) DeepCopy() *MonoKSConfigSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(MonoKSConfigSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *MonoKSConfigStatus) DeepCopyInto(out *MonoKSConfigStatus) {
+	*out = *in
+	if in.Conditions != nil {
+		in, out := &in.Conditions, &out.Conditions
+		*out = make([]v1.Condition, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+	if in.AppliedSteps != nil {
+		in, out := &in.AppliedSteps, &out.AppliedSteps
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MonoKSConfigStatus.
+func (in *MonoKSConfigStatus) DeepCopy() *MonoKSConfigStatus {
+	if in == nil {
+		return nil
+	}
+	out := new(MonoKSConfigStatus)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *NetworkSpec) DeepCopyInto(out *NetworkSpec) {
+	*out = *in
+	if in.DNSNameservers != nil {
+		in, out := &in.DNSNameservers, &out.DNSNameservers
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+	if in.DNSSearchDomains != nil {
+		in, out := &in.DNSSearchDomains, &out.DNSSearchDomains
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new NetworkSpec.
+func (in *NetworkSpec) DeepCopy() *NetworkSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(NetworkSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgrade) DeepCopyInto(out *OSUpgrade) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	in.Spec.DeepCopyInto(&out.Spec)
+	if in.Status != nil {
+		in, out := &in.Status, &out.Status
+		*out = new(OSUpgradeStatus)
+		(*in).DeepCopyInto(*out)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgrade.
+func (in *OSUpgrade) DeepCopy() *OSUpgrade {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgrade)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *OSUpgrade) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeList) DeepCopyInto(out *OSUpgradeList) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ListMeta.DeepCopyInto(&out.ListMeta)
+	if in.Items != nil {
+		in, out := &in.Items, &out.Items
+		*out = make([]OSUpgrade, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeList.
+func (in *OSUpgradeList) DeepCopy() *OSUpgradeList {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeList)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *OSUpgradeList) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeProgress) DeepCopyInto(out *OSUpgradeProgress) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	out.Spec = in.Spec
+	if in.Status != nil {
+		in, out := &in.Status, &out.Status
+		*out = new(OSUpgradeProgressStatus)
+		(*in).DeepCopyInto(*out)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeProgress.
+func (in *OSUpgradeProgress) DeepCopy() *OSUpgradeProgress {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeProgress)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *OSUpgradeProgress) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeProgressList) DeepCopyInto(out *OSUpgradeProgressList) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ListMeta.DeepCopyInto(&out.ListMeta)
+	if in.Items != nil {
+		in, out := &in.Items, &out.Items
+		*out = make([]OSUpgradeProgress, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeProgressList.
+func (in *OSUpgradeProgressList) DeepCopy() *OSUpgradeProgressList {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeProgressList)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *OSUpgradeProgressList) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeProgressSpec) DeepCopyInto(out *OSUpgradeProgressSpec) {
+	*out = *in
+	out.SourceRef = in.SourceRef
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeProgressSpec.
+func (in *OSUpgradeProgressSpec) DeepCopy() *OSUpgradeProgressSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeProgressSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeProgressStatus) DeepCopyInto(out *OSUpgradeProgressStatus) {
+	*out = *in
+	if in.StartedAt != nil {
+		in, out := &in.StartedAt, &out.StartedAt
+		*out = (*in).DeepCopy()
+	}
+	if in.CompletedAt != nil {
+		in, out := &in.CompletedAt, &out.CompletedAt
+		*out = (*in).DeepCopy()
+	}
+	if in.LastUpdatedAt != nil {
+		in, out := &in.LastUpdatedAt, &out.LastUpdatedAt
+		*out = (*in).DeepCopy()
+	}
+	if in.PlannedPath != nil {
+		in, out := &in.PlannedPath, &out.PlannedPath
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeProgressStatus.
+func (in *OSUpgradeProgressStatus) DeepCopy() *OSUpgradeProgressStatus {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeProgressStatus)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeSourceRef) DeepCopyInto(out *OSUpgradeSourceRef) {
+	*out = *in
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeSourceRef.
+func (in *OSUpgradeSourceRef) DeepCopy() *OSUpgradeSourceRef {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeSourceRef)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeSpec) DeepCopyInto(out *OSUpgradeSpec) {
+	*out = *in
+	if in.Catalog != nil {
+		in, out := &in.Catalog, &out.Catalog
+		*out = new(VersionCatalogSource)
+		**out = **in
+	}
+	if in.NodeSelector != nil {
+		in, out := &in.NodeSelector, &out.NodeSelector
+		*out = new(v1.LabelSelector)
+		(*in).DeepCopyInto(*out)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeSpec.
+func (in *OSUpgradeSpec) DeepCopy() *OSUpgradeSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeStatus) DeepCopyInto(out *OSUpgradeStatus) {
+	*out = *in
+	if in.Conditions != nil {
+		in, out := &in.Conditions, &out.Conditions
+		*out = make([]v1.Condition, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeStatus.
+func (in *OSUpgradeStatus) DeepCopy() *OSUpgradeStatus {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeStatus)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *VersionCatalogSource) DeepCopyInto(out *VersionCatalogSource) {
+	*out = *in
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new VersionCatalogSource.
+func (in *VersionCatalogSource) DeepCopy() *VersionCatalogSource {
+	if in == nil {
+		return nil
+	}
+	out := new(VersionCatalogSource)
+	in.DeepCopyInto(out)
+	return out
+}

clitools/pkg/assets/crds/monok8s.io_monoksconfigs.yaml

@@ -0,0 +1,178 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.20.1
+  name: monoksconfigs.monok8s.io
+spec:
+  group: monok8s.io
+  names:
+    kind: MonoKSConfig
+    listKind: MonoKSConfigList
+    plural: monoksconfigs
+    singular: monoksconfig
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              allowSchedulingOnControlPlane:
+                type: boolean
+              apiServerAdvertiseAddress:
+                type: string
+              apiServerEndpoint:
+                type: string
+              bootstrapToken:
+                type: string
+              clusterDomain:
+                type: string
+              clusterName:
+                type: string
+              clusterRole:
+                type: string
+              cniPlugin:
+                type: string
+              containerRuntimeEndpoint:
+                type: string
+              controlPlaneCertKey:
+                type: string
+              discoveryTokenCACertHash:
+                type: string
+              enableNodeControl:
+                type: boolean
+              initControlPlane:
+                type: boolean
+              kubeProxyNodePortAddresses:
+                items:
+                  type: string
+                type: array
+              kubernetesVersion:
+                type: string
+              network:
+                properties:
+                  dnsNameservers:
+                    items:
+                      type: string
+                    type: array
+                  dnsSearchDomains:
+                    items:
+                      type: string
+                    type: array
+                  hostname:
+                    type: string
+                  managementCIDR:
+                    type: string
+                  managementGateway:
+                    type: string
+                  managementIface:
+                    type: string
+                type: object
+              nodeLabels:
+                additionalProperties:
+                  type: string
+                type: object
+              nodeName:
+                type: string
+              podSubnet:
+                type: string
+              serviceSubnet:
+                type: string
+              skipImageCheck:
+                type: boolean
+              subjectAltNames:
+                items:
+                  type: string
+                type: array
+            type: object
+          status:
+            properties:
+              appliedSteps:
+                items:
+                  type: string
+                type: array
+              conditions:
+                items:
+                  description: Condition contains details for one aspect of the current
+                    state of this API Resource.
+                  properties:
+                    lastTransitionTime:
+                      description: |-
+                        lastTransitionTime is the last time the condition transitioned from one status to another.
+                        This should be when the underlying condition changed.  If that is not known, then using the time when the API field changed is acceptable.
+                      format: date-time
+                      type: string
+                    message:
+                      description: |-
+                        message is a human readable message indicating details about the transition.
+                        This may be an empty string.
+                      maxLength: 32768
+                      type: string
+                    observedGeneration:
+                      description: |-
+                        observedGeneration represents the .metadata.generation that the condition was set based upon.
+                        For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
+                        with respect to the current state of the instance.
+                      format: int64
+                      minimum: 0
+                      type: integer
+                    reason:
+                      description: |-
+                        reason contains a programmatic identifier indicating the reason for the condition's last transition.
+                        Producers of specific condition types may define expected values and meanings for this field,
+                        and whether the values are considered a guaranteed API.
+                        The value should be a CamelCase string.
+                        This field may not be empty.
+                      maxLength: 1024
+                      minLength: 1
+                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
+                      type: string
+                    status:
+                      description: status of the condition, one of True, False, Unknown.
+                      enum:
+                      - "True"
+                      - "False"
+                      - Unknown
+                      type: string
+                    type:
+                      description: type of condition in CamelCase or in foo.example.com/CamelCase.
+                      maxLength: 316
+                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
+                      type: string
+                  required:
+                  - lastTransitionTime
+                  - message
+                  - reason
+                  - status
+                  - type
+                  type: object
+                type: array
+              observedGeneration:
+                format: int64
+                type: integer
+              phase:
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true

clitools/pkg/assets/crds/monok8s.io_osupgradeprogresses.yaml

@@ -0,0 +1,124 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.20.1
+  name: osupgradeprogresses.monok8s.io
+spec:
+  group: monok8s.io
+  names:
+    kind: OSUpgradeProgress
+    listKind: OSUpgradeProgressList
+    plural: osupgradeprogresses
+    shortNames:
+    - osup
+    singular: osupgradeprogress
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.nodeName
+      name: Node
+      type: string
+    - jsonPath: .spec.sourceRef.name
+      name: Source
+      type: string
+    - jsonPath: .status.currentVersion
+      name: Current
+      type: string
+    - jsonPath: .status.targetVersion
+      name: Target
+      type: string
+    - jsonPath: .status.phase
+      name: Phase
+      type: string
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: Specification of the desired behavior of the OSUpgradeProgress.
+            properties:
+              nodeName:
+                type: string
+              retryNonce:
+                description: |-
+                  RetryNonce triggers a retry when its value changes.
+                  Users can update this field (for example, set it to the current time)
+                  to request a retry of a failed OS upgrade.
+                type: string
+              sourceRef:
+                properties:
+                  name:
+                    type: string
+                  namespace:
+                    type: string
+                type: object
+            type: object
+          status:
+            description: Most recently observed status of the OSUpgradeProgress.
+            properties:
+              completedAt:
+                format: date-time
+                type: string
+              currentFrom:
+                type: string
+              currentStep:
+                format: int32
+                type: integer
+              currentTo:
+                type: string
+              currentVersion:
+                type: string
+              failureReason:
+                type: string
+              inactivePartition:
+                type: string
+              lastUpdatedAt:
+                format: date-time
+                type: string
+              message:
+                type: string
+              observedRetryNonce:
+                description: |-
+                  ObservedRetryNonce records the last retryNonce value the agent accepted.
+                  When spec.retryNonce is changed by the user and differs from this value,
+                  the agent may retry a failed upgrade.
+                type: string
+              phase:
+                type: string
+              plannedPath:
+                items:
+                  type: string
+                type: array
+              retryCount:
+                format: int32
+                type: integer
+              startedAt:
+                format: date-time
+                type: string
+              targetVersion:
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}

clitools/pkg/assets/crds/monok8s.io_osupgrades.yaml

@@ -0,0 +1,202 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.20.1
+  name: osupgrades.monok8s.io
+spec:
+  group: monok8s.io
+  names:
+    kind: OSUpgrade
+    listKind: OSUpgradeList
+    plural: osupgrades
+    shortNames:
+    - osu
+    singular: osupgrade
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.desiredVersion
+      name: Desired
+      type: string
+    - jsonPath: .status.resolvedVersion
+      name: Resolved
+      type: string
+    - jsonPath: .status.phase
+      name: Phase
+      type: string
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: Specification of the desired behavior of the OSUpgrade.
+            properties:
+              catalog:
+                properties:
+                  configMapRef:
+                    type: string
+                  inline:
+                    type: string
+                  url:
+                    type: string
+                type: object
+              desiredVersion:
+                minLength: 1
+                type: string
+              flashProfile:
+                default: balanced
+                description: |-
+                  Profiles (TODO)
+                     safe     - api-server can be responsive most of the time
+                     balanced - api-server can sometimes be unresponsive
+                     fast     - disable throttling. Good for worker node.
+                enum:
+                - fast
+                - balanced
+                - safe
+                type: string
+              nodeSelector:
+                description: |-
+                  A label selector is a label query over a set of resources. The result of matchLabels and
+                  matchExpressions are ANDed. An empty label selector matches all objects. A null
+                  label selector matches no objects.
+                properties:
+                  matchExpressions:
+                    description: matchExpressions is a list of label selector requirements.
+                      The requirements are ANDed.
+                    items:
+                      description: |-
+                        A label selector requirement is a selector that contains values, a key, and an operator that
+                        relates the key and values.
+                      properties:
+                        key:
+                          description: key is the label key that the selector applies
+                            to.
+                          type: string
+                        operator:
+                          description: |-
+                            operator represents a key's relationship to a set of values.
+                            Valid operators are In, NotIn, Exists and DoesNotExist.
+                          type: string
+                        values:
+                          description: |-
+                            values is an array of string values. If the operator is In or NotIn,
+                            the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                            the values array must be empty. This array is replaced during a strategic
+                            merge patch.
+                          items:
+                            type: string
+                          type: array
+                          x-kubernetes-list-type: atomic
+                      required:
+                      - key
+                      - operator
+                      type: object
+                    type: array
+                    x-kubernetes-list-type: atomic
+                  matchLabels:
+                    additionalProperties:
+                      type: string
+                    description: |-
+                      matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                      map is equivalent to an element of matchExpressions, whose key field is "key", the
+                      operator is "In", and the values array contains only "value". The requirements are ANDed.
+                    type: object
+                type: object
+                x-kubernetes-map-type: atomic
+            type: object
+          status:
+            description: Most recently observed status of the OSUpgrade.
+            properties:
+              conditions:
+                items:
+                  description: Condition contains details for one aspect of the current
+                    state of this API Resource.
+                  properties:
+                    lastTransitionTime:
+                      description: |-
+                        lastTransitionTime is the last time the condition transitioned from one status to another.
+                        This should be when the underlying condition changed.  If that is not known, then using the time when the API field changed is acceptable.
+                      format: date-time
+                      type: string
+                    message:
+                      description: |-
+                        message is a human readable message indicating details about the transition.
+                        This may be an empty string.
+                      maxLength: 32768
+                      type: string
+                    observedGeneration:
+                      description: |-
+                        observedGeneration represents the .metadata.generation that the condition was set based upon.
+                        For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
+                        with respect to the current state of the instance.
+                      format: int64
+                      minimum: 0
+                      type: integer
+                    reason:
+                      description: |-
+                        reason contains a programmatic identifier indicating the reason for the condition's last transition.
+                        Producers of specific condition types may define expected values and meanings for this field,
+                        and whether the values are considered a guaranteed API.
+                        The value should be a CamelCase string.
+                        This field may not be empty.
+                      maxLength: 1024
+                      minLength: 1
+                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
+                      type: string
+                    status:
+                      description: status of the condition, one of True, False, Unknown.
+                      enum:
+                      - "True"
+                      - "False"
+                      - Unknown
+                      type: string
+                    type:
+                      description: type of condition in CamelCase or in foo.example.com/CamelCase.
+                      maxLength: 316
+                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
+                      type: string
+                  required:
+                  - lastTransitionTime
+                  - message
+                  - reason
+                  - status
+                  - type
+                  type: object
+                type: array
+              message:
+                type: string
+              observedGeneration:
+                format: int64
+                type: integer
+              phase:
+                type: string
+              reason:
+                type: string
+              resolvedVersion:
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}

clitools/pkg/assets/embed.go

@@ -0,0 +1,6 @@
+package assets
+
+import "embed"
+
+//go:embed crds/*.yaml
+var CRDs embed.FS

clitools/pkg/assets/render.go

@@ -0,0 +1,49 @@
+package assets
+
+import (
+	"fmt"
+	"io"
+	"path/filepath"
+	"sort"
+)
+
+func PrintCRDs(out io.Writer) error {
+	entries, err := CRDs.ReadDir("crds")
+	if err != nil {
+		return err
+	}
+
+	names := make([]string, 0, len(entries))
+	for _, entry := range entries {
+		if entry.IsDir() {
+			continue
+		}
+		if filepath.Ext(entry.Name()) != ".yaml" {
+			continue
+		}
+		names = append(names, entry.Name())
+	}
+
+	sort.Strings(names)
+
+	for _, name := range names {
+		b, err := CRDs.ReadFile("crds/" + name)
+		if err != nil {
+			return err
+		}
+
+		if _, err := fmt.Fprintln(out, "---"); err != nil {
+			return err
+		}
+		if _, err := out.Write(b); err != nil {
+			return err
+		}
+		if len(b) == 0 || b[len(b)-1] != '\n' {
+			if _, err := fmt.Fprintln(out); err != nil {
+				return err
+			}
+		}
+	}
+
+	return nil
+}

clitools/pkg/bootstrap/registry.go

@@ -23,7 +23,7 @@ func NewRegistry(ctx *node.NodeContext) *Registry {
 
 	return &Registry{
 		steps: map[string]node.Step{
-			"ApplyControlAgentDaemonSetResources":    node.ApplyControlAgentDaemonSetResources,
+			"ApplyNodeControlDaemonSetResources":    node.ApplyNodeControlDaemonSetResources,
 			"ApplyLocalNodeMetadataIfPossible":       node.ApplyLocalNodeMetadataIfPossible,
 			"CheckForVersionSkew":                    node.CheckForVersionSkew,
 			"ClassifyBootstrapAction":                node.ClassifyBootstrapAction,

clitools/pkg/bootstrap/runner.go

@@ -70,7 +70,7 @@ func NewRunner(cfg *monov1alpha1.MonoKSConfig) *Runner {
 			{
 				RegKey: "EngageControlGate",
 				Name:   "Engage the control gate",
-				Desc:   "Prevents agent polling resources prematurely",
+				Desc:   "Prevents agent watching resources prematurely",
 			},
 			{
 				RegKey: "StartCRIO",
@@ -112,6 +112,11 @@ func NewRunner(cfg *monov1alpha1.MonoKSConfig) *Runner {
 				Name:   "Wait for existing cluster",
 				Desc:   "Block until control plane is reachable when joining or reconciling an existing cluster",
 			},
+			{
+				RegKey: "CheckForVersionSkew",
+				Name:   "Check for version skew",
+				Desc:   "Validate wether version satisfy the requirements againts current cluster if any",
+			},
 			{
 				RegKey: "ReconcileControlPlane",
 				Name:   "Reconcile control plane",
@@ -122,11 +127,6 @@ func NewRunner(cfg *monov1alpha1.MonoKSConfig) *Runner {
 				Name:   "Reconcile worker node",
 				Desc:   "Reconcile the worker node",
 			},
-			{
-				RegKey: "CheckForVersionSkew",
-				Name:   "Check for version skew",
-				Desc:   "Validate wether version satisfy the requirements againts current cluster if any",
-			},
 			{
 				RegKey: "RunKubeadmUpgradeApply",
 				Name:   "Run kubeadm upgrade apply",
@@ -158,14 +158,14 @@ func NewRunner(cfg *monov1alpha1.MonoKSConfig) *Runner {
 				Desc:   "Make A/B booting possible",
 			},
 			{
-				RegKey: "ApplyControlAgentDaemonSetResources",
+				RegKey: "ApplyNodeControlDaemonSetResources",
 				Name:   "Apply daemonset for control agent",
 				Desc:   "Control agent handles OSUpgrade resources",
 			},
 			{
 				RegKey: "ReleaseControlGate",
 				Name:   "Release the control gate",
-				Desc:   "Allow agent to start polling resources",
+				Desc:   "Allow agent to start watching resources",
 			},
 		},
 	}

clitools/pkg/cmd/agent/agent.go

@@ -9,10 +9,7 @@ import (
 
 	"github.com/spf13/cobra"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
+	"k8s.io/apimachinery/pkg/watch"
-	"k8s.io/apimachinery/pkg/labels"
-	"k8s.io/apimachinery/pkg/runtime"
-	"k8s.io/apimachinery/pkg/runtime/schema"
 	"k8s.io/cli-runtime/pkg/genericclioptions"
 	"k8s.io/klog/v2"
 
@@ -23,19 +20,18 @@ import (
 	"example.com/monok8s/pkg/templates"
 )
 
-const defaultPollInterval = 15 * time.Second
-
-var runtimeDefaultUnstructuredConverter = runtime.DefaultUnstructuredConverter
-
 func NewCmdAgent(flags *genericclioptions.ConfigFlags) *cobra.Command {
-	var namespace string
 	var envFile string
-	var pollInterval time.Duration
 
 	cmd := &cobra.Command{
 		Use:   "agent --env-file path",
-		Short: "Watch OSUpgrade resources and process matching upgrades for this node",
+		Short: "Watch OSUpgradeProgress resources for this node and process upgrades",
 		RunE: func(cmd *cobra.Command, _ []string) error {
+			ns, _, err := flags.ToRawKubeConfigLoader().Namespace()
+			if err != nil {
+				return err
+			}
+
 			if envFile == "" {
 				return fmt.Errorf("--env-file is required")
 			}
@@ -59,9 +55,8 @@ func NewCmdAgent(flags *genericclioptions.ConfigFlags) *cobra.Command {
 
 			klog.InfoS("starting agent",
 				"node", cfg.Spec.NodeName,
-				"namespace", namespace,
+				"namespace", ns,
 				"envFile", envFile,
-				"pollInterval", pollInterval,
 			)
 
 			clients, err := kube.NewClients(flags)
@@ -69,13 +64,11 @@ func NewCmdAgent(flags *genericclioptions.ConfigFlags) *cobra.Command {
 				return fmt.Errorf("create kube clients: %w", err)
 			}
 
-			return runPollLoop(ctx, clients, namespace, cfg.Spec.NodeName, pollInterval)
+			return runWatchLoop(ctx, clients, ns, cfg.Spec.NodeName)
 		},
 	}
 
-	cmd.Flags().StringVar(&namespace, "namespace", "kube-system", "namespace to watch")
 	cmd.Flags().StringVar(&envFile, "env-file", "", "path to env file containing MKS_* variables")
-	cmd.Flags().DurationVar(&pollInterval, "poll-interval", defaultPollInterval, "poll interval for OSUpgrade resources")
 
 	return cmd
 }
@@ -94,9 +87,9 @@ func waitForControlGate(ctx context.Context, envFile string, pollInterval time.D
 	for {
 		_, err := os.Stat(marker)
 		if err == nil {
-			klog.InfoS("Control gate is present; waiting before starting poll loop", "path", marker)
+			klog.InfoS("Control gate is present; waiting before starting watch loop", "path", marker)
 		} else if os.IsNotExist(err) {
-			klog.InfoS("Control gate not present; starting poll loop", "path", marker)
+			klog.InfoS("Control gate not present; starting watch loop", "path", marker)
 			return nil
 		} else {
 			return fmt.Errorf("stat upgrade marker %s: %w", marker, err)
@@ -110,128 +103,150 @@ func waitForControlGate(ctx context.Context, envFile string, pollInterval time.D
 	}
 }
 
-func runPollLoop(ctx context.Context, clients *kube.Clients, namespace, nodeName string, interval time.Duration) error {
+func runWatchLoop(ctx context.Context, clients *kube.Clients, namespace, nodeName string) error {
-	gvr := schema.GroupVersionResource{
+	var resourceVersion string
-		Group:    monov1alpha1.Group,
-		Version:  monov1alpha1.Version,
-		Resource: "osupgrades",
-	}
-
-	ticker := time.NewTicker(interval)
-	defer ticker.Stop()
 
 	for {
-		if err := pollOnce(ctx, clients, gvr, namespace, nodeName); err != nil {
+		if ctx.Err() != nil {
-			klog.ErrorS(err, "poll failed", "namespace", namespace, "node", nodeName)
+			return ctx.Err()
 		}
 
+		err := watchOnce(ctx, clients, namespace, nodeName, &resourceVersion)
+		if err != nil {
+			if ctx.Err() != nil {
+				return ctx.Err()
+			}
+			klog.ErrorS(err, "watch failed; retrying",
+				"namespace", namespace,
+				"node", nodeName,
+				"resourceVersion", resourceVersion,
+			)
 			select {
 			case <-ctx.Done():
 				return ctx.Err()
-		case <-ticker.C:
+			case <-time.After(2 * time.Second):
+			}
+			continue
 		}
 	}
 }
 
-func pollOnce(
+func watchOnce(
 	ctx context.Context,
 	clients *kube.Clients,
-	gvr schema.GroupVersionResource,
 	namespace string,
 	nodeName string,
+	resourceVersion *string,
 ) error {
-	list, err := clients.Dynamic.Resource(gvr).Namespace(namespace).List(ctx, metav1.ListOptions{})
+
+	list, err := clients.MonoKS.
+		Monok8sV1alpha1().
+		OSUpgradeProgresses(namespace).
+		List(ctx, metav1.ListOptions{})
 	if err != nil {
-		return fmt.Errorf("list osupgrades: %w", err)
+		return fmt.Errorf("list osupgradeprogresses: %w", err)
-	}
-
-	klog.InfoS("agent tick", "namespace", namespace, "items", len(list.Items), "node", nodeName)
-
-	nodeLabels := labels.Set{
-		"kubernetes.io/hostname":   nodeName,
-		"monok8s.io/node-name":     nodeName,
-		"monok8s.io/control-agent": "true",
 	}
 
 	for i := range list.Items {
 		item := &list.Items[i]
+		if !targetsNode(item, nodeName) {
+			continue
+		}
 
-		osu, err := decodeOSUpgrade(item)
+		klog.InfoS("found existing osupgradeprogress",
+			"name", item.Name,
+			"node", nodeName,
+			"phase", item.StatusPhase(),
+			"resourceVersion", item.ResourceVersion,
+		)
+
+		if err := osupgradeController.HandleOSUpgradeProgress(ctx, clients, namespace, nodeName, item); err != nil {
+			klog.ErrorS(err, "failed to handle existing osupgradeprogress",
+				"name", item.Name,
+				"node", nodeName,
+			)
+		}
+	}
+
+	*resourceVersion = list.ResourceVersion
+
+	w, err := clients.MonoKS.
+		Monok8sV1alpha1().
+		OSUpgradeProgresses(namespace).
+		Watch(ctx, metav1.ListOptions{
+			ResourceVersion: *resourceVersion,
+		})
 	if err != nil {
-			klog.ErrorS(err, "failed to decode osupgrade",
+		return fmt.Errorf("watch osupgradeprogresses: %w", err)
-				"name", item.GetName(),
+	}
-				"resourceVersion", item.GetResourceVersion(),
+	defer w.Stop()
+
+	klog.InfoS("watching osupgradeprogresses",
+		"namespace", namespace,
+		"node", nodeName,
+		"resourceVersion", *resourceVersion,
+	)
+
+	for {
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+
+		case evt, ok := <-w.ResultChan():
+			if !ok {
+				return fmt.Errorf("watch channel closed")
+			}
+
+			switch evt.Type {
+			case watch.Bookmark:
+				obj, ok := evt.Object.(*monov1alpha1.OSUpgradeProgress)
+				if ok && obj != nil && obj.ResourceVersion != "" {
+					*resourceVersion = obj.ResourceVersion
+				}
+				continue
+
+			case watch.Error:
+				return fmt.Errorf("watch returned error event")
+			}
+
+			osup, ok := evt.Object.(*monov1alpha1.OSUpgradeProgress)
+			if !ok {
+				klog.V(1).InfoS("skipping unexpected watch object type",
+					"type", fmt.Sprintf("%T", evt.Object),
 				)
 				continue
 			}
 
-		if !matchesNode(osu, nodeName, nodeLabels) {
+			if osup.ResourceVersion != "" {
-			klog.V(2).InfoS("skipping osupgrade; not targeted to this node",
+				*resourceVersion = osup.ResourceVersion
-				"name", osu.Name,
+			}
-				"node", nodeName,
+
-			)
+			if !targetsNode(osup, nodeName) {
 				continue
 			}
 
-		klog.InfoS("matched osupgrade",
+			klog.V(4).InfoS("received osupgradeprogress event",
-			"name", osu.Name,
+				"name", osup.Name,
 				"node", nodeName,
-			"desiredVersion", osu.Spec.DesiredVersion,
+				"phase", osup.StatusPhase(),
-			"phase", statusPhase(osu.Status),
+				"eventType", evt.Type,
-			"resourceVersion", osu.ResourceVersion,
+				"resourceVersion", osup.ResourceVersion,
 			)
 
-		if err := osupgradeController.HandleOSUpgrade(ctx, clients, namespace, nodeName, osu); err != nil {
+			if err := osupgradeController.HandleOSUpgradeProgress(ctx, clients, namespace, nodeName, osup); err != nil {
-			klog.ErrorS(err, "failed to handle osupgrade",
+				klog.ErrorS(err, "failed to handle osupgradeprogress",
-				"name", osu.Name,
+					"name", osup.Name,
 					"node", nodeName,
+					"eventType", evt.Type,
 				)
-			continue
 			}
 		}
-
+	}
-	return nil
 }
 
-func decodeOSUpgrade(item *unstructured.Unstructured) (*monov1alpha1.OSUpgrade, error) {
+func targetsNode(osup *monov1alpha1.OSUpgradeProgress, nodeName string) bool {
-	var osu monov1alpha1.OSUpgrade
+	if osup == nil {
-	if err := runtimeDefaultUnstructuredConverter.FromUnstructured(item.Object, &osu); err != nil {
-		return nil, fmt.Errorf("convert unstructured to OSUpgrade: %w", err)
-	}
-	return &osu, nil
-}
-
-func matchesNode(osu *monov1alpha1.OSUpgrade, nodeName string, nodeLabels labels.Set) bool {
-	if osu == nil {
 		return false
 	}
-
+	return osup.Spec.NodeName == nodeName
-	sel := osu.Spec.NodeSelector
-	if sel == nil {
-		// No selector means "match all nodes".
-		return true
-	}
-
-	selector, err := metav1.LabelSelectorAsSelector(sel)
-	if err != nil {
-		klog.ErrorS(err, "invalid node selector on osupgrade", "name", osu.Name)
-		return false
-	}
-
-	if selector.Empty() {
-		return true
-	}
-
-	if selector.Matches(nodeLabels) {
-		return true
-	}
-
-	return false
-}
-
-func statusPhase(st *monov1alpha1.OSUpgradeStatus) string {
-	if st == nil {
-		return ""
-	}
-	return string(st.Phase)
 }

clitools/pkg/cmd/controller/controller.go

@@ -0,0 +1,217 @@
+package controller
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"net"
+	"net/http"
+	"os"
+	"time"
+
+	"github.com/spf13/cobra"
+	"k8s.io/cli-runtime/pkg/genericclioptions"
+	"k8s.io/klog/v2"
+
+	mkscontroller "example.com/monok8s/pkg/controller"
+	osupgradectrl "example.com/monok8s/pkg/controller/osupgrade"
+	"example.com/monok8s/pkg/kube"
+)
+
+type ServerConfig struct {
+	Namespace         string `json:"namespace,omitempty"`
+	TLSCertFile       string `json:"tlsCertFile,omitempty"`
+	TLSPrivateKeyFile string `json:"tlsPrivateKeyFile,omitempty"`
+}
+
+func NewCmdController(flags *genericclioptions.ConfigFlags) *cobra.Command {
+	var conf ServerConfig
+
+	cmd := &cobra.Command{
+		Use:   "controller",
+		Short: "Start a controller that handles OSUpgrade resources",
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			ns, _, err := flags.ToRawKubeConfigLoader().Namespace()
+			if err != nil {
+				return err
+			}
+			conf.Namespace = ns
+
+			ctx := cmd.Context()
+
+			klog.InfoS("starting controller", "namespace", conf.Namespace)
+
+			clients, err := kube.NewClients(flags)
+			if err != nil {
+				return err
+			}
+
+			ctx, cancel := context.WithCancel(ctx)
+			defer cancel()
+
+			httpErrCh := make(chan error, 1)
+			watchErrCh := make(chan error, 1)
+
+			go func() {
+				klog.InfoS("starting OSUpgrade watch loop", "namespace", conf.Namespace)
+				watchErrCh <- osupgradectrl.Watch(ctx, clients, conf.Namespace)
+			}()
+
+			go func() {
+				httpErrCh <- listenAndServe(ctx, clients, conf)
+			}()
+
+			select {
+			case <-ctx.Done():
+				klog.InfoS("controller context canceled")
+				return ctx.Err()
+
+			case err := <-watchErrCh:
+				if err != nil && !errors.Is(err, context.Canceled) {
+					cancel()
+					return err
+				}
+				cancel()
+				return nil
+
+			case err := <-httpErrCh:
+				if err != nil && !errors.Is(err, context.Canceled) {
+					cancel()
+					return err
+				}
+				cancel()
+				return nil
+			}
+		},
+	}
+
+	cmd.Flags().StringVar(&conf.TLSCertFile, "tls-cert-file", conf.TLSCertFile,
+		"File containing x509 Certificate used for serving HTTPS (with intermediate certs, if any, concatenated after server cert).")
+	cmd.Flags().StringVar(&conf.TLSPrivateKeyFile, "tls-private-key-file", conf.TLSPrivateKeyFile,
+		"File containing x509 private key matching --tls-cert-file.")
+
+	return cmd
+}
+
+func listenAndServe(ctx context.Context, clients *kube.Clients, conf ServerConfig) error {
+	nodeName := os.Getenv("NODE_NAME")
+
+	controllerServer := mkscontroller.NewServer(ctx, clients, conf.Namespace, nodeName)
+
+	healthMux := http.NewServeMux()
+	healthMux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("ok\n"))
+	})
+	healthMux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("ok\n"))
+	})
+
+	healthAddr := net.JoinHostPort("", "8080")
+	controllerAddr := net.JoinHostPort("", "8443")
+
+	healthHTTPServer := &http.Server{
+		Addr:           healthAddr,
+		Handler:        healthMux,
+		IdleTimeout:    90 * time.Second,
+		ReadTimeout:    10 * time.Second,
+		WriteTimeout:   10 * time.Second,
+		MaxHeaderBytes: 1 << 20,
+	}
+
+	controllerHTTPServer := &http.Server{
+		Addr:           controllerAddr,
+		Handler:        controllerServer,
+		IdleTimeout:    90 * time.Second,
+		ReadTimeout:    4 * time.Minute,
+		WriteTimeout:   4 * time.Minute,
+		MaxHeaderBytes: 1 << 20,
+	}
+
+	serverErrCh := make(chan error, 2)
+
+	go func() {
+		klog.InfoS("starting health HTTP server", "addr", healthAddr)
+
+		err := healthHTTPServer.ListenAndServe()
+		if err != nil && !errors.Is(err, http.ErrServerClosed) {
+			serverErrCh <- fmt.Errorf("health HTTP server: %w", err)
+			return
+		}
+
+		serverErrCh <- nil
+	}()
+
+	go func() {
+		if conf.TLSCertFile != "" {
+			klog.InfoS("starting controller HTTPS server",
+				"addr", controllerAddr,
+				"certFile", conf.TLSCertFile,
+				"keyFile", conf.TLSPrivateKeyFile,
+			)
+
+			err := controllerHTTPServer.ListenAndServeTLS(conf.TLSCertFile, conf.TLSPrivateKeyFile)
+			if err != nil && !errors.Is(err, http.ErrServerClosed) {
+				serverErrCh <- fmt.Errorf("controller HTTPS server: %w", err)
+				return
+			}
+
+			serverErrCh <- nil
+			return
+		}
+
+		klog.InfoS("starting controller HTTP server", "addr", controllerAddr)
+
+		err := controllerHTTPServer.ListenAndServe()
+		if err != nil && !errors.Is(err, http.ErrServerClosed) {
+			serverErrCh <- fmt.Errorf("controller HTTP server: %w", err)
+			return
+		}
+
+		serverErrCh <- nil
+	}()
+
+	select {
+	case <-ctx.Done():
+		klog.InfoS("shutting down HTTP servers",
+			"healthAddr", healthAddr,
+			"controllerAddr", controllerAddr,
+		)
+
+		shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+		defer cancel()
+
+		var errs []error
+
+		if err := healthHTTPServer.Shutdown(shutdownCtx); err != nil {
+			errs = append(errs, fmt.Errorf("shutdown health HTTP server: %w", err))
+		}
+
+		if err := controllerHTTPServer.Shutdown(shutdownCtx); err != nil {
+			errs = append(errs, fmt.Errorf("shutdown controller HTTP server: %w", err))
+		}
+
+		for i := 0; i < 2; i++ {
+			if err := <-serverErrCh; err != nil {
+				errs = append(errs, err)
+			}
+		}
+
+		if len(errs) > 0 {
+			return errors.Join(errs...)
+		}
+
+		return context.Canceled
+
+	case err := <-serverErrCh:
+		if err != nil {
+			klog.ErrorS(err, "HTTP server failed")
+			return err
+		}
+
+		// One server exited cleanly unexpectedly. Treat that as failure because
+		// the process should keep both servers alive until ctx is canceled.
+		return fmt.Errorf("HTTP server exited unexpectedly")
+	}
+}

clitools/pkg/cmd/create/create.go

@@ -1,13 +1,18 @@
 package create
 
 import (
+	"bytes"
 	"fmt"
 	"github.com/spf13/cobra"
+	"k8s.io/cli-runtime/pkg/genericclioptions"
+	"os"
+	"strings"
 
+	assets "example.com/monok8s/pkg/assets"
 	render "example.com/monok8s/pkg/render"
 )
 
-func NewCmdCreate() *cobra.Command {
+func NewCmdCreate(flags *genericclioptions.ConfigFlags) *cobra.Command {
 	cmd := &cobra.Command{Use: "create", Short: "Create starter resources"}
 	cmd.AddCommand(
 		&cobra.Command{
@@ -26,7 +31,12 @@ func NewCmdCreate() *cobra.Command {
 			Use:   "osupgrade",
 			Short: "Print an OSUpgrade template",
 			RunE: func(cmd *cobra.Command, _ []string) error {
-				out, err := render.RenderOSUpgrade()
+				ns, _, err := flags.ToRawKubeConfigLoader().Namespace()
+				if err != nil {
+					return err
+				}
+
+				out, err := render.RenderOSUpgrade(ns)
 				if err != nil {
 					return err
 				}
@@ -34,6 +44,143 @@ func NewCmdCreate() *cobra.Command {
 				return err
 			},
 		},
+		&cobra.Command{
+			Use:   "crds",
+			Short: "Print the bundled CRDs",
+			RunE: func(cmd *cobra.Command, _ []string) error {
+				return assets.PrintCRDs(cmd.OutOrStdout())
+			},
+		},
 	)
+
+	var authorizedKeysPath string
+
+	sshdcmd := cobra.Command{
+		Use:   "sshd",
+		Short: "Print sshd deployments template",
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			ns, _, err := flags.ToRawKubeConfigLoader().Namespace()
+			if err != nil {
+				return err
+			}
+
+			authorizedKeys, err := readAuthorizedKeysFile(authorizedKeysPath)
+			if err != nil {
+				return err
+			}
+
+			out, err := render.RenderSSHDDeployments(ns, authorizedKeys)
+			if err != nil {
+				return err
+			}
+
+			_, err = fmt.Fprint(cmd.OutOrStdout(), out)
+			return err
+		},
+	}
+
+	sshdcmd.Flags().StringVar(&authorizedKeysPath, "authkeys", "", "path to authorized_keys file")
+
+	cmd.AddCommand(&sshdcmd)
+
+	cconf := render.ControllerConf{}
+	controllercmd := cobra.Command{
+		Use:   "controller",
+		Short: "Print controller deployments template",
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if len(cconf.ImagePullSecrets) > 0 && strings.TrimSpace(cconf.Image) == "" {
+				return fmt.Errorf("--image-pull-secret requires --image")
+			}
+
+			ns, _, err := flags.ToRawKubeConfigLoader().Namespace()
+			if err != nil {
+				return err
+			}
+
+			cconf.Namespace = ns
+
+			out, err := render.RenderControllerDeployments(cconf)
+			if err != nil {
+				return err
+			}
+
+			_, err = fmt.Fprint(cmd.OutOrStdout(), out)
+			return err
+		},
+	}
+
+	controllercmd.Flags().StringVar(
+		&cconf.Image,
+		"image",
+		"",
+		"Controller image, including optional registry and tag",
+	)
+	controllercmd.Flags().StringSliceVar(
+		&cconf.ImagePullSecrets,
+		"image-pull-secret",
+		nil,
+		"Image pull secret name for the agent image; may be specified multiple times or as a comma-separated list",
+	)
+
+	cmd.AddCommand(&controllercmd)
+
+	aconf := render.AgentConf{}
+	agentcmd := cobra.Command{
+		Use:   "agent",
+		Short: "Print agent daemonsets template",
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if len(aconf.ImagePullSecrets) > 0 && strings.TrimSpace(aconf.Image) == "" {
+				return fmt.Errorf("--image-pull-secret requires --image")
+			}
+
+			ns, _, err := flags.ToRawKubeConfigLoader().Namespace()
+			if err != nil {
+				return err
+			}
+
+			aconf.Namespace = ns
+
+			out, err := render.RenderAgentDaemonSets(aconf)
+			if err != nil {
+				return err
+			}
+
+			_, err = fmt.Fprint(cmd.OutOrStdout(), out)
+			return err
+		},
+	}
+
+	agentcmd.Flags().StringVar(
+		&aconf.Image,
+		"image",
+		"",
+		"Agent image, including optional registry and tag",
+	)
+	agentcmd.Flags().StringSliceVar(
+		&aconf.ImagePullSecrets,
+		"image-pull-secret",
+		nil,
+		"Image pull secret name for the agent image; may be specified multiple times or as a comma-separated list",
+	)
+
+	cmd.AddCommand(&agentcmd)
+
 	return cmd
 }
+
+func readAuthorizedKeysFile(path string) (string, error) {
+	if path == "" {
+		return "", fmt.Errorf("--authkeys is required")
+	}
+
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return "", fmt.Errorf("read authorized_keys file %q: %w", path, err)
+	}
+
+	if len(bytes.TrimSpace(b)) == 0 {
+		return "", fmt.Errorf("authorized_keys file %q is empty", path)
+	}
+
+	return string(b), nil
+}

clitools/pkg/cmd/root/root.go

@@ -6,6 +6,7 @@ import (
 
 	agentcmd "example.com/monok8s/pkg/cmd/agent"
 	checkconfigcmd "example.com/monok8s/pkg/cmd/checkconfig"
+	controllercmd "example.com/monok8s/pkg/cmd/controller"
 	createcmd "example.com/monok8s/pkg/cmd/create"
 	initcmd "example.com/monok8s/pkg/cmd/initcmd"
 	internalcmd "example.com/monok8s/pkg/cmd/internal"
@@ -18,8 +19,10 @@ import (
 func init() {
 	klog.InitFlags(nil)
 
+	_ = flag.Set("logtostderr", "true")
+
 	if os.Getenv("DEBUG") != "" {
-		_ = flag.Set("v", "4") // debug level
+		_ = flag.Set("v", "4")
 	} else {
 		_ = flag.Set("v", "0")
 	}
@@ -38,14 +41,20 @@ func NewRootCmd() *cobra.Command {
 		},
 	}
 
+	// Expose klog stdlib flags through Cobra/pflag.
+	cmd.PersistentFlags().AddGoFlagSet(flag.CommandLine)
+
 	flags.AddFlags(cmd.PersistentFlags())
+
 	cmd.AddCommand(
 		versioncmd.NewCmdVersion(),
 		initcmd.NewCmdInit(flags),
 		checkconfigcmd.NewCmdCheckConfig(),
-		createcmd.NewCmdCreate(),
+		createcmd.NewCmdCreate(flags),
 		agentcmd.NewCmdAgent(flags),
+		controllercmd.NewCmdController(flags),
 		internalcmd.NewCmdInternal(),
 	)
+
 	return cmd
 }

clitools/pkg/controller/osupgrade/handler.go

@@ -41,26 +41,62 @@ func (r *UpgradeRunner) Run(fn func() error) error {
 	return fn()
 }
 
-func HandleOSUpgrade(ctx context.Context, clients *kube.Clients,
+func HandleOSUpgradeProgress(
-	namespace string, nodeName string,
+	ctx context.Context,
-	osu *monov1alpha1.OSUpgrade,
+	clients *kube.Clients,
+	namespace string,
+	nodeName string,
+	osup *monov1alpha1.OSUpgradeProgress,
 ) error {
 	return r.Run(func() error {
-		return handleOSUpgradeLocked(ctx, clients, namespace, nodeName, osu)
+		return handleOSUpgradeProgressLocked(ctx, clients, namespace, nodeName, osup)
 	})
 }
 
-func handleOSUpgradeLocked(ctx context.Context, clients *kube.Clients,
+func handleOSUpgradeProgressLocked(
-	namespace string, nodeName string,
+	ctx context.Context,
-	osu *monov1alpha1.OSUpgrade,
+	clients *kube.Clients,
+	namespace string,
+	nodeName string,
+	osup *monov1alpha1.OSUpgradeProgress,
 ) error {
-	osup, err := ensureProgressHeartbeat(ctx, clients, namespace, nodeName, osu)
+	if osup == nil {
-	if err != nil {
+		return fmt.Errorf("osupgradeprogress is nil")
-		return err
 	}
 
-	klog.InfoS("handling osupgrade",
+	if osup.Spec.NodeName != nodeName {
-		"name", osu.Name,
+		klog.V(4).InfoS("skipping osupgradeprogress due to nodeName mismatch",
+			"name", osup.Name,
+			"node", nodeName,
+			"target", osup.Spec.NodeName,
+		)
+		return nil
+	}
+
+	if !shouldProcessProgress(osup) {
+		klog.V(2).InfoS("skipping osupgradeprogress due to phase",
+			"name", osup.Name,
+			"node", nodeName,
+			"phase", osup.StatusPhase(),
+		)
+		return nil
+	}
+
+	parentName := osup.Spec.SourceRef.Name
+	if parentName == "" {
+		return failProgress(ctx, clients, osup, "resolve parent osupgrade", fmt.Errorf("missing spec.osUpgradeName"))
+	}
+
+	osu, err := clients.MonoKS.Monok8sV1alpha1().
+		OSUpgrades(namespace).
+		Get(ctx, parentName, metav1.GetOptions{})
+	if err != nil {
+		return failProgress(ctx, clients, osup, "resolve parent osupgrade", err)
+	}
+
+	klog.InfoS("handling osupgradeprogress",
+		"name", osup.Name,
+		"osupgrade", osu.Name,
 		"node", nodeName,
 		"desiredVersion", osu.Spec.DesiredVersion,
 	)
@@ -95,7 +131,9 @@ func handleOSUpgradeLocked(ctx context.Context, clients *kube.Clients,
 		now := metav1.Now()
 		cur.Status.CurrentVersion = buildinfo.KubeVersion
 		cur.Status.TargetVersion = plan.ResolvedTarget
+		cur.Status.PlannedPath = plannedPath(plan)
 		cur.Status.Phase = monov1alpha1.OSUpgradeProgressPhaseDownloading
+		cur.Status.ObservedRetryNonce = cur.Spec.RetryNonce
 		cur.Status.Message = fmt.Sprintf("downloading image: %s", first.URL)
 		cur.Status.LastUpdatedAt = &now
 	})
@@ -209,6 +247,27 @@ func handleOSUpgradeLocked(ctx context.Context, clients *kube.Clients,
 	select {}
 }
 
+func shouldProcessProgress(osup *monov1alpha1.OSUpgradeProgress) bool {
+	if osup == nil {
+		return false
+	}
+
+	if osup.Status == nil {
+		return false
+	}
+
+	switch osup.Status.Phase {
+	case "",
+		monov1alpha1.OSUpgradeProgressPhasePending,
+		monov1alpha1.OSUpgradeProgressPhaseRebooting:
+		return true
+	case monov1alpha1.OSUpgradeProgressPhaseFailed:
+		return osup.Spec.RetryNonce != osup.Status.ObservedRetryNonce
+	default:
+		return false
+	}
+}
+
 func triggerReboot() error {
 	_ = os.WriteFile("/proc/sysrq-trigger", []byte("s\n"), 0)
 	_ = os.WriteFile("/proc/sysrq-trigger", []byte("u\n"), 0)

clitools/pkg/controller/osupgrade/planner.go

@@ -335,6 +335,14 @@ func lowestPatchInMinor(versions []Version, major, minor int) (Version, bool) {
 	return Version{}, false
 }
 
+func plannedPath(plan *Plan) []string {
+	ppath := []string{}
+	for _, img := range plan.Path {
+		ppath = append(ppath, img.Version)
+	}
+	return ppath
+}
+
 func versionsToStrings(vs []Version) []string {
 	out := make([]string, 0, len(vs))
 	for _, v := range vs {

clitools/pkg/controller/osupgrade/progress.go

@@ -7,40 +7,27 @@ import (
 
 	apierrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
-	"k8s.io/apimachinery/pkg/runtime"
-	"k8s.io/apimachinery/pkg/runtime/schema"
 	"k8s.io/client-go/util/retry"
 	"k8s.io/klog/v2"
 
 	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
-	"example.com/monok8s/pkg/buildinfo"
 	"example.com/monok8s/pkg/kube"
 )
 
-var (
+func EnsureOSUpgradeProgressForNode(
-	unstructuredConverter = runtime.DefaultUnstructuredConverter
+	ctx context.Context,
-
+	clients *kube.Clients,
-	osup_gvr = schema.GroupVersionResource{
+	namespace string,
-		Group:    monov1alpha1.Group,
+	nodeName string,
-		Version:  monov1alpha1.Version,
-		Resource: "osupgradeprogresses",
-	}
-)
-
-func ensureProgressHeartbeat(ctx context.Context, clients *kube.Clients,
-	namespace string, nodeName string,
 	osu *monov1alpha1.OSUpgrade,
-) (*monov1alpha1.OSUpgradeProgress, error) {
+) error {
+	if osu == nil {
+		return fmt.Errorf("osupgrade is nil")
+	}
+
 	name := fmt.Sprintf("%s-%s", osu.Name, nodeName)
 	now := metav1.Now()
 
-	currentVersion := buildinfo.KubeVersion
-	targetVersion := ""
-	if osu.Status != nil {
-		targetVersion = osu.Status.ResolvedVersion
-	}
-
 	progress := &monov1alpha1.OSUpgradeProgress{
 		TypeMeta: metav1.TypeMeta{
 			APIVersion: monov1alpha1.APIVersion,
@@ -57,83 +44,38 @@ func ensureProgressHeartbeat(ctx context.Context, clients *kube.Clients,
 			},
 		},
 		Status: &monov1alpha1.OSUpgradeProgressStatus{
-			CurrentVersion: currentVersion,
-			TargetVersion:  targetVersion,
 			Phase:         monov1alpha1.OSUpgradeProgressPhasePending,
 			LastUpdatedAt: &now,
-			Message:        "acknowledged",
 		},
 	}
 
-	created, err := createProgress(ctx, clients, osup_gvr, progress)
+	created, err := createProgress(ctx, clients, progress)
 	if err == nil {
 		klog.InfoS("created osupgradeprogress", "name", created.Name, "namespace", created.Namespace)
-		return created, nil
+		return nil
 	}
 	if !apierrors.IsAlreadyExists(err) {
-		return nil, fmt.Errorf("create OSUpgradeProgress %s/%s: %w", namespace, name, err)
+		return fmt.Errorf("create OSUpgradeProgress %s/%s: %w", namespace, name, err)
 	}
 
-	var out *monov1alpha1.OSUpgradeProgress
+	existing, err := getProgress(ctx, clients, namespace, name)
-	err = retry.RetryOnConflict(retry.DefaultRetry, func() error {
-		existing, err := getProgress(ctx, clients, osup_gvr, namespace, name)
 	if err != nil {
 		return fmt.Errorf("get existing OSUpgradeProgress %s/%s: %w", namespace, name, err)
 	}
 
-		// Keep spec aligned with source and node.
+	if existing.Spec.NodeName != nodeName || existing.Spec.SourceRef.Name != osu.Name {
-		existing.Spec.NodeName = nodeName
+		return fmt.Errorf(
-		existing.Spec.SourceRef.Name = osu.Name
+			"conflicting OSUpgradeProgress %s/%s already exists: got spec.nodeName=%q spec.sourceRef.name=%q, want nodeName=%q sourceRef.name=%q",
-
+			namespace,
-		existing, err = updateProgressSpec(ctx, clients, osup_gvr, existing)
+			name,
-		if err != nil {
+			existing.Spec.NodeName,
-			if isUnknownUpdateResult(err) {
+			existing.Spec.SourceRef.Name,
-				latest, getErr := getProgress(ctx, clients, osup_gvr, namespace, name)
+			nodeName,
-				if getErr == nil {
+			osu.Name,
-					out = latest
+		)
-				}
-			}
-			return fmt.Errorf("update OSUpgradeProgress spec %s/%s: %w", namespace, name, err)
 	}
 
-		if existing.Status == nil {
-			existing.Status = &monov1alpha1.OSUpgradeProgressStatus{}
-		}
-
-		existing.Status.CurrentVersion = currentVersion
-		existing.Status.TargetVersion = targetVersion
-		existing.Status.LastUpdatedAt = &now
-
-		if existing.Status.Phase == "" {
-			existing.Status.Phase = monov1alpha1.OSUpgradeProgressPhasePending
-		}
-		if existing.Status.Message == "" {
-			existing.Status.Message = "acknowledged"
-		}
-
-		existing, err = updateProgressStatus(ctx, clients, osup_gvr, existing)
-		if err != nil {
-			if isUnknownUpdateResult(err) {
-				latest, getErr := getProgress(ctx, clients, osup_gvr, namespace, name)
-				if getErr == nil {
-					out = latest
-				}
-			}
-			return fmt.Errorf("update OSUpgradeProgress status %s/%s: %w", namespace, name, err)
-		}
-
-		out = existing
 	return nil
-	})
-	if err != nil {
-		if out != nil {
-			return out, nil
-		}
-		return nil, err
-	}
-
-	klog.InfoS("updated osupgradeprogress", "name", out.Name, "namespace", out.Namespace)
-	return out, nil
 }
 
 func updateProgressRobust(
@@ -146,7 +88,7 @@ func updateProgressRobust(
 	var out *monov1alpha1.OSUpgradeProgress
 
 	err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
-		current, err := getProgress(ctx, clients, osup_gvr, namespace, name)
+		current, err := getProgress(ctx, clients, namespace, name)
 		if err != nil {
 			return err
 		}
@@ -157,10 +99,10 @@ func updateProgressRobust(
 
 		mutate(current)
 
-		updated, err := updateProgressStatus(ctx, clients, osup_gvr, current)
+		updated, err := updateProgressStatus(ctx, clients, current)
 		if err != nil {
 			if isUnknownUpdateResult(err) {
-				latest, getErr := getProgress(ctx, clients, osup_gvr, namespace, name)
+				latest, getErr := getProgress(ctx, clients, namespace, name)
 				if getErr == nil {
 					out = latest
 				}
@@ -202,84 +144,55 @@ func isUnknownUpdateResult(err error) bool {
 func createProgress(
 	ctx context.Context,
 	clients *kube.Clients,
-	gvr schema.GroupVersionResource,
 	progress *monov1alpha1.OSUpgradeProgress,
 ) (*monov1alpha1.OSUpgradeProgress, error) {
-	obj, err := toUnstructured(progress)
+	toCreate := progress.DeepCopy()
+	toCreate.Status = nil
+
+	created, err := clients.MonoKS.
+		Monok8sV1alpha1().
+		OSUpgradeProgresses(toCreate.Namespace).
+		Create(ctx, toCreate, metav1.CreateOptions{})
 	if err != nil {
 		return nil, err
 	}
 
-	created, err := clients.Dynamic.
+	if progress.Status != nil {
-		Resource(gvr).
+		toUpdate := created.DeepCopy()
-		Namespace(progress.Namespace).
+		toUpdate.Status = progress.Status
-		Create(ctx, obj, metav1.CreateOptions{})
+		return updateProgressStatus(ctx, clients, toUpdate)
-	if err != nil {
-		return nil, err
 	}
 
-	return fromUnstructuredProgress(created)
+	return created, nil
-}
-
-func getProgress(
-	ctx context.Context,
-	clients *kube.Clients,
-	gvr schema.GroupVersionResource,
-	namespace, name string,
-) (*monov1alpha1.OSUpgradeProgress, error) {
-	got, err := clients.Dynamic.
-		Resource(gvr).
-		Namespace(namespace).
-		Get(ctx, name, metav1.GetOptions{})
-	if err != nil {
-		return nil, err
-	}
-
-	return fromUnstructuredProgress(got)
-}
-
-func updateProgressSpec(
-	ctx context.Context,
-	clients *kube.Clients,
-	gvr schema.GroupVersionResource,
-	progress *monov1alpha1.OSUpgradeProgress,
-) (*monov1alpha1.OSUpgradeProgress, error) {
-	obj, err := toUnstructured(progress)
-	if err != nil {
-		return nil, err
-	}
-
-	updated, err := clients.Dynamic.
-		Resource(gvr).
-		Namespace(progress.Namespace).
-		Update(ctx, obj, metav1.UpdateOptions{})
-	if err != nil {
-		return nil, err
-	}
-
-	return fromUnstructuredProgress(updated)
 }
 
 func updateProgressStatus(
 	ctx context.Context,
 	clients *kube.Clients,
-	gvr schema.GroupVersionResource,
 	progress *monov1alpha1.OSUpgradeProgress,
 ) (*monov1alpha1.OSUpgradeProgress, error) {
-	obj, err := toUnstructured(progress)
+	updated, err := clients.MonoKS.
+		Monok8sV1alpha1().
+		OSUpgradeProgresses(progress.Namespace).
+		UpdateStatus(ctx, progress, metav1.UpdateOptions{})
 	if err != nil {
-		return nil, err
+		return nil, fmt.Errorf(
+			"update status for OSUpgradeProgress %s/%s: %w",
+			progress.Namespace, progress.Name, err,
+		)
 	}
+	return updated, nil
+}
 
-	updated, err := clients.Dynamic.
+func getProgress(
-		Resource(gvr).
+	ctx context.Context,
-		Namespace(progress.Namespace).
+	clients *kube.Clients,
-		UpdateStatus(ctx, obj, metav1.UpdateOptions{})
+	namespace, name string,
-	if err != nil {
+) (*monov1alpha1.OSUpgradeProgress, error) {
-		return nil, err
+	return clients.MonoKS.
-	}
+		Monok8sV1alpha1().
-
+		OSUpgradeProgresses(namespace).
-	return fromUnstructuredProgress(updated)
+		Get(ctx, name, metav1.GetOptions{})
 }
 
 func failProgress(
@@ -296,6 +209,7 @@ func failProgress(
 			cur.Status = &monov1alpha1.OSUpgradeProgressStatus{}
 		}
 
+		cur.Status.ObservedRetryNonce = cur.Spec.RetryNonce
 		cur.Status.LastUpdatedAt = &now
 		cur.Status.Message = fmt.Sprintf("%s: %v", action, cause)
 		cur.Status.Phase = monov1alpha1.OSUpgradeProgressPhaseFailed
@@ -324,6 +238,7 @@ func markProgressCompleted(
 			cur.Status = &monov1alpha1.OSUpgradeProgressStatus{}
 		}
 
+		cur.Status.ObservedRetryNonce = cur.Spec.RetryNonce
 		cur.Status.Phase = monov1alpha1.OSUpgradeProgressPhaseCompleted
 		cur.Status.Message = message
 		cur.Status.CurrentVersion = osup.Status.CurrentVersion
@@ -337,19 +252,3 @@ func markProgressCompleted(
 
 	return nil
 }
-
-func toUnstructured(progress *monov1alpha1.OSUpgradeProgress) (*unstructured.Unstructured, error) {
-	m, err := unstructuredConverter.ToUnstructured(progress)
-	if err != nil {
-		return nil, fmt.Errorf("convert OSUpgradeProgress to unstructured: %w", err)
-	}
-	return &unstructured.Unstructured{Object: m}, nil
-}
-
-func fromUnstructuredProgress(obj *unstructured.Unstructured) (*monov1alpha1.OSUpgradeProgress, error) {
-	var progress monov1alpha1.OSUpgradeProgress
-	if err := unstructuredConverter.FromUnstructured(obj.Object, &progress); err != nil {
-		return nil, fmt.Errorf("convert unstructured to OSUpgradeProgress: %w", err)
-	}
-	return &progress, nil
-}

clitools/pkg/controller/osupgrade/watch.go

@@ -0,0 +1,345 @@
+package osupgrade
+
+import (
+	"context"
+	"fmt"
+	"time"
+
+	corev1 "k8s.io/api/core/v1"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/labels"
+	"k8s.io/apimachinery/pkg/watch"
+	"k8s.io/klog/v2"
+
+	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	"example.com/monok8s/pkg/kube"
+)
+
+func Watch(ctx context.Context, clients *kube.Clients, namespace string) error {
+	var resourceVersion string
+
+	for {
+		if ctx.Err() != nil {
+			return ctx.Err()
+		}
+
+		err := watchOnce(ctx, clients, namespace, &resourceVersion)
+		if err != nil {
+			if ctx.Err() != nil {
+				return ctx.Err()
+			}
+
+			// Expired RV is normal enough; clear it and relist.
+			if apierrors.IsResourceExpired(err) {
+				klog.InfoS("OSUpgrade watch resourceVersion expired; resetting",
+					"namespace", namespace,
+					"resourceVersion", resourceVersion,
+				)
+				resourceVersion = ""
+			} else {
+				klog.ErrorS(err, "OSUpgrade watch failed; retrying",
+					"namespace", namespace,
+					"resourceVersion", resourceVersion,
+				)
+			}
+
+			select {
+			case <-ctx.Done():
+				return ctx.Err()
+			case <-time.After(2 * time.Second):
+			}
+			continue
+		}
+	}
+}
+
+func watchOnce(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace string,
+	resourceVersion *string,
+) error {
+	// Cold start: list existing objects once, handle them, then watch from list RV.
+	if *resourceVersion == "" {
+		list, err := clients.MonoKS.
+			Monok8sV1alpha1().
+			OSUpgrades(namespace).
+			List(ctx, metav1.ListOptions{})
+		if err != nil {
+			return fmt.Errorf("list OSUpgrades: %w", err)
+		}
+
+		for i := range list.Items {
+			osu := &list.Items[i]
+			handled, err := handleOSUpgrade(ctx, clients, namespace, osu)
+			if err != nil {
+				klog.ErrorS(err, "reconcile existing OSUpgrade failed",
+					"name", osu.Name,
+					"resourceVersion", osu.ResourceVersion,
+				)
+				continue
+			}
+			if !handled {
+				klog.V(2).InfoS("skipping existing OSUpgrade",
+					"name", osu.Name,
+					"phase", osu.StatusPhase(),
+				)
+			}
+		}
+
+		*resourceVersion = list.ResourceVersion
+		klog.InfoS("initial OSUpgrade sync complete",
+			"namespace", namespace,
+			"resourceVersion", *resourceVersion,
+			"count", len(list.Items),
+		)
+	}
+
+	w, err := clients.MonoKS.
+		Monok8sV1alpha1().
+		OSUpgrades(namespace).
+		Watch(ctx, metav1.ListOptions{
+			ResourceVersion:     *resourceVersion,
+			AllowWatchBookmarks: true,
+		})
+	if err != nil {
+		return fmt.Errorf("watch OSUpgrades: %w", err)
+	}
+	defer w.Stop()
+
+	klog.InfoS("watching OSUpgrades",
+		"namespace", namespace,
+		"resourceVersion", *resourceVersion,
+	)
+
+	for {
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+
+		case evt, ok := <-w.ResultChan():
+			if !ok {
+				return fmt.Errorf("watch channel closed")
+			}
+
+			switch evt.Type {
+			case watch.Bookmark:
+				if rv := extractResourceVersion(evt.Object); rv != "" {
+					*resourceVersion = rv
+				}
+				continue
+
+			case watch.Error:
+				// Let outer loop retry / relist.
+				return fmt.Errorf("watch returned error event")
+
+			case watch.Deleted:
+				// Top-level delete does not require action here.
+				continue
+
+			case watch.Added, watch.Modified:
+				// handled below
+
+			default:
+				klog.V(1).InfoS("skipping unexpected watch event type",
+					"eventType", evt.Type,
+				)
+				continue
+			}
+
+			osu, ok := evt.Object.(*monov1alpha1.OSUpgrade)
+			if !ok {
+				klog.V(1).InfoS("skipping unexpected watch object type",
+					"type", fmt.Sprintf("%T", evt.Object),
+				)
+				continue
+			}
+
+			if osu.ResourceVersion != "" {
+				*resourceVersion = osu.ResourceVersion
+			}
+
+			handled, err := handleOSUpgrade(ctx, clients, namespace, osu)
+			if err != nil {
+				klog.ErrorS(err, "reconcile OSUpgrade failed",
+					"name", osu.Name,
+					"eventType", evt.Type,
+					"resourceVersion", osu.ResourceVersion,
+				)
+				continue
+			}
+			if !handled {
+				klog.V(2).InfoS("skipping OSUpgrade",
+					"name", osu.Name,
+					"eventType", evt.Type,
+					"phase", osu.StatusPhase(),
+				)
+			}
+		}
+	}
+}
+
+func handleOSUpgrade(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace string,
+	osu *monov1alpha1.OSUpgrade,
+) (bool, error) {
+	if !shouldHandle(osu) {
+		return false, nil
+	}
+
+	if osu.Status == nil || osu.Status.ObservedGeneration != osu.Generation {
+		return true, reconcileSpec(ctx, clients, namespace, osu)
+	}
+
+	if osu.Status.Phase == monov1alpha1.OSUpgradePhaseAccepted {
+		return true, reconcileFanout(ctx, clients, namespace, osu)
+	}
+
+	return false, nil
+}
+
+func reconcileSpec(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace string,
+	osu *monov1alpha1.OSUpgrade,
+) error {
+	osu = osu.DeepCopy()
+
+	osu.Status = &monov1alpha1.OSUpgradeStatus{
+		Phase:              monov1alpha1.OSUpgradePhaseAccepted,
+		ResolvedVersion:    osu.Spec.DesiredVersion,
+		ObservedGeneration: osu.Generation,
+	}
+
+	_, err := clients.MonoKS.
+		Monok8sV1alpha1().
+		OSUpgrades(namespace).
+		UpdateStatus(ctx, osu, metav1.UpdateOptions{})
+
+	return err
+}
+
+func reconcileFanout(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace string,
+	osu *monov1alpha1.OSUpgrade,
+) error {
+	nodeNames, err := listTargetNodeNames(ctx, clients, osu)
+	if err != nil {
+		return fmt.Errorf("list target nodes for %s: %w", osu.Name, err)
+	}
+
+	if len(nodeNames) == 0 {
+		klog.InfoS("no targets", "osupgrade", osu.Name)
+		return nil
+	}
+
+	klog.InfoS("ensuring OSUpgradeProgress for target nodes",
+		"osupgrade", osu.Name,
+		"targets", len(nodeNames),
+	)
+
+	for _, nodeName := range nodeNames {
+		if err := EnsureOSUpgradeProgressForNode(
+			ctx,
+			clients,
+			namespace,
+			nodeName,
+			osu,
+		); err != nil {
+			klog.ErrorS(err, "ensure OSUpgradeProgress for node failed",
+				"osupgrade", osu.Name,
+				"node", nodeName,
+			)
+		}
+	}
+
+	return nil
+}
+
+func listTargetNodeNames(
+	ctx context.Context,
+	clients *kube.Clients,
+	osu *monov1alpha1.OSUpgrade,
+) ([]string, error) {
+	selector := labels.SelectorFromSet(labels.Set{
+		monov1alpha1.NodeControlKey: "true",
+	})
+
+	if osu.Spec.NodeSelector != nil {
+		userSelector, err := metav1.LabelSelectorAsSelector(osu.Spec.NodeSelector)
+		if err != nil {
+			return nil, fmt.Errorf("invalid nodeSelector: %w", err)
+		}
+
+		reqs, selectable := userSelector.Requirements()
+		if !selectable {
+			selector = labels.Nothing()
+		} else {
+			selector = selector.Add(reqs...)
+		}
+	}
+
+	list, err := clients.Kubernetes.CoreV1().
+		Nodes().
+		List(ctx, metav1.ListOptions{
+			LabelSelector: selector.String(),
+		})
+	if err != nil {
+		return nil, fmt.Errorf("list nodes: %w", err)
+	}
+
+	out := make([]string, 0, len(list.Items))
+	for i := range list.Items {
+		node := &list.Items[i]
+		if shouldUseNode(node) {
+			out = append(out, node.Name)
+		}
+	}
+
+	return out, nil
+}
+
+func shouldUseNode(node *corev1.Node) bool {
+	return node != nil && node.Name != ""
+}
+
+func shouldHandle(osu *monov1alpha1.OSUpgrade) bool {
+	if osu == nil || osu.DeletionTimestamp != nil {
+		return false
+	}
+
+	if osu.Spec.DesiredVersion == "" {
+		return false
+	}
+
+	// NEW: initial processing stage
+	if osu.Status == nil {
+		return true
+	}
+
+	// Reconcile if spec changed
+	if osu.Status.ObservedGeneration != osu.Generation {
+		return true
+	}
+
+	// Fanout stage
+	return osu.Status.Phase == monov1alpha1.OSUpgradePhaseAccepted
+}
+
+func extractResourceVersion(obj interface{}) string {
+	type hasRV interface {
+		GetResourceVersion() string
+	}
+
+	if o, ok := obj.(hasRV); ok {
+		return o.GetResourceVersion()
+	}
+
+	return ""
+}

clitools/pkg/controller/server.go

@@ -0,0 +1,92 @@
+package controller
+
+import (
+	"context"
+	"net/http"
+	"time"
+
+	"example.com/monok8s/pkg/kube"
+
+	"github.com/emicklei/go-restful/v3"
+	"k8s.io/apiserver/pkg/server/httplog"
+)
+
+var statusesNoTracePred = httplog.StatusIsNot(
+	http.StatusOK,
+	http.StatusFound,
+	http.StatusMovedPermanently,
+	http.StatusTemporaryRedirect,
+	http.StatusBadRequest,
+	http.StatusNotFound,
+	http.StatusSwitchingProtocols,
+)
+
+type Server struct {
+	restfulCont *restful.Container
+
+	ctx       context.Context
+	clients   *kube.Clients
+	namespace string
+	nodeName  string
+	startedAt time.Time
+}
+
+type StatusResponse struct {
+	OK        bool   `json:"ok"`
+	Service   string `json:"service"`
+	Namespace string `json:"namespace,omitempty"`
+	NodeName  string `json:"nodeName,omitempty"`
+	UptimeSec int64  `json:"uptimeSec"`
+}
+
+func NewServer(ctx context.Context, clients *kube.Clients, namespace, nodeName string) *Server {
+	s := &Server{
+		ctx:       ctx,
+		clients:   clients,
+		namespace: namespace,
+		nodeName:  nodeName,
+		startedAt: time.Now(),
+	}
+	s.Initialize()
+	return s
+}
+
+func (s *Server) ServeHTTP(w http.ResponseWriter, req *http.Request) {
+	if s == nil {
+		http.Error(w, "server is nil", http.StatusInternalServerError)
+		return
+	}
+	if s.restfulCont == nil {
+		http.Error(w, "server not initialized", http.StatusInternalServerError)
+		return
+	}
+
+	handler := httplog.WithLogging(s.restfulCont, statusesNoTracePred)
+	handler.ServeHTTP(w, req)
+}
+
+func (s *Server) Initialize() {
+	s.restfulCont = restful.NewContainer()
+
+	ws := new(restful.WebService)
+	ws.Path("/")
+	ws.Consumes(restful.MIME_JSON)
+	ws.Produces(restful.MIME_JSON)
+
+	ws.Route(ws.GET("/status").To(s.queryStatus).
+		Doc("Return basic controller status"))
+
+	s.restfulCont.Add(ws)
+}
+
+func (s *Server) queryStatus(request *restful.Request, response *restful.Response) {
+	resp := StatusResponse{
+		OK:        true,
+		Service:   "monok8s-controller",
+		Namespace: s.namespace,
+		NodeName:  s.nodeName,
+		UptimeSec: int64(time.Since(s.startedAt).Seconds()),
+	}
+
+	_ = response.WriteHeaderAndEntity(http.StatusOK, resp)
+}

clitools/pkg/crds/crds.go

@@ -1,76 +0,0 @@
-package crds
-
-import (
-	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
-	apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-)
-
-func Definitions() []*apiextensionsv1.CustomResourceDefinition {
-	return []*apiextensionsv1.CustomResourceDefinition{
-		monoKSConfigCRD(),
-		osUpgradeCRD(),
-	}
-}
-
-func monoKSConfigCRD() *apiextensionsv1.CustomResourceDefinition {
-	return &apiextensionsv1.CustomResourceDefinition{
-		ObjectMeta: metav1.ObjectMeta{
-			Name: monov1alpha1.MonoKSConfigCRD,
-		},
-		Spec: apiextensionsv1.CustomResourceDefinitionSpec{
-			Group: monov1alpha1.Group,
-			Scope: apiextensionsv1.NamespaceScoped,
-			Names: apiextensionsv1.CustomResourceDefinitionNames{
-				Plural:     "monoksconfigs",
-				Singular:   "monoksconfig",
-				Kind:       "MonoKSConfig",
-				ShortNames: []string{"mkscfg"},
-			},
-			Versions: []apiextensionsv1.CustomResourceDefinitionVersion{{
-				Name:    "v1alpha1",
-				Served:  true,
-				Storage: true,
-				Schema: &apiextensionsv1.CustomResourceValidation{OpenAPIV3Schema: &apiextensionsv1.JSONSchemaProps{
-					Type: "object",
-					Properties: map[string]apiextensionsv1.JSONSchemaProps{
-						"spec":   {Type: "object", XPreserveUnknownFields: boolPtr(true)},
-						"status": {Type: "object", XPreserveUnknownFields: boolPtr(true)},
-					},
-				}},
-			}},
-		},
-	}
-}
-
-func osUpgradeCRD() *apiextensionsv1.CustomResourceDefinition {
-	return &apiextensionsv1.CustomResourceDefinition{
-		ObjectMeta: metav1.ObjectMeta{
-			Name: monov1alpha1.OSUpgradeCRD,
-		},
-		Spec: apiextensionsv1.CustomResourceDefinitionSpec{
-			Group: monov1alpha1.Group,
-			Scope: apiextensionsv1.NamespaceScoped,
-			Names: apiextensionsv1.CustomResourceDefinitionNames{
-				Plural:     "osupgrades",
-				Singular:   "osupgrade",
-				Kind:       "OSUpgrade",
-				ShortNames: []string{"osup"},
-			},
-			Versions: []apiextensionsv1.CustomResourceDefinitionVersion{{
-				Name:    "v1alpha1",
-				Served:  true,
-				Storage: true,
-				Schema: &apiextensionsv1.CustomResourceValidation{OpenAPIV3Schema: &apiextensionsv1.JSONSchemaProps{
-					Type: "object",
-					Properties: map[string]apiextensionsv1.JSONSchemaProps{
-						"spec":   {Type: "object", XPreserveUnknownFields: boolPtr(true)},
-						"status": {Type: "object", XPreserveUnknownFields: boolPtr(true)},
-					},
-				}},
-			}},
-		},
-	}
-}
-
-func boolPtr(v bool) *bool { return &v }

clitools/pkg/generated/clientset/versioned/clientset.go

@@ -0,0 +1,106 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package versioned
+
+import (
+	fmt "fmt"
+	http "net/http"
+
+	monok8sv1alpha1 "example.com/monok8s/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1"
+	discovery "k8s.io/client-go/discovery"
+	rest "k8s.io/client-go/rest"
+	flowcontrol "k8s.io/client-go/util/flowcontrol"
+)
+
+type Interface interface {
+	Discovery() discovery.DiscoveryInterface
+	Monok8sV1alpha1() monok8sv1alpha1.Monok8sV1alpha1Interface
+}
+
+// Clientset contains the clients for groups.
+type Clientset struct {
+	*discovery.DiscoveryClient
+	monok8sV1alpha1 *monok8sv1alpha1.Monok8sV1alpha1Client
+}
+
+// Monok8sV1alpha1 retrieves the Monok8sV1alpha1Client
+func (c *Clientset) Monok8sV1alpha1() monok8sv1alpha1.Monok8sV1alpha1Interface {
+	return c.monok8sV1alpha1
+}
+
+// Discovery retrieves the DiscoveryClient
+func (c *Clientset) Discovery() discovery.DiscoveryInterface {
+	if c == nil {
+		return nil
+	}
+	return c.DiscoveryClient
+}
+
+// NewForConfig creates a new Clientset for the given config.
+// If config's RateLimiter is not set and QPS and Burst are acceptable,
+// NewForConfig will generate a rate-limiter in configShallowCopy.
+// NewForConfig is equivalent to NewForConfigAndClient(c, httpClient),
+// where httpClient was generated with rest.HTTPClientFor(c).
+func NewForConfig(c *rest.Config) (*Clientset, error) {
+	configShallowCopy := *c
+
+	if configShallowCopy.UserAgent == "" {
+		configShallowCopy.UserAgent = rest.DefaultKubernetesUserAgent()
+	}
+
+	// share the transport between all clients
+	httpClient, err := rest.HTTPClientFor(&configShallowCopy)
+	if err != nil {
+		return nil, err
+	}
+
+	return NewForConfigAndClient(&configShallowCopy, httpClient)
+}
+
+// NewForConfigAndClient creates a new Clientset for the given config and http client.
+// Note the http client provided takes precedence over the configured transport values.
+// If config's RateLimiter is not set and QPS and Burst are acceptable,
+// NewForConfigAndClient will generate a rate-limiter in configShallowCopy.
+func NewForConfigAndClient(c *rest.Config, httpClient *http.Client) (*Clientset, error) {
+	configShallowCopy := *c
+	if configShallowCopy.RateLimiter == nil && configShallowCopy.QPS > 0 {
+		if configShallowCopy.Burst <= 0 {
+			return nil, fmt.Errorf("burst is required to be greater than 0 when RateLimiter is not set and QPS is set to greater than 0")
+		}
+		configShallowCopy.RateLimiter = flowcontrol.NewTokenBucketRateLimiter(configShallowCopy.QPS, configShallowCopy.Burst)
+	}
+
+	var cs Clientset
+	var err error
+	cs.monok8sV1alpha1, err = monok8sv1alpha1.NewForConfigAndClient(&configShallowCopy, httpClient)
+	if err != nil {
+		return nil, err
+	}
+
+	cs.DiscoveryClient, err = discovery.NewDiscoveryClientForConfigAndClient(&configShallowCopy, httpClient)
+	if err != nil {
+		return nil, err
+	}
+	return &cs, nil
+}
+
+// NewForConfigOrDie creates a new Clientset for the given config and
+// panics if there is an error in the config.
+func NewForConfigOrDie(c *rest.Config) *Clientset {
+	cs, err := NewForConfig(c)
+	if err != nil {
+		panic(err)
+	}
+	return cs
+}
+
+// New creates a new Clientset for the given RESTClient.
+func New(c rest.Interface) *Clientset {
+	var cs Clientset
+	cs.monok8sV1alpha1 = monok8sv1alpha1.New(c)
+
+	cs.DiscoveryClient = discovery.NewDiscoveryClient(c)
+	return &cs
+}

clitools/pkg/generated/clientset/versioned/fake/clientset_generated.go

@@ -0,0 +1,91 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package fake
+
+import (
+	clientset "example.com/monok8s/pkg/generated/clientset/versioned"
+	monok8sv1alpha1 "example.com/monok8s/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1"
+	fakemonok8sv1alpha1 "example.com/monok8s/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/fake"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/watch"
+	"k8s.io/client-go/discovery"
+	fakediscovery "k8s.io/client-go/discovery/fake"
+	"k8s.io/client-go/testing"
+)
+
+// NewSimpleClientset returns a clientset that will respond with the provided objects.
+// It's backed by a very simple object tracker that processes creates, updates and deletions as-is,
+// without applying any field management, validations and/or defaults. It shouldn't be considered a replacement
+// for a real clientset and is mostly useful in simple unit tests.
+//
+// Deprecated: NewClientset replaces this with support for field management, which significantly improves
+// server side apply testing. NewClientset is only available when apply configurations are generated (e.g.
+// via --with-applyconfig).
+func NewSimpleClientset(objects ...runtime.Object) *Clientset {
+	o := testing.NewObjectTracker(scheme, codecs.UniversalDecoder())
+	for _, obj := range objects {
+		if err := o.Add(obj); err != nil {
+			panic(err)
+		}
+	}
+
+	cs := &Clientset{tracker: o}
+	cs.discovery = &fakediscovery.FakeDiscovery{Fake: &cs.Fake}
+	cs.AddReactor("*", "*", testing.ObjectReaction(o))
+	cs.AddWatchReactor("*", func(action testing.Action) (handled bool, ret watch.Interface, err error) {
+		var opts metav1.ListOptions
+		if watchAction, ok := action.(testing.WatchActionImpl); ok {
+			opts = watchAction.ListOptions
+		}
+		gvr := action.GetResource()
+		ns := action.GetNamespace()
+		watch, err := o.Watch(gvr, ns, opts)
+		if err != nil {
+			return false, nil, err
+		}
+		return true, watch, nil
+	})
+
+	return cs
+}
+
+// Clientset implements clientset.Interface. Meant to be embedded into a
+// struct to get a default implementation. This makes faking out just the method
+// you want to test easier.
+type Clientset struct {
+	testing.Fake
+	discovery *fakediscovery.FakeDiscovery
+	tracker   testing.ObjectTracker
+}
+
+func (c *Clientset) Discovery() discovery.DiscoveryInterface {
+	return c.discovery
+}
+
+func (c *Clientset) Tracker() testing.ObjectTracker {
+	return c.tracker
+}
+
+// IsWatchListSemanticsSupported informs the reflector that this client
+// doesn't support WatchList semantics.
+//
+// This is a synthetic method whose sole purpose is to satisfy the optional
+// interface check performed by the reflector.
+// Returning true signals that WatchList can NOT be used.
+// No additional logic is implemented here.
+func (c *Clientset) IsWatchListSemanticsUnSupported() bool {
+	return true
+}
+
+var (
+	_ clientset.Interface = &Clientset{}
+	_ testing.FakeClient  = &Clientset{}
+)
+
+// Monok8sV1alpha1 retrieves the Monok8sV1alpha1Client
+func (c *Clientset) Monok8sV1alpha1() monok8sv1alpha1.Monok8sV1alpha1Interface {
+	return &fakemonok8sv1alpha1.FakeMonok8sV1alpha1{Fake: &c.Fake}
+}

clitools/pkg/generated/clientset/versioned/fake/doc.go

@@ -0,0 +1,6 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+// This package has the automatically generated fake clientset.
+package fake

clitools/pkg/generated/clientset/versioned/fake/register.go

@@ -0,0 +1,42 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package fake
+
+import (
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+	schema "k8s.io/apimachinery/pkg/runtime/schema"
+	serializer "k8s.io/apimachinery/pkg/runtime/serializer"
+	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
+)
+
+var scheme = runtime.NewScheme()
+var codecs = serializer.NewCodecFactory(scheme)
+
+var localSchemeBuilder = runtime.SchemeBuilder{
+	monok8sv1alpha1.AddToScheme,
+}
+
+// AddToScheme adds all types of this clientset into the given scheme. This allows composition
+// of clientsets, like in:
+//
+//	import (
+//	  "k8s.io/client-go/kubernetes"
+//	  clientsetscheme "k8s.io/client-go/kubernetes/scheme"
+//	  aggregatorclientsetscheme "k8s.io/kube-aggregator/pkg/client/clientset_generated/clientset/scheme"
+//	)
+//
+//	kclientset, _ := kubernetes.NewForConfig(c)
+//	_ = aggregatorclientsetscheme.AddToScheme(clientsetscheme.Scheme)
+//
+// After this, RawExtensions in Kubernetes types will serialize kube-aggregator types
+// correctly.
+var AddToScheme = localSchemeBuilder.AddToScheme
+
+func init() {
+	v1.AddToGroupVersion(scheme, schema.GroupVersion{Version: "v1"})
+	utilruntime.Must(AddToScheme(scheme))
+}

clitools/pkg/generated/clientset/versioned/scheme/doc.go

@@ -0,0 +1,6 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+// This package contains the scheme of the automatically generated clientset.
+package scheme

clitools/pkg/generated/clientset/versioned/scheme/register.go

@@ -0,0 +1,42 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package scheme
+
+import (
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+	schema "k8s.io/apimachinery/pkg/runtime/schema"
+	serializer "k8s.io/apimachinery/pkg/runtime/serializer"
+	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
+)
+
+var Scheme = runtime.NewScheme()
+var Codecs = serializer.NewCodecFactory(Scheme)
+var ParameterCodec = runtime.NewParameterCodec(Scheme)
+var localSchemeBuilder = runtime.SchemeBuilder{
+	monok8sv1alpha1.AddToScheme,
+}
+
+// AddToScheme adds all types of this clientset into the given scheme. This allows composition
+// of clientsets, like in:
+//
+//	import (
+//	  "k8s.io/client-go/kubernetes"
+//	  clientsetscheme "k8s.io/client-go/kubernetes/scheme"
+//	  aggregatorclientsetscheme "k8s.io/kube-aggregator/pkg/client/clientset_generated/clientset/scheme"
+//	)
+//
+//	kclientset, _ := kubernetes.NewForConfig(c)
+//	_ = aggregatorclientsetscheme.AddToScheme(clientsetscheme.Scheme)
+//
+// After this, RawExtensions in Kubernetes types will serialize kube-aggregator types
+// correctly.
+var AddToScheme = localSchemeBuilder.AddToScheme
+
+func init() {
+	v1.AddToGroupVersion(Scheme, schema.GroupVersion{Version: "v1"})
+	utilruntime.Must(AddToScheme(Scheme))
+}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/doc.go

@@ -0,0 +1,6 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+// This package has the automatically generated typed clients.
+package v1alpha1

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/fake/doc.go

@@ -0,0 +1,6 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+// Package fake has the automatically generated clients.
+package fake

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/fake/fake_monok8s_client.go

@@ -0,0 +1,30 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package fake
+
+import (
+	v1alpha1 "example.com/monok8s/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1"
+	rest "k8s.io/client-go/rest"
+	testing "k8s.io/client-go/testing"
+)
+
+type FakeMonok8sV1alpha1 struct {
+	*testing.Fake
+}
+
+func (c *FakeMonok8sV1alpha1) OSUpgrades(namespace string) v1alpha1.OSUpgradeInterface {
+	return newFakeOSUpgrades(c, namespace)
+}
+
+func (c *FakeMonok8sV1alpha1) OSUpgradeProgresses(namespace string) v1alpha1.OSUpgradeProgressInterface {
+	return newFakeOSUpgradeProgresses(c, namespace)
+}
+
+// RESTClient returns a RESTClient that is used to communicate
+// with API server by this client implementation.
+func (c *FakeMonok8sV1alpha1) RESTClient() rest.Interface {
+	var ret *rest.RESTClient
+	return ret
+}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/fake/fake_osupgrade.go

@@ -0,0 +1,36 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package fake
+
+import (
+	v1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	monok8sv1alpha1 "example.com/monok8s/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1"
+	gentype "k8s.io/client-go/gentype"
+)
+
+// fakeOSUpgrades implements OSUpgradeInterface
+type fakeOSUpgrades struct {
+	*gentype.FakeClientWithList[*v1alpha1.OSUpgrade, *v1alpha1.OSUpgradeList]
+	Fake *FakeMonok8sV1alpha1
+}
+
+func newFakeOSUpgrades(fake *FakeMonok8sV1alpha1, namespace string) monok8sv1alpha1.OSUpgradeInterface {
+	return &fakeOSUpgrades{
+		gentype.NewFakeClientWithList[*v1alpha1.OSUpgrade, *v1alpha1.OSUpgradeList](
+			fake.Fake,
+			namespace,
+			v1alpha1.SchemeGroupVersion.WithResource("osupgrades"),
+			v1alpha1.SchemeGroupVersion.WithKind("OSUpgrade"),
+			func() *v1alpha1.OSUpgrade { return &v1alpha1.OSUpgrade{} },
+			func() *v1alpha1.OSUpgradeList { return &v1alpha1.OSUpgradeList{} },
+			func(dst, src *v1alpha1.OSUpgradeList) { dst.ListMeta = src.ListMeta },
+			func(list *v1alpha1.OSUpgradeList) []*v1alpha1.OSUpgrade { return gentype.ToPointerSlice(list.Items) },
+			func(list *v1alpha1.OSUpgradeList, items []*v1alpha1.OSUpgrade) {
+				list.Items = gentype.FromPointerSlice(items)
+			},
+		),
+		fake,
+	}
+}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/fake/fake_osupgradeprogress.go

@@ -0,0 +1,38 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package fake
+
+import (
+	v1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	monok8sv1alpha1 "example.com/monok8s/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1"
+	gentype "k8s.io/client-go/gentype"
+)
+
+// fakeOSUpgradeProgresses implements OSUpgradeProgressInterface
+type fakeOSUpgradeProgresses struct {
+	*gentype.FakeClientWithList[*v1alpha1.OSUpgradeProgress, *v1alpha1.OSUpgradeProgressList]
+	Fake *FakeMonok8sV1alpha1
+}
+
+func newFakeOSUpgradeProgresses(fake *FakeMonok8sV1alpha1, namespace string) monok8sv1alpha1.OSUpgradeProgressInterface {
+	return &fakeOSUpgradeProgresses{
+		gentype.NewFakeClientWithList[*v1alpha1.OSUpgradeProgress, *v1alpha1.OSUpgradeProgressList](
+			fake.Fake,
+			namespace,
+			v1alpha1.SchemeGroupVersion.WithResource("osupgradeprogresses"),
+			v1alpha1.SchemeGroupVersion.WithKind("OSUpgradeProgress"),
+			func() *v1alpha1.OSUpgradeProgress { return &v1alpha1.OSUpgradeProgress{} },
+			func() *v1alpha1.OSUpgradeProgressList { return &v1alpha1.OSUpgradeProgressList{} },
+			func(dst, src *v1alpha1.OSUpgradeProgressList) { dst.ListMeta = src.ListMeta },
+			func(list *v1alpha1.OSUpgradeProgressList) []*v1alpha1.OSUpgradeProgress {
+				return gentype.ToPointerSlice(list.Items)
+			},
+			func(list *v1alpha1.OSUpgradeProgressList, items []*v1alpha1.OSUpgradeProgress) {
+				list.Items = gentype.FromPointerSlice(items)
+			},
+		),
+		fake,
+	}
+}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/generated_expansion.go

@@ -0,0 +1,9 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package v1alpha1
+
+type OSUpgradeExpansion interface{}
+
+type OSUpgradeProgressExpansion interface{}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/monok8s_client.go

@@ -0,0 +1,92 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	http "net/http"
+
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	scheme "example.com/monok8s/pkg/generated/clientset/versioned/scheme"
+	rest "k8s.io/client-go/rest"
+)
+
+type Monok8sV1alpha1Interface interface {
+	RESTClient() rest.Interface
+	OSUpgradesGetter
+	OSUpgradeProgressesGetter
+}
+
+// Monok8sV1alpha1Client is used to interact with features provided by the monok8s group.
+type Monok8sV1alpha1Client struct {
+	restClient rest.Interface
+}
+
+func (c *Monok8sV1alpha1Client) OSUpgrades(namespace string) OSUpgradeInterface {
+	return newOSUpgrades(c, namespace)
+}
+
+func (c *Monok8sV1alpha1Client) OSUpgradeProgresses(namespace string) OSUpgradeProgressInterface {
+	return newOSUpgradeProgresses(c, namespace)
+}
+
+// NewForConfig creates a new Monok8sV1alpha1Client for the given config.
+// NewForConfig is equivalent to NewForConfigAndClient(c, httpClient),
+// where httpClient was generated with rest.HTTPClientFor(c).
+func NewForConfig(c *rest.Config) (*Monok8sV1alpha1Client, error) {
+	config := *c
+	setConfigDefaults(&config)
+	httpClient, err := rest.HTTPClientFor(&config)
+	if err != nil {
+		return nil, err
+	}
+	return NewForConfigAndClient(&config, httpClient)
+}
+
+// NewForConfigAndClient creates a new Monok8sV1alpha1Client for the given config and http client.
+// Note the http client provided takes precedence over the configured transport values.
+func NewForConfigAndClient(c *rest.Config, h *http.Client) (*Monok8sV1alpha1Client, error) {
+	config := *c
+	setConfigDefaults(&config)
+	client, err := rest.RESTClientForConfigAndClient(&config, h)
+	if err != nil {
+		return nil, err
+	}
+	return &Monok8sV1alpha1Client{client}, nil
+}
+
+// NewForConfigOrDie creates a new Monok8sV1alpha1Client for the given config and
+// panics if there is an error in the config.
+func NewForConfigOrDie(c *rest.Config) *Monok8sV1alpha1Client {
+	client, err := NewForConfig(c)
+	if err != nil {
+		panic(err)
+	}
+	return client
+}
+
+// New creates a new Monok8sV1alpha1Client for the given RESTClient.
+func New(c rest.Interface) *Monok8sV1alpha1Client {
+	return &Monok8sV1alpha1Client{c}
+}
+
+func setConfigDefaults(config *rest.Config) {
+	gv := monok8sv1alpha1.SchemeGroupVersion
+	config.GroupVersion = &gv
+	config.APIPath = "/apis"
+	config.NegotiatedSerializer = rest.CodecFactoryForGeneratedClient(scheme.Scheme, scheme.Codecs).WithoutConversion()
+
+	if config.UserAgent == "" {
+		config.UserAgent = rest.DefaultKubernetesUserAgent()
+	}
+}
+
+// RESTClient returns a RESTClient that is used to communicate
+// with API server by this client implementation.
+func (c *Monok8sV1alpha1Client) RESTClient() rest.Interface {
+	if c == nil {
+		return nil
+	}
+	return c.restClient
+}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/osupgrade.go

@@ -0,0 +1,56 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	context "context"
+
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	scheme "example.com/monok8s/pkg/generated/clientset/versioned/scheme"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	types "k8s.io/apimachinery/pkg/types"
+	watch "k8s.io/apimachinery/pkg/watch"
+	gentype "k8s.io/client-go/gentype"
+)
+
+// OSUpgradesGetter has a method to return a OSUpgradeInterface.
+// A group's client should implement this interface.
+type OSUpgradesGetter interface {
+	OSUpgrades(namespace string) OSUpgradeInterface
+}
+
+// OSUpgradeInterface has methods to work with OSUpgrade resources.
+type OSUpgradeInterface interface {
+	Create(ctx context.Context, oSUpgrade *monok8sv1alpha1.OSUpgrade, opts v1.CreateOptions) (*monok8sv1alpha1.OSUpgrade, error)
+	Update(ctx context.Context, oSUpgrade *monok8sv1alpha1.OSUpgrade, opts v1.UpdateOptions) (*monok8sv1alpha1.OSUpgrade, error)
+	// Add a +genclient:noStatus comment above the type to avoid generating UpdateStatus().
+	UpdateStatus(ctx context.Context, oSUpgrade *monok8sv1alpha1.OSUpgrade, opts v1.UpdateOptions) (*monok8sv1alpha1.OSUpgrade, error)
+	Delete(ctx context.Context, name string, opts v1.DeleteOptions) error
+	DeleteCollection(ctx context.Context, opts v1.DeleteOptions, listOpts v1.ListOptions) error
+	Get(ctx context.Context, name string, opts v1.GetOptions) (*monok8sv1alpha1.OSUpgrade, error)
+	List(ctx context.Context, opts v1.ListOptions) (*monok8sv1alpha1.OSUpgradeList, error)
+	Watch(ctx context.Context, opts v1.ListOptions) (watch.Interface, error)
+	Patch(ctx context.Context, name string, pt types.PatchType, data []byte, opts v1.PatchOptions, subresources ...string) (result *monok8sv1alpha1.OSUpgrade, err error)
+	OSUpgradeExpansion
+}
+
+// oSUpgrades implements OSUpgradeInterface
+type oSUpgrades struct {
+	*gentype.ClientWithList[*monok8sv1alpha1.OSUpgrade, *monok8sv1alpha1.OSUpgradeList]
+}
+
+// newOSUpgrades returns a OSUpgrades
+func newOSUpgrades(c *Monok8sV1alpha1Client, namespace string) *oSUpgrades {
+	return &oSUpgrades{
+		gentype.NewClientWithList[*monok8sv1alpha1.OSUpgrade, *monok8sv1alpha1.OSUpgradeList](
+			"osupgrades",
+			c.RESTClient(),
+			scheme.ParameterCodec,
+			namespace,
+			func() *monok8sv1alpha1.OSUpgrade { return &monok8sv1alpha1.OSUpgrade{} },
+			func() *monok8sv1alpha1.OSUpgradeList { return &monok8sv1alpha1.OSUpgradeList{} },
+		),
+	}
+}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/osupgradeprogress.go

@@ -0,0 +1,56 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	context "context"
+
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	scheme "example.com/monok8s/pkg/generated/clientset/versioned/scheme"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	types "k8s.io/apimachinery/pkg/types"
+	watch "k8s.io/apimachinery/pkg/watch"
+	gentype "k8s.io/client-go/gentype"
+)
+
+// OSUpgradeProgressesGetter has a method to return a OSUpgradeProgressInterface.
+// A group's client should implement this interface.
+type OSUpgradeProgressesGetter interface {
+	OSUpgradeProgresses(namespace string) OSUpgradeProgressInterface
+}
+
+// OSUpgradeProgressInterface has methods to work with OSUpgradeProgress resources.
+type OSUpgradeProgressInterface interface {
+	Create(ctx context.Context, oSUpgradeProgress *monok8sv1alpha1.OSUpgradeProgress, opts v1.CreateOptions) (*monok8sv1alpha1.OSUpgradeProgress, error)
+	Update(ctx context.Context, oSUpgradeProgress *monok8sv1alpha1.OSUpgradeProgress, opts v1.UpdateOptions) (*monok8sv1alpha1.OSUpgradeProgress, error)
+	// Add a +genclient:noStatus comment above the type to avoid generating UpdateStatus().
+	UpdateStatus(ctx context.Context, oSUpgradeProgress *monok8sv1alpha1.OSUpgradeProgress, opts v1.UpdateOptions) (*monok8sv1alpha1.OSUpgradeProgress, error)
+	Delete(ctx context.Context, name string, opts v1.DeleteOptions) error
+	DeleteCollection(ctx context.Context, opts v1.DeleteOptions, listOpts v1.ListOptions) error
+	Get(ctx context.Context, name string, opts v1.GetOptions) (*monok8sv1alpha1.OSUpgradeProgress, error)
+	List(ctx context.Context, opts v1.ListOptions) (*monok8sv1alpha1.OSUpgradeProgressList, error)
+	Watch(ctx context.Context, opts v1.ListOptions) (watch.Interface, error)
+	Patch(ctx context.Context, name string, pt types.PatchType, data []byte, opts v1.PatchOptions, subresources ...string) (result *monok8sv1alpha1.OSUpgradeProgress, err error)
+	OSUpgradeProgressExpansion
+}
+
+// oSUpgradeProgresses implements OSUpgradeProgressInterface
+type oSUpgradeProgresses struct {
+	*gentype.ClientWithList[*monok8sv1alpha1.OSUpgradeProgress, *monok8sv1alpha1.OSUpgradeProgressList]
+}
+
+// newOSUpgradeProgresses returns a OSUpgradeProgresses
+func newOSUpgradeProgresses(c *Monok8sV1alpha1Client, namespace string) *oSUpgradeProgresses {
+	return &oSUpgradeProgresses{
+		gentype.NewClientWithList[*monok8sv1alpha1.OSUpgradeProgress, *monok8sv1alpha1.OSUpgradeProgressList](
+			"osupgradeprogresses",
+			c.RESTClient(),
+			scheme.ParameterCodec,
+			namespace,
+			func() *monok8sv1alpha1.OSUpgradeProgress { return &monok8sv1alpha1.OSUpgradeProgress{} },
+			func() *monok8sv1alpha1.OSUpgradeProgressList { return &monok8sv1alpha1.OSUpgradeProgressList{} },
+		),
+	}
+}

clitools/pkg/generated/informers/externalversions/factory.go

@@ -0,0 +1,249 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package externalversions
+
+import (
+	reflect "reflect"
+	sync "sync"
+	time "time"
+
+	versioned "example.com/monok8s/pkg/generated/clientset/versioned"
+	internalinterfaces "example.com/monok8s/pkg/generated/informers/externalversions/internalinterfaces"
+	monok8s "example.com/monok8s/pkg/generated/informers/externalversions/monok8s"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+	schema "k8s.io/apimachinery/pkg/runtime/schema"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// SharedInformerOption defines the functional option type for SharedInformerFactory.
+type SharedInformerOption func(*sharedInformerFactory) *sharedInformerFactory
+
+type sharedInformerFactory struct {
+	client           versioned.Interface
+	namespace        string
+	tweakListOptions internalinterfaces.TweakListOptionsFunc
+	lock             sync.Mutex
+	defaultResync    time.Duration
+	customResync     map[reflect.Type]time.Duration
+	transform        cache.TransformFunc
+
+	informers map[reflect.Type]cache.SharedIndexInformer
+	// startedInformers is used for tracking which informers have been started.
+	// This allows Start() to be called multiple times safely.
+	startedInformers map[reflect.Type]bool
+	// wg tracks how many goroutines were started.
+	wg sync.WaitGroup
+	// shuttingDown is true when Shutdown has been called. It may still be running
+	// because it needs to wait for goroutines.
+	shuttingDown bool
+}
+
+// WithCustomResyncConfig sets a custom resync period for the specified informer types.
+func WithCustomResyncConfig(resyncConfig map[v1.Object]time.Duration) SharedInformerOption {
+	return func(factory *sharedInformerFactory) *sharedInformerFactory {
+		for k, v := range resyncConfig {
+			factory.customResync[reflect.TypeOf(k)] = v
+		}
+		return factory
+	}
+}
+
+// WithTweakListOptions sets a custom filter on all listers of the configured SharedInformerFactory.
+func WithTweakListOptions(tweakListOptions internalinterfaces.TweakListOptionsFunc) SharedInformerOption {
+	return func(factory *sharedInformerFactory) *sharedInformerFactory {
+		factory.tweakListOptions = tweakListOptions
+		return factory
+	}
+}
+
+// WithNamespace limits the SharedInformerFactory to the specified namespace.
+func WithNamespace(namespace string) SharedInformerOption {
+	return func(factory *sharedInformerFactory) *sharedInformerFactory {
+		factory.namespace = namespace
+		return factory
+	}
+}
+
+// WithTransform sets a transform on all informers.
+func WithTransform(transform cache.TransformFunc) SharedInformerOption {
+	return func(factory *sharedInformerFactory) *sharedInformerFactory {
+		factory.transform = transform
+		return factory
+	}
+}
+
+// NewSharedInformerFactory constructs a new instance of sharedInformerFactory for all namespaces.
+func NewSharedInformerFactory(client versioned.Interface, defaultResync time.Duration) SharedInformerFactory {
+	return NewSharedInformerFactoryWithOptions(client, defaultResync)
+}
+
+// NewFilteredSharedInformerFactory constructs a new instance of sharedInformerFactory.
+// Listers obtained via this SharedInformerFactory will be subject to the same filters
+// as specified here.
+//
+// Deprecated: Please use NewSharedInformerFactoryWithOptions instead
+func NewFilteredSharedInformerFactory(client versioned.Interface, defaultResync time.Duration, namespace string, tweakListOptions internalinterfaces.TweakListOptionsFunc) SharedInformerFactory {
+	return NewSharedInformerFactoryWithOptions(client, defaultResync, WithNamespace(namespace), WithTweakListOptions(tweakListOptions))
+}
+
+// NewSharedInformerFactoryWithOptions constructs a new instance of a SharedInformerFactory with additional options.
+func NewSharedInformerFactoryWithOptions(client versioned.Interface, defaultResync time.Duration, options ...SharedInformerOption) SharedInformerFactory {
+	factory := &sharedInformerFactory{
+		client:           client,
+		namespace:        v1.NamespaceAll,
+		defaultResync:    defaultResync,
+		informers:        make(map[reflect.Type]cache.SharedIndexInformer),
+		startedInformers: make(map[reflect.Type]bool),
+		customResync:     make(map[reflect.Type]time.Duration),
+	}
+
+	// Apply all options
+	for _, opt := range options {
+		factory = opt(factory)
+	}
+
+	return factory
+}
+
+func (f *sharedInformerFactory) Start(stopCh <-chan struct{}) {
+	f.lock.Lock()
+	defer f.lock.Unlock()
+
+	if f.shuttingDown {
+		return
+	}
+
+	for informerType, informer := range f.informers {
+		if !f.startedInformers[informerType] {
+			f.wg.Add(1)
+			// We need a new variable in each loop iteration,
+			// otherwise the goroutine would use the loop variable
+			// and that keeps changing.
+			informer := informer
+			go func() {
+				defer f.wg.Done()
+				informer.Run(stopCh)
+			}()
+			f.startedInformers[informerType] = true
+		}
+	}
+}
+
+func (f *sharedInformerFactory) Shutdown() {
+	f.lock.Lock()
+	f.shuttingDown = true
+	f.lock.Unlock()
+
+	// Will return immediately if there is nothing to wait for.
+	f.wg.Wait()
+}
+
+func (f *sharedInformerFactory) WaitForCacheSync(stopCh <-chan struct{}) map[reflect.Type]bool {
+	informers := func() map[reflect.Type]cache.SharedIndexInformer {
+		f.lock.Lock()
+		defer f.lock.Unlock()
+
+		informers := map[reflect.Type]cache.SharedIndexInformer{}
+		for informerType, informer := range f.informers {
+			if f.startedInformers[informerType] {
+				informers[informerType] = informer
+			}
+		}
+		return informers
+	}()
+
+	res := map[reflect.Type]bool{}
+	for informType, informer := range informers {
+		res[informType] = cache.WaitForCacheSync(stopCh, informer.HasSynced)
+	}
+	return res
+}
+
+// InformerFor returns the SharedIndexInformer for obj using an internal
+// client.
+func (f *sharedInformerFactory) InformerFor(obj runtime.Object, newFunc internalinterfaces.NewInformerFunc) cache.SharedIndexInformer {
+	f.lock.Lock()
+	defer f.lock.Unlock()
+
+	informerType := reflect.TypeOf(obj)
+	informer, exists := f.informers[informerType]
+	if exists {
+		return informer
+	}
+
+	resyncPeriod, exists := f.customResync[informerType]
+	if !exists {
+		resyncPeriod = f.defaultResync
+	}
+
+	informer = newFunc(f.client, resyncPeriod)
+	informer.SetTransform(f.transform)
+	f.informers[informerType] = informer
+
+	return informer
+}
+
+// SharedInformerFactory provides shared informers for resources in all known
+// API group versions.
+//
+// It is typically used like this:
+//
+//	ctx, cancel := context.WithCancel(context.Background())
+//	defer cancel()
+//	factory := NewSharedInformerFactory(client, resyncPeriod)
+//	defer factory.WaitForStop()    // Returns immediately if nothing was started.
+//	genericInformer := factory.ForResource(resource)
+//	typedInformer := factory.SomeAPIGroup().V1().SomeType()
+//	factory.Start(ctx.Done())          // Start processing these informers.
+//	synced := factory.WaitForCacheSync(ctx.Done())
+//	for v, ok := range synced {
+//	    if !ok {
+//	        fmt.Fprintf(os.Stderr, "caches failed to sync: %v", v)
+//	        return
+//	    }
+//	}
+//
+//	// Creating informers can also be created after Start, but then
+//	// Start must be called again:
+//	anotherGenericInformer := factory.ForResource(resource)
+//	factory.Start(ctx.Done())
+type SharedInformerFactory interface {
+	internalinterfaces.SharedInformerFactory
+
+	// Start initializes all requested informers. They are handled in goroutines
+	// which run until the stop channel gets closed.
+	// Warning: Start does not block. When run in a go-routine, it will race with a later WaitForCacheSync.
+	Start(stopCh <-chan struct{})
+
+	// Shutdown marks a factory as shutting down. At that point no new
+	// informers can be started anymore and Start will return without
+	// doing anything.
+	//
+	// In addition, Shutdown blocks until all goroutines have terminated. For that
+	// to happen, the close channel(s) that they were started with must be closed,
+	// either before Shutdown gets called or while it is waiting.
+	//
+	// Shutdown may be called multiple times, even concurrently. All such calls will
+	// block until all goroutines have terminated.
+	Shutdown()
+
+	// WaitForCacheSync blocks until all started informers' caches were synced
+	// or the stop channel gets closed.
+	WaitForCacheSync(stopCh <-chan struct{}) map[reflect.Type]bool
+
+	// ForResource gives generic access to a shared informer of the matching type.
+	ForResource(resource schema.GroupVersionResource) (GenericInformer, error)
+
+	// InformerFor returns the SharedIndexInformer for obj using an internal
+	// client.
+	InformerFor(obj runtime.Object, newFunc internalinterfaces.NewInformerFunc) cache.SharedIndexInformer
+
+	Monok8s() monok8s.Interface
+}
+
+func (f *sharedInformerFactory) Monok8s() monok8s.Interface {
+	return monok8s.New(f, f.namespace, f.tweakListOptions)
+}

clitools/pkg/generated/informers/externalversions/generic.go

@@ -0,0 +1,50 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package externalversions
+
+import (
+	fmt "fmt"
+
+	v1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	schema "k8s.io/apimachinery/pkg/runtime/schema"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// GenericInformer is type of SharedIndexInformer which will locate and delegate to other
+// sharedInformers based on type
+type GenericInformer interface {
+	Informer() cache.SharedIndexInformer
+	Lister() cache.GenericLister
+}
+
+type genericInformer struct {
+	informer cache.SharedIndexInformer
+	resource schema.GroupResource
+}
+
+// Informer returns the SharedIndexInformer.
+func (f *genericInformer) Informer() cache.SharedIndexInformer {
+	return f.informer
+}
+
+// Lister returns the GenericLister.
+func (f *genericInformer) Lister() cache.GenericLister {
+	return cache.NewGenericLister(f.Informer().GetIndexer(), f.resource)
+}
+
+// ForResource gives generic access to a shared informer of the matching type
+// TODO extend this to unknown resources with a client pool
+func (f *sharedInformerFactory) ForResource(resource schema.GroupVersionResource) (GenericInformer, error) {
+	switch resource {
+	// Group=monok8s, Version=v1alpha1
+	case v1alpha1.SchemeGroupVersion.WithResource("osupgrades"):
+		return &genericInformer{resource: resource.GroupResource(), informer: f.Monok8s().V1alpha1().OSUpgrades().Informer()}, nil
+	case v1alpha1.SchemeGroupVersion.WithResource("osupgradeprogresses"):
+		return &genericInformer{resource: resource.GroupResource(), informer: f.Monok8s().V1alpha1().OSUpgradeProgresses().Informer()}, nil
+
+	}
+
+	return nil, fmt.Errorf("no informer found for %v", resource)
+}

clitools/pkg/generated/informers/externalversions/internalinterfaces/factory_interfaces.go

@@ -0,0 +1,26 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package internalinterfaces
+
+import (
+	time "time"
+
+	versioned "example.com/monok8s/pkg/generated/clientset/versioned"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// NewInformerFunc takes versioned.Interface and time.Duration to return a SharedIndexInformer.
+type NewInformerFunc func(versioned.Interface, time.Duration) cache.SharedIndexInformer
+
+// SharedInformerFactory a small interface to allow for adding an informer without an import cycle
+type SharedInformerFactory interface {
+	Start(stopCh <-chan struct{})
+	InformerFor(obj runtime.Object, newFunc NewInformerFunc) cache.SharedIndexInformer
+}
+
+// TweakListOptionsFunc is a function that transforms a v1.ListOptions.
+type TweakListOptionsFunc func(*v1.ListOptions)

clitools/pkg/generated/informers/externalversions/monok8s/interface.go

@@ -0,0 +1,32 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package monok8s
+
+import (
+	internalinterfaces "example.com/monok8s/pkg/generated/informers/externalversions/internalinterfaces"
+	v1alpha1 "example.com/monok8s/pkg/generated/informers/externalversions/monok8s/v1alpha1"
+)
+
+// Interface provides access to each of this group's versions.
+type Interface interface {
+	// V1alpha1 provides access to shared informers for resources in V1alpha1.
+	V1alpha1() v1alpha1.Interface
+}
+
+type group struct {
+	factory          internalinterfaces.SharedInformerFactory
+	namespace        string
+	tweakListOptions internalinterfaces.TweakListOptionsFunc
+}
+
+// New returns a new Interface.
+func New(f internalinterfaces.SharedInformerFactory, namespace string, tweakListOptions internalinterfaces.TweakListOptionsFunc) Interface {
+	return &group{factory: f, namespace: namespace, tweakListOptions: tweakListOptions}
+}
+
+// V1alpha1 returns a new v1alpha1.Interface.
+func (g *group) V1alpha1() v1alpha1.Interface {
+	return v1alpha1.New(g.factory, g.namespace, g.tweakListOptions)
+}

clitools/pkg/generated/informers/externalversions/monok8s/v1alpha1/interface.go

@@ -0,0 +1,38 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	internalinterfaces "example.com/monok8s/pkg/generated/informers/externalversions/internalinterfaces"
+)
+
+// Interface provides access to all the informers in this group version.
+type Interface interface {
+	// OSUpgrades returns a OSUpgradeInformer.
+	OSUpgrades() OSUpgradeInformer
+	// OSUpgradeProgresses returns a OSUpgradeProgressInformer.
+	OSUpgradeProgresses() OSUpgradeProgressInformer
+}
+
+type version struct {
+	factory          internalinterfaces.SharedInformerFactory
+	namespace        string
+	tweakListOptions internalinterfaces.TweakListOptionsFunc
+}
+
+// New returns a new Interface.
+func New(f internalinterfaces.SharedInformerFactory, namespace string, tweakListOptions internalinterfaces.TweakListOptionsFunc) Interface {
+	return &version{factory: f, namespace: namespace, tweakListOptions: tweakListOptions}
+}
+
+// OSUpgrades returns a OSUpgradeInformer.
+func (v *version) OSUpgrades() OSUpgradeInformer {
+	return &oSUpgradeInformer{factory: v.factory, namespace: v.namespace, tweakListOptions: v.tweakListOptions}
+}
+
+// OSUpgradeProgresses returns a OSUpgradeProgressInformer.
+func (v *version) OSUpgradeProgresses() OSUpgradeProgressInformer {
+	return &oSUpgradeProgressInformer{factory: v.factory, namespace: v.namespace, tweakListOptions: v.tweakListOptions}
+}

clitools/pkg/generated/informers/externalversions/monok8s/v1alpha1/osupgrade.go

@@ -0,0 +1,88 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	context "context"
+	time "time"
+
+	apismonok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	versioned "example.com/monok8s/pkg/generated/clientset/versioned"
+	internalinterfaces "example.com/monok8s/pkg/generated/informers/externalversions/internalinterfaces"
+	monok8sv1alpha1 "example.com/monok8s/pkg/generated/listers/monok8s/v1alpha1"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+	watch "k8s.io/apimachinery/pkg/watch"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// OSUpgradeInformer provides access to a shared informer and lister for
+// OSUpgrades.
+type OSUpgradeInformer interface {
+	Informer() cache.SharedIndexInformer
+	Lister() monok8sv1alpha1.OSUpgradeLister
+}
+
+type oSUpgradeInformer struct {
+	factory          internalinterfaces.SharedInformerFactory
+	tweakListOptions internalinterfaces.TweakListOptionsFunc
+	namespace        string
+}
+
+// NewOSUpgradeInformer constructs a new informer for OSUpgrade type.
+// Always prefer using an informer factory to get a shared informer instead of getting an independent
+// one. This reduces memory footprint and number of connections to the server.
+func NewOSUpgradeInformer(client versioned.Interface, namespace string, resyncPeriod time.Duration, indexers cache.Indexers) cache.SharedIndexInformer {
+	return NewFilteredOSUpgradeInformer(client, namespace, resyncPeriod, indexers, nil)
+}
+
+// NewFilteredOSUpgradeInformer constructs a new informer for OSUpgrade type.
+// Always prefer using an informer factory to get a shared informer instead of getting an independent
+// one. This reduces memory footprint and number of connections to the server.
+func NewFilteredOSUpgradeInformer(client versioned.Interface, namespace string, resyncPeriod time.Duration, indexers cache.Indexers, tweakListOptions internalinterfaces.TweakListOptionsFunc) cache.SharedIndexInformer {
+	return cache.NewSharedIndexInformer(
+		cache.ToListWatcherWithWatchListSemantics(&cache.ListWatch{
+			ListFunc: func(options v1.ListOptions) (runtime.Object, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgrades(namespace).List(context.Background(), options)
+			},
+			WatchFunc: func(options v1.ListOptions) (watch.Interface, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgrades(namespace).Watch(context.Background(), options)
+			},
+			ListWithContextFunc: func(ctx context.Context, options v1.ListOptions) (runtime.Object, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgrades(namespace).List(ctx, options)
+			},
+			WatchFuncWithContext: func(ctx context.Context, options v1.ListOptions) (watch.Interface, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgrades(namespace).Watch(ctx, options)
+			},
+		}, client),
+		&apismonok8sv1alpha1.OSUpgrade{},
+		resyncPeriod,
+		indexers,
+	)
+}
+
+func (f *oSUpgradeInformer) defaultInformer(client versioned.Interface, resyncPeriod time.Duration) cache.SharedIndexInformer {
+	return NewFilteredOSUpgradeInformer(client, f.namespace, resyncPeriod, cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc}, f.tweakListOptions)
+}
+
+func (f *oSUpgradeInformer) Informer() cache.SharedIndexInformer {
+	return f.factory.InformerFor(&apismonok8sv1alpha1.OSUpgrade{}, f.defaultInformer)
+}
+
+func (f *oSUpgradeInformer) Lister() monok8sv1alpha1.OSUpgradeLister {
+	return monok8sv1alpha1.NewOSUpgradeLister(f.Informer().GetIndexer())
+}

clitools/pkg/generated/informers/externalversions/monok8s/v1alpha1/osupgradeprogress.go

@@ -0,0 +1,88 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	context "context"
+	time "time"
+
+	apismonok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	versioned "example.com/monok8s/pkg/generated/clientset/versioned"
+	internalinterfaces "example.com/monok8s/pkg/generated/informers/externalversions/internalinterfaces"
+	monok8sv1alpha1 "example.com/monok8s/pkg/generated/listers/monok8s/v1alpha1"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+	watch "k8s.io/apimachinery/pkg/watch"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// OSUpgradeProgressInformer provides access to a shared informer and lister for
+// OSUpgradeProgresses.
+type OSUpgradeProgressInformer interface {
+	Informer() cache.SharedIndexInformer
+	Lister() monok8sv1alpha1.OSUpgradeProgressLister
+}
+
+type oSUpgradeProgressInformer struct {
+	factory          internalinterfaces.SharedInformerFactory
+	tweakListOptions internalinterfaces.TweakListOptionsFunc
+	namespace        string
+}
+
+// NewOSUpgradeProgressInformer constructs a new informer for OSUpgradeProgress type.
+// Always prefer using an informer factory to get a shared informer instead of getting an independent
+// one. This reduces memory footprint and number of connections to the server.
+func NewOSUpgradeProgressInformer(client versioned.Interface, namespace string, resyncPeriod time.Duration, indexers cache.Indexers) cache.SharedIndexInformer {
+	return NewFilteredOSUpgradeProgressInformer(client, namespace, resyncPeriod, indexers, nil)
+}
+
+// NewFilteredOSUpgradeProgressInformer constructs a new informer for OSUpgradeProgress type.
+// Always prefer using an informer factory to get a shared informer instead of getting an independent
+// one. This reduces memory footprint and number of connections to the server.
+func NewFilteredOSUpgradeProgressInformer(client versioned.Interface, namespace string, resyncPeriod time.Duration, indexers cache.Indexers, tweakListOptions internalinterfaces.TweakListOptionsFunc) cache.SharedIndexInformer {
+	return cache.NewSharedIndexInformer(
+		cache.ToListWatcherWithWatchListSemantics(&cache.ListWatch{
+			ListFunc: func(options v1.ListOptions) (runtime.Object, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgradeProgresses(namespace).List(context.Background(), options)
+			},
+			WatchFunc: func(options v1.ListOptions) (watch.Interface, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgradeProgresses(namespace).Watch(context.Background(), options)
+			},
+			ListWithContextFunc: func(ctx context.Context, options v1.ListOptions) (runtime.Object, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgradeProgresses(namespace).List(ctx, options)
+			},
+			WatchFuncWithContext: func(ctx context.Context, options v1.ListOptions) (watch.Interface, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgradeProgresses(namespace).Watch(ctx, options)
+			},
+		}, client),
+		&apismonok8sv1alpha1.OSUpgradeProgress{},
+		resyncPeriod,
+		indexers,
+	)
+}
+
+func (f *oSUpgradeProgressInformer) defaultInformer(client versioned.Interface, resyncPeriod time.Duration) cache.SharedIndexInformer {
+	return NewFilteredOSUpgradeProgressInformer(client, f.namespace, resyncPeriod, cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc}, f.tweakListOptions)
+}
+
+func (f *oSUpgradeProgressInformer) Informer() cache.SharedIndexInformer {
+	return f.factory.InformerFor(&apismonok8sv1alpha1.OSUpgradeProgress{}, f.defaultInformer)
+}
+
+func (f *oSUpgradeProgressInformer) Lister() monok8sv1alpha1.OSUpgradeProgressLister {
+	return monok8sv1alpha1.NewOSUpgradeProgressLister(f.Informer().GetIndexer())
+}

clitools/pkg/generated/listers/monok8s/v1alpha1/expansion_generated.go

@@ -0,0 +1,21 @@
+/* MIT License */
+
+// Code generated by lister-gen. DO NOT EDIT.
+
+package v1alpha1
+
+// OSUpgradeListerExpansion allows custom methods to be added to
+// OSUpgradeLister.
+type OSUpgradeListerExpansion interface{}
+
+// OSUpgradeNamespaceListerExpansion allows custom methods to be added to
+// OSUpgradeNamespaceLister.
+type OSUpgradeNamespaceListerExpansion interface{}
+
+// OSUpgradeProgressListerExpansion allows custom methods to be added to
+// OSUpgradeProgressLister.
+type OSUpgradeProgressListerExpansion interface{}
+
+// OSUpgradeProgressNamespaceListerExpansion allows custom methods to be added to
+// OSUpgradeProgressNamespaceLister.
+type OSUpgradeProgressNamespaceListerExpansion interface{}

clitools/pkg/generated/listers/monok8s/v1alpha1/osupgrade.go

@@ -0,0 +1,56 @@
+/* MIT License */
+
+// Code generated by lister-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	labels "k8s.io/apimachinery/pkg/labels"
+	listers "k8s.io/client-go/listers"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// OSUpgradeLister helps list OSUpgrades.
+// All objects returned here must be treated as read-only.
+type OSUpgradeLister interface {
+	// List lists all OSUpgrades in the indexer.
+	// Objects returned here must be treated as read-only.
+	List(selector labels.Selector) (ret []*monok8sv1alpha1.OSUpgrade, err error)
+	// OSUpgrades returns an object that can list and get OSUpgrades.
+	OSUpgrades(namespace string) OSUpgradeNamespaceLister
+	OSUpgradeListerExpansion
+}
+
+// oSUpgradeLister implements the OSUpgradeLister interface.
+type oSUpgradeLister struct {
+	listers.ResourceIndexer[*monok8sv1alpha1.OSUpgrade]
+}
+
+// NewOSUpgradeLister returns a new OSUpgradeLister.
+func NewOSUpgradeLister(indexer cache.Indexer) OSUpgradeLister {
+	return &oSUpgradeLister{listers.New[*monok8sv1alpha1.OSUpgrade](indexer, monok8sv1alpha1.Resource("osupgrade"))}
+}
+
+// OSUpgrades returns an object that can list and get OSUpgrades.
+func (s *oSUpgradeLister) OSUpgrades(namespace string) OSUpgradeNamespaceLister {
+	return oSUpgradeNamespaceLister{listers.NewNamespaced[*monok8sv1alpha1.OSUpgrade](s.ResourceIndexer, namespace)}
+}
+
+// OSUpgradeNamespaceLister helps list and get OSUpgrades.
+// All objects returned here must be treated as read-only.
+type OSUpgradeNamespaceLister interface {
+	// List lists all OSUpgrades in the indexer for a given namespace.
+	// Objects returned here must be treated as read-only.
+	List(selector labels.Selector) (ret []*monok8sv1alpha1.OSUpgrade, err error)
+	// Get retrieves the OSUpgrade from the indexer for a given namespace and name.
+	// Objects returned here must be treated as read-only.
+	Get(name string) (*monok8sv1alpha1.OSUpgrade, error)
+	OSUpgradeNamespaceListerExpansion
+}
+
+// oSUpgradeNamespaceLister implements the OSUpgradeNamespaceLister
+// interface.
+type oSUpgradeNamespaceLister struct {
+	listers.ResourceIndexer[*monok8sv1alpha1.OSUpgrade]
+}

clitools/pkg/generated/listers/monok8s/v1alpha1/osupgradeprogress.go

@@ -0,0 +1,56 @@
+/* MIT License */
+
+// Code generated by lister-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	labels "k8s.io/apimachinery/pkg/labels"
+	listers "k8s.io/client-go/listers"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// OSUpgradeProgressLister helps list OSUpgradeProgresses.
+// All objects returned here must be treated as read-only.
+type OSUpgradeProgressLister interface {
+	// List lists all OSUpgradeProgresses in the indexer.
+	// Objects returned here must be treated as read-only.
+	List(selector labels.Selector) (ret []*monok8sv1alpha1.OSUpgradeProgress, err error)
+	// OSUpgradeProgresses returns an object that can list and get OSUpgradeProgresses.
+	OSUpgradeProgresses(namespace string) OSUpgradeProgressNamespaceLister
+	OSUpgradeProgressListerExpansion
+}
+
+// oSUpgradeProgressLister implements the OSUpgradeProgressLister interface.
+type oSUpgradeProgressLister struct {
+	listers.ResourceIndexer[*monok8sv1alpha1.OSUpgradeProgress]
+}
+
+// NewOSUpgradeProgressLister returns a new OSUpgradeProgressLister.
+func NewOSUpgradeProgressLister(indexer cache.Indexer) OSUpgradeProgressLister {
+	return &oSUpgradeProgressLister{listers.New[*monok8sv1alpha1.OSUpgradeProgress](indexer, monok8sv1alpha1.Resource("osupgradeprogress"))}
+}
+
+// OSUpgradeProgresses returns an object that can list and get OSUpgradeProgresses.
+func (s *oSUpgradeProgressLister) OSUpgradeProgresses(namespace string) OSUpgradeProgressNamespaceLister {
+	return oSUpgradeProgressNamespaceLister{listers.NewNamespaced[*monok8sv1alpha1.OSUpgradeProgress](s.ResourceIndexer, namespace)}
+}
+
+// OSUpgradeProgressNamespaceLister helps list and get OSUpgradeProgresses.
+// All objects returned here must be treated as read-only.
+type OSUpgradeProgressNamespaceLister interface {
+	// List lists all OSUpgradeProgresses in the indexer for a given namespace.
+	// Objects returned here must be treated as read-only.
+	List(selector labels.Selector) (ret []*monok8sv1alpha1.OSUpgradeProgress, err error)
+	// Get retrieves the OSUpgradeProgress from the indexer for a given namespace and name.
+	// Objects returned here must be treated as read-only.
+	Get(name string) (*monok8sv1alpha1.OSUpgradeProgress, error)
+	OSUpgradeProgressNamespaceListerExpansion
+}
+
+// oSUpgradeProgressNamespaceLister implements the OSUpgradeProgressNamespaceLister
+// interface.
+type oSUpgradeProgressNamespaceLister struct {
+	listers.ResourceIndexer[*monok8sv1alpha1.OSUpgradeProgress]
+}

clitools/pkg/kube/clients.go

@@ -3,11 +3,12 @@ package kube
 import (
 	"fmt"
 
+	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	monoclientset "example.com/monok8s/pkg/generated/clientset/versioned"
 	apiextensionsclientset "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
 	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
 	kubernetes "k8s.io/client-go/kubernetes"
 	clientgoscheme "k8s.io/client-go/kubernetes/scheme"
-	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
 
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/cli-runtime/pkg/genericclioptions"
@@ -21,6 +22,7 @@ type Clients struct {
 	Dynamic          dynamic.Interface
 	APIExtensions    apiextensionsclientset.Interface
 	RESTClientGetter genericclioptions.RESTClientGetter
+	MonoKS           monoclientset.Interface
 }
 
 func NewClients(flags *genericclioptions.ConfigFlags) (*Clients, error) {
@@ -40,7 +42,19 @@ func NewClients(flags *genericclioptions.ConfigFlags) (*Clients, error) {
 	if err != nil {
 		return nil, fmt.Errorf("build apiextensions client: %w", err)
 	}
-	return &Clients{Config: cfg, Kubernetes: kubeClient, Dynamic: dyn, APIExtensions: ext, RESTClientGetter: flags}, nil
+	mono, err := monoclientset.NewForConfig(cfg)
+	if err != nil {
+		return nil, fmt.Errorf("build monok8s client: %w", err)
+	}
+
+	return &Clients{
+		Config:           cfg,
+		Kubernetes:       kubeClient,
+		Dynamic:          dyn,
+		APIExtensions:    ext,
+		RESTClientGetter: flags,
+		MonoKS:           mono,
+	}, nil
 }
 
 func NewClientsFromKubeconfig(kubeconfigPath string) (*Clients, error) {

clitools/pkg/node/agent.go

@@ -3,39 +3,27 @@ package node
 import (
 	"context"
 	"fmt"
-	"reflect"
 	"strings"
 
-	appsv1 "k8s.io/api/apps/v1"
-	corev1 "k8s.io/api/core/v1"
-	rbacv1 "k8s.io/api/rbac/v1"
-	apierrors "k8s.io/apimachinery/pkg/api/errors"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/client-go/kubernetes"
 	"k8s.io/klog/v2"
 
-	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
 	"example.com/monok8s/pkg/kube"
+	"example.com/monok8s/pkg/render"
 	templates "example.com/monok8s/pkg/templates"
 )
 
-const (
+const kubeconfig = "/etc/kubernetes/admin.conf"
-	controlAgentName              = "control-agent"
-	controlAgentNodeSelectorValue = "true"
-	controlAgentImage             = "localhost/monok8s/control-agent:dev"
-	kubeconfig                    = "/etc/kubernetes/admin.conf"
-)
 
-func ApplyControlAgentDaemonSetResources(ctx context.Context, n *NodeContext) error {
+func ApplyNodeControlDaemonSetResources(ctx context.Context, n *NodeContext) error {
-	// Only the control-plane should bootstrap this DaemonSet definition.
+	if strings.TrimSpace(n.Config.Spec.ClusterRole) != "control-plane" || !n.Config.Spec.EnableNodeControl {
-	// And only when the feature is enabled.
+		klog.InfoS("skipped for",
-	if strings.TrimSpace(n.Config.Spec.ClusterRole) != "control-plane" || !n.Config.Spec.EnableControlAgent {
+			"clusterRole", n.Config.Spec.ClusterRole,
-		klog.InfoS("skipped for", "clusterRole", n.Config.Spec.ClusterRole, "enableControlAgent", n.Config.Spec.EnableControlAgent)
+			"enableNodeAgent", n.Config.Spec.EnableNodeControl,
+		)
 		return nil
 	}
 
-	err := ApplyCRDs(ctx, n)
+	if err := ApplyCRDs(ctx, n); err != nil {
-	if err != nil {
 		return err
 	}
 
@@ -49,330 +37,13 @@ func ApplyControlAgentDaemonSetResources(ctx context.Context, n *NodeContext) er
 		return fmt.Errorf("build kube clients from %s: %w", kubeconfig, err)
 	}
 
-	labels := map[string]string{
+	conf := render.AgentConf{
-		"app.kubernetes.io/name":       controlAgentName,
-		"app.kubernetes.io/component":  "agent",
-		"app.kubernetes.io/part-of":    "monok8s",
-		"app.kubernetes.io/managed-by": "ctl",
-	}
-
-	kubeClient := clients.Kubernetes
-
-	if err := applyControlAgentServiceAccount(ctx, kubeClient, namespace, labels); err != nil {
-		return fmt.Errorf("apply serviceaccount: %w", err)
-	}
-	if err := applyControlAgentClusterRole(ctx, kubeClient, labels); err != nil {
-		return fmt.Errorf("apply clusterrole: %w", err)
-	}
-	if err := applyControlAgentClusterRoleBinding(ctx, kubeClient, namespace, labels); err != nil {
-		return fmt.Errorf("apply clusterrolebinding: %w", err)
-	}
-	if err := applyControlAgentDaemonSet(ctx, kubeClient, namespace, labels); err != nil {
-		return fmt.Errorf("apply daemonset: %w", err)
-	}
-
-	return nil
-}
-
-func applyControlAgentServiceAccount(ctx context.Context, kubeClient kubernetes.Interface, namespace string, labels map[string]string) error {
-	want := &corev1.ServiceAccount{
-		ObjectMeta: metav1.ObjectMeta{
-			Name:      controlAgentName,
 		Namespace: namespace,
-			Labels:    labels,
-		},
 	}
 
-	existing, err := kubeClient.CoreV1().ServiceAccounts(namespace).Get(ctx, controlAgentName, metav1.GetOptions{})
+	if err := render.ApplyAgentDaemonSets(ctx, clients.Kubernetes, conf); err != nil {
-	if apierrors.IsNotFound(err) {
+		return fmt.Errorf("apply node agent daemonset resources: %w", err)
-		_, err = kubeClient.CoreV1().ServiceAccounts(namespace).Create(ctx, want, metav1.CreateOptions{})
-		return err
-	}
-	if err != nil {
-		return err
 	}
 
-	changed := false
-	if !reflect.DeepEqual(existing.Labels, want.Labels) {
-		existing.Labels = want.Labels
-		changed = true
-	}
-
-	if !changed {
 	return nil
-	}
-
-	_, err = kubeClient.CoreV1().ServiceAccounts(namespace).Update(ctx, existing, metav1.UpdateOptions{})
-	return err
-}
-
-func applyControlAgentClusterRole(ctx context.Context, kubeClient kubernetes.Interface, labels map[string]string) error {
-	wantRules := []rbacv1.PolicyRule{
-		{
-			APIGroups: []string{monov1alpha1.Group},
-			Resources: []string{"osupgrades"},
-			Verbs:     []string{"get", "list", "watch"},
-		},
-		{
-			APIGroups: []string{monov1alpha1.Group},
-			Resources: []string{"osupgrades/status"},
-			Verbs:     []string{"get", "patch", "update"},
-		},
-		{
-			APIGroups: []string{monov1alpha1.Group},
-			Resources: []string{"osupgradeprogresses"},
-			Verbs:     []string{"get", "list", "watch", "create", "patch", "update"},
-		},
-		{
-			APIGroups: []string{monov1alpha1.Group},
-			Resources: []string{"osupgradeprogresses/status"},
-			Verbs:     []string{"get", "list", "watch", "create", "patch", "update"},
-		},
-		{
-			APIGroups: []string{""},
-			Resources: []string{"nodes"},
-			Verbs:     []string{"get", "list", "watch"},
-		},
-	}
-
-	want := &rbacv1.ClusterRole{
-		ObjectMeta: metav1.ObjectMeta{
-			Name:   controlAgentName,
-			Labels: labels,
-		},
-		Rules: wantRules,
-	}
-
-	existing, err := kubeClient.RbacV1().ClusterRoles().Get(ctx, controlAgentName, metav1.GetOptions{})
-	if apierrors.IsNotFound(err) {
-		_, err = kubeClient.RbacV1().ClusterRoles().Create(ctx, want, metav1.CreateOptions{})
-		return err
-	}
-	if err != nil {
-		return err
-	}
-
-	changed := false
-	if !reflect.DeepEqual(existing.Labels, want.Labels) {
-		existing.Labels = want.Labels
-		changed = true
-	}
-	if !reflect.DeepEqual(existing.Rules, want.Rules) {
-		existing.Rules = want.Rules
-		changed = true
-	}
-
-	if !changed {
-		return nil
-	}
-
-	_, err = kubeClient.RbacV1().ClusterRoles().Update(ctx, existing, metav1.UpdateOptions{})
-	return err
-}
-
-func applyControlAgentClusterRoleBinding(ctx context.Context, kubeClient kubernetes.Interface, namespace string, labels map[string]string) error {
-	wantRoleRef := rbacv1.RoleRef{
-		APIGroup: rbacv1.GroupName,
-		Kind:     "ClusterRole",
-		Name:     controlAgentName,
-	}
-	wantSubjects := []rbacv1.Subject{
-		{
-			Kind:      "ServiceAccount",
-			Name:      controlAgentName,
-			Namespace: namespace,
-		},
-	}
-
-	want := &rbacv1.ClusterRoleBinding{
-		ObjectMeta: metav1.ObjectMeta{
-			Name:   controlAgentName,
-			Labels: labels,
-		},
-		RoleRef:  wantRoleRef,
-		Subjects: wantSubjects,
-	}
-
-	existing, err := kubeClient.RbacV1().ClusterRoleBindings().Get(ctx, controlAgentName, metav1.GetOptions{})
-	if apierrors.IsNotFound(err) {
-		_, err = kubeClient.RbacV1().ClusterRoleBindings().Create(ctx, want, metav1.CreateOptions{})
-		return err
-	}
-	if err != nil {
-		return err
-	}
-
-	// roleRef is immutable. If it differs, fail loudly instead of pretending we can patch it.
-	if !reflect.DeepEqual(existing.RoleRef, want.RoleRef) {
-		return fmt.Errorf("existing ClusterRoleBinding %q has different roleRef and must be recreated", controlAgentName)
-	}
-
-	changed := false
-	if !reflect.DeepEqual(existing.Labels, want.Labels) {
-		existing.Labels = want.Labels
-		changed = true
-	}
-	if !reflect.DeepEqual(existing.Subjects, want.Subjects) {
-		existing.Subjects = want.Subjects
-		changed = true
-	}
-
-	if !changed {
-		return nil
-	}
-
-	_, err = kubeClient.RbacV1().ClusterRoleBindings().Update(ctx, existing, metav1.UpdateOptions{})
-	return err
-}
-
-func applyControlAgentDaemonSet(ctx context.Context, kubeClient kubernetes.Interface, namespace string, labels map[string]string) error {
-	privileged := true
-
-	dsLabels := map[string]string{
-		"app.kubernetes.io/name":       controlAgentName,
-		"app.kubernetes.io/component":  "agent",
-		"app.kubernetes.io/part-of":    "monok8s",
-		"app.kubernetes.io/managed-by": "ctl",
-	}
-
-	want := &appsv1.DaemonSet{
-		ObjectMeta: metav1.ObjectMeta{
-			Name:      controlAgentName,
-			Namespace: namespace,
-			Labels:    labels,
-		},
-		Spec: appsv1.DaemonSetSpec{
-			Selector: &metav1.LabelSelector{
-				MatchLabels: map[string]string{
-					"app.kubernetes.io/name": controlAgentName,
-				},
-			},
-			Template: corev1.PodTemplateSpec{
-				ObjectMeta: metav1.ObjectMeta{
-					Labels: dsLabels,
-				},
-				Spec: corev1.PodSpec{
-					ServiceAccountName: controlAgentName,
-					HostNetwork:        true,
-					HostPID:            true,
-					DNSPolicy:          corev1.DNSClusterFirstWithHostNet,
-					NodeSelector: map[string]string{
-						monov1alpha1.ControlAgentKey: controlAgentNodeSelectorValue,
-					},
-					Tolerations: []corev1.Toleration{
-						{Operator: corev1.TolerationOpExists},
-					},
-					Containers: []corev1.Container{
-						{
-							Name:            "agent",
-							Image:           controlAgentImage,
-							ImagePullPolicy: corev1.PullNever,
-							Args:            []string{"agent", "--env-file", "$(CLUSTER_ENV_FILE)"},
-							Env: []corev1.EnvVar{
-								{
-									Name: "NODE_NAME",
-									ValueFrom: &corev1.EnvVarSource{
-										FieldRef: &corev1.ObjectFieldSelector{
-											APIVersion: "v1",
-											FieldPath:  "spec.nodeName",
-										},
-									},
-								},
-								{
-									Name:  "CLUSTER_ENV_FILE",
-									Value: "/host/opt/monok8s/config/cluster.env",
-								},
-								{
-									Name:  "FW_ENV_CONFIG_FILE",
-									Value: "/host/etc/fw_env.config",
-								},
-							},
-							SecurityContext: &corev1.SecurityContext{
-								Privileged: &privileged,
-							},
-							VolumeMounts: []corev1.VolumeMount{
-								{
-									Name:      "host-dev",
-									MountPath: "/dev",
-								},
-								{
-									Name:      "host-etc",
-									MountPath: "/host/etc",
-									ReadOnly:  true,
-								},
-								{
-									Name:      "host-config",
-									MountPath: "/host/opt/monok8s/config",
-									ReadOnly:  true,
-								},
-							},
-						},
-					},
-					Volumes: []corev1.Volume{
-						{
-							Name: "host-dev",
-							VolumeSource: corev1.VolumeSource{
-								HostPath: &corev1.HostPathVolumeSource{
-									Path: "/dev",
-									Type: hostPathType(corev1.HostPathDirectory),
-								},
-							},
-						},
-						{
-							Name: "host-etc",
-							VolumeSource: corev1.VolumeSource{
-								HostPath: &corev1.HostPathVolumeSource{
-									Path: "/etc",
-									Type: hostPathType(corev1.HostPathDirectory),
-								},
-							},
-						},
-						{
-							Name: "host-config",
-							VolumeSource: corev1.VolumeSource{
-								HostPath: &corev1.HostPathVolumeSource{
-									Path: "/opt/monok8s/config",
-									Type: hostPathType(corev1.HostPathDirectory),
-								},
-							},
-						},
-					},
-				},
-			},
-		},
-	}
-
-	existing, err := kubeClient.AppsV1().DaemonSets(namespace).Get(ctx, controlAgentName, metav1.GetOptions{})
-	if apierrors.IsNotFound(err) {
-		_, err = kubeClient.AppsV1().DaemonSets(namespace).Create(ctx, want, metav1.CreateOptions{})
-		return err
-	}
-	if err != nil {
-		return err
-	}
-
-	changed := false
-	if !reflect.DeepEqual(existing.Labels, want.Labels) {
-		existing.Labels = want.Labels
-		changed = true
-	}
-	if !reflect.DeepEqual(existing.Spec, want.Spec) {
-		existing.Spec = want.Spec
-		changed = true
-	}
-
-	if !changed {
-		return nil
-	}
-
-	_, err = kubeClient.AppsV1().DaemonSets(namespace).Update(ctx, existing, metav1.UpdateOptions{})
-	return err
-}
-
-func hostPathType(t corev1.HostPathType) *corev1.HostPathType {
-	return &t
-}
-
-func mountPropagationMode(m corev1.MountPropagationMode) *corev1.MountPropagationMode {
-	return &m
 }

clitools/pkg/node/kubeadm.go

@@ -7,13 +7,11 @@ import (
 	"fmt"
 	"net"
 	"os"
+	"sort"
 	"strings"
 	"time"
 
 	"gopkg.in/yaml.v3"
-	"k8s.io/client-go/discovery"
-	"k8s.io/client-go/kubernetes"
-	"k8s.io/client-go/tools/clientcmd"
 	"k8s.io/klog/v2"
 
 	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
@@ -26,6 +24,16 @@ const (
 	tmpKubeadmInitConf    = "/tmp/kubeadm-init.yaml"
 )
 
+func chooseVersionKubeconfig(state *LocalClusterState) string {
+	if state.HasAdminKubeconfig {
+		return adminKubeconfigPath
+	}
+	if state.HasKubeletKubeconfig {
+		return kubeletKubeconfigPath
+	}
+	return ""
+}
+
 func DetectLocalClusterState(ctx context.Context, nctx *NodeContext) error {
 	_ = ctx
 
@@ -258,110 +266,6 @@ func waitForAPIViaKubeconfig(ctx context.Context, kubeconfigPath string, timeout
 	}
 }
 
-func getServerVersion(ctx context.Context, kubeconfigPath string) (string, error) {
-	restCfg, err := clientcmd.BuildConfigFromFlags("", kubeconfigPath)
-	if err != nil {
-		return "", fmt.Errorf("build kubeconfig %s: %w", kubeconfigPath, err)
-	}
-
-	// Keep this short. This is a probe, not a long-running client.
-	restCfg.Timeout = 5 * time.Second
-
-	clientset, err := kubernetes.NewForConfig(restCfg)
-	if err != nil {
-		return "", fmt.Errorf("create clientset: %w", err)
-	}
-
-	disc := clientset.Discovery()
-	return discoverServerVersion(ctx, disc)
-}
-
-func discoverServerVersion(ctx context.Context, disc discovery.DiscoveryInterface) (string, error) {
-	info, err := disc.ServerVersion()
-	if err != nil {
-		return "", err
-	}
-	if info == nil || strings.TrimSpace(info.GitVersion) == "" {
-		return "", errors.New("server version is empty")
-	}
-	return normalizeKubeVersion(info.GitVersion), nil
-}
-
-type kubeVersion struct {
-	Major int
-	Minor int
-	Patch int
-}
-
-func parseKubeVersion(s string) (kubeVersion, error) {
-	s = strings.TrimSpace(s)
-	s = strings.TrimPrefix(s, "v")
-
-	var v kubeVersion
-	n, err := fmt.Sscanf(s, "%d.%d.%d", &v.Major, &v.Minor, &v.Patch)
-	// Accepts "1.29" or "1.29.3"
-	if err != nil || n < 2 {
-		return kubeVersion{}, fmt.Errorf("invalid kubernetes version %q", s)
-	}
-	return v, nil
-}
-
-// Control-plane: keep this strict.
-// Accept same version, or a one-minor step where the node binary is newer than the current cluster.
-// That covers normal control-plane upgrade flow but blocks nonsense.
-func isSupportedControlPlaneSkew(clusterVersion, nodeVersion string) bool {
-	cv, err := parseKubeVersion(clusterVersion)
-	if err != nil {
-		return false
-	}
-	nv, err := parseKubeVersion(nodeVersion)
-	if err != nil {
-		return false
-	}
-
-	if cv.Major != nv.Major {
-		return false
-	}
-	if cv.Minor == nv.Minor {
-		return true
-	}
-	if nv.Minor == cv.Minor+1 {
-		return true
-	}
-	return false
-}
-
-// Worker: kubelet generally must not be newer than the apiserver.
-// Older kubelets are allowed within supported skew range.
-// Your requirement says unsupported worker skew should still proceed, so this
-// only classifies support status and must NOT be used to block this function.
-func isSupportedWorkerSkew(clusterVersion, nodeVersion string) bool {
-	cv, err := parseKubeVersion(clusterVersion)
-	if err != nil {
-		return false
-	}
-	nv, err := parseKubeVersion(nodeVersion)
-	if err != nil {
-		return false
-	}
-
-	if cv.Major != nv.Major {
-		return false
-	}
-
-	// kubelet newer than apiserver => unsupported
-	if nv.Minor > cv.Minor {
-		return false
-	}
-
-	// kubelet up to 3 minors older than apiserver => supported
-	if cv.Minor-nv.Minor <= 3 {
-		return true
-	}
-
-	return false
-}
-
 func ValidateRequiredImagesPresent(ctx context.Context, n *NodeContext) error {
 	if n.Config.Spec.SkipImageCheck {
 		klog.Infof("skipping image check (skipImageCheck=true)")
@@ -418,31 +322,6 @@ func checkImagePresent(ctx context.Context, n *NodeContext, image string) error
 	return nil
 }
 
-func chooseVersionKubeconfig(state *LocalClusterState) string {
-	if state.HasAdminKubeconfig {
-		return adminKubeconfigPath
-	}
-	if state.HasKubeletKubeconfig {
-		return kubeletKubeconfigPath
-	}
-	return ""
-}
-
-func versionEq(a, b string) bool {
-	return normalizeKubeVersion(a) == normalizeKubeVersion(b)
-}
-
-func normalizeKubeVersion(v string) string {
-	v = strings.TrimSpace(v)
-	if v == "" {
-		return ""
-	}
-	if !strings.HasPrefix(v, "v") {
-		v = "v" + v
-	}
-	return v
-}
-
 func buildNodeRegistration(spec monov1alpha1.MonoKSConfigSpec) NodeRegistrationOptions {
 	nodeName := strings.TrimSpace(spec.NodeName)
 	criSocket := strings.TrimSpace(spec.ContainerRuntimeEndpoint)
@@ -464,9 +343,57 @@ func buildNodeRegistration(spec monov1alpha1.MonoKSConfigSpec) NodeRegistrationO
 		)
 	}
 
+	labels := effectiveNodeLabels(spec)
+	if nodeLabels := buildNodeLabelsArg(labels); nodeLabels != "" {
+		nr.KubeletExtraArgs = append(nr.KubeletExtraArgs,
+			KubeadmArg{Name: "node-labels", Value: nodeLabels},
+		)
+	}
+
 	return nr
 }
 
+func effectiveNodeLabels(spec monov1alpha1.MonoKSConfigSpec) map[string]string {
+	if len(spec.NodeLabels) == 0 && !spec.EnableNodeControl {
+		return nil
+	}
+
+	labels := make(map[string]string, len(spec.NodeLabels)+1)
+	for k, v := range spec.NodeLabels {
+		labels[k] = v
+	}
+
+	if spec.EnableNodeControl {
+		labels[monov1alpha1.NodeControlKey] = "true"
+	}
+
+	return labels
+}
+
+func buildNodeLabelsArg(labels map[string]string) string {
+	if len(labels) == 0 {
+		return ""
+	}
+
+	keys := make([]string, 0, len(labels))
+	for k := range labels {
+		k = strings.TrimSpace(k)
+		if k == "" {
+			continue
+		}
+		keys = append(keys, k)
+	}
+	sort.Strings(keys)
+
+	parts := make([]string, 0, len(keys))
+	for _, k := range keys {
+		v := strings.TrimSpace(labels[k])
+		parts = append(parts, k+"="+v)
+	}
+
+	return strings.Join(parts, ",")
+}
+
 func maybeAddBootstrapTaint(nr *NodeRegistrationOptions, role string) {
 	if strings.TrimSpace(role) != "worker" {
 		return
@@ -732,11 +659,6 @@ func RunKubeadmJoin(ctx context.Context, nctx *NodeContext) error {
 	return nil
 }
 
-func RunKubeadmUpgradeNode(context.Context, *NodeContext) error {
-	klog.Info("run_kubeadm_upgrade_node: TODO implement kubeadm upgrade node")
-	return nil
-}
-
 func ReconcileControlPlane(ctx context.Context, nctx *NodeContext) error {
 	if nctx.BootstrapState == nil {
 		return errors.New("BootstrapState is nil, call ClassifyBootstrapAction() first")

clitools/pkg/node/kubeadm_compat.go

@@ -0,0 +1,108 @@
+package node
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"os"
+	"strings"
+
+	"k8s.io/klog/v2"
+
+	"example.com/monok8s/pkg/system"
+)
+
+const kubeadmUpgradeNodeHostnameBugFixedIn = "v1.35.0"
+
+// COMPAT(kubeadm-upgrade-node-hostname)
+// Affects: Kubernetes/kubeadm < v1.35.0
+// Upstream: kubernetes/kubeadm#3244, kubernetes/kubernetes#134319
+// RemoveWhen: minimum supported Kubernetes version >= v1.35.0
+//
+// Affected kubeadm versions can derive the target Node name for
+// `kubeadm upgrade node` from the local OS hostname instead of the existing
+// kubeadm NodeRegistration / kubelet --hostname-override state.
+func needsKubeadmUpgradeNodeHostnameWorkaround(kubeadmVersion string) bool {
+	lt, err := versionLt(kubeadmVersion, kubeadmUpgradeNodeHostnameBugFixedIn)
+	if err != nil {
+		klog.Warningf(
+			"could not parse kubeadm version %q; enabling kubeadm upgrade node hostname workaround: %v",
+			kubeadmVersion,
+			err,
+		)
+		return true
+	}
+	return lt
+}
+
+// runWithTemporaryHostname works around kubernetes/kubeadm#3244, fixed by
+// kubernetes/kubernetes#134319 in Kubernetes v1.35.0.
+//
+// Affected kubeadm versions can derive the target Node name for
+// `kubeadm upgrade node` from the local OS hostname instead of the existing
+// kubeadm NodeRegistration / kubelet --hostname-override state. That breaks
+// valid setups where the machine hostname differs from the Kubernetes Node
+// name: kubeadm may authenticate as one node but try to get/patch another Node,
+// and the Node authorizer correctly rejects it.
+//
+// Keep this workaround scoped to affected kubeadm versions only. Set the
+// temporary hostname to the Kubernetes Node name, run kubeadm, then restore the
+// configured machine hostname immediately afterward.
+func runWithTemporaryHostname(ctx context.Context, nctx *NodeContext, fn func(context.Context) error) error {
+	if nctx == nil {
+		return errors.New("node context is nil")
+	}
+
+	temporaryHostname := strings.TrimSpace(nctx.Config.Spec.NodeName)
+	if temporaryHostname == "" {
+		return errors.New("temporary hostname is required")
+	}
+
+	originalHostname, err := os.Hostname()
+	if err != nil {
+		return fmt.Errorf("get current hostname: %w", err)
+	}
+
+	if originalHostname == temporaryHostname {
+		return fn(ctx)
+	}
+
+	restoreHostname := strings.TrimSpace(nctx.Config.Spec.Network.Hostname)
+	if restoreHostname == "" {
+		restoreHostname = originalHostname
+	}
+
+	klog.Warningf(
+		"temporarily changing hostname for kubeadm upgrade node: current=%q temporary=%q restore=%q",
+		originalHostname,
+		temporaryHostname,
+		restoreHostname,
+	)
+
+	if err := system.SetHostname(temporaryHostname); err != nil {
+		return fmt.Errorf("set temporary hostname to %q: %w", temporaryHostname, err)
+	}
+
+	defer func() {
+		if err := system.SetHostname(restoreHostname); err != nil {
+			klog.Errorf("failed to restore hostname to %q: %v", restoreHostname, err)
+		}
+	}()
+
+	return fn(ctx)
+}
+
+// COMPAT(kubeadm-upgrade-node-hostname)
+// RemoveWhen: minimum supported Kubernetes version >= v1.35.0
+func runKubeadmUpgradeNodeWithCompat(
+	ctx context.Context,
+	nctx *NodeContext,
+	kubeadmVersion string,
+	fn func(context.Context) error,
+) error {
+	if needsKubeadmUpgradeNodeHostnameWorkaround(kubeadmVersion) {
+		return runWithTemporaryHostname(ctx, nctx, fn)
+	}
+
+	return fn(ctx)
+}

clitools/pkg/node/kubeadm_upgrade.go

@@ -14,6 +14,7 @@ import (
 	"k8s.io/client-go/kubernetes"
 	"k8s.io/klog/v2"
 
+	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
 	"example.com/monok8s/pkg/kube"
 	"example.com/monok8s/pkg/system"
 )
@@ -113,7 +114,7 @@ func runUpgradeSelfHealthCheck(ctx context.Context, kubeClient kubernetes.Interf
 			Namespace: healthCheckNamespace,
 			Labels: map[string]string{
 				"app.kubernetes.io/name":       "preupgrade-health-check",
-				"app.kubernetes.io/managed-by": "monok8s",
+				"app.kubernetes.io/managed-by": monov1alpha1.NodeControlName,
 			},
 		},
 		Spec: corev1.PodSpec{
@@ -256,3 +257,102 @@ func describeHealthCheckFailure(ctx context.Context, kubeClient kubernetes.Inter
 
 	return nil
 }
+
+func RunKubeadmUpgradeNode(ctx context.Context, nctx *NodeContext) error {
+	if nctx == nil {
+		return errors.New("node context is nil")
+	}
+	if nctx.Config == nil {
+		return errors.New("node config is nil")
+	}
+	if nctx.LocalClusterState == nil {
+		return errors.New("LocalClusterState is nil. Please run earlier steps first")
+	}
+	if nctx.BootstrapState == nil {
+		return errors.New("BootstrapState is nil. Please run earlier steps first")
+	}
+
+	switch nctx.BootstrapState.Action {
+	case BootstrapActionUpgradeWorker:
+		// continue
+	default:
+		klog.V(4).Infof("RunKubeadmUpgradeNode skipped for action %q", nctx.BootstrapState.Action)
+		return nil
+	}
+
+	wantVersion := normalizeKubeVersion(strings.TrimSpace(nctx.Config.Spec.KubernetesVersion))
+	if wantVersion == "" {
+		return errors.New("spec.kubernetesVersion is required")
+	}
+
+	kubeconfigPath := chooseVersionKubeconfig(nctx.LocalClusterState)
+	if kubeconfigPath == "" {
+		return errors.New("no kubeconfig available for detecting cluster version")
+	}
+
+	clusterVersion := strings.TrimSpace(nctx.BootstrapState.DetectedClusterVersion)
+	if clusterVersion == "" {
+		var err error
+		clusterVersion, err = getServerVersion(ctx, kubeconfigPath)
+		if err != nil {
+			if nctx.BootstrapState.UnsupportedWorkerVersionSkew {
+				klog.Warningf(
+					"cluster version unavailable but worker skew was marked unsupported/permissive, continuing: reason=%s",
+					nctx.BootstrapState.VersionSkewReason,
+				)
+			} else {
+				return fmt.Errorf("get cluster version via %s: %w", kubeconfigPath, err)
+			}
+		}
+	}
+
+	if clusterVersion != "" && !isSupportedWorkerSkew(clusterVersion, wantVersion) {
+		klog.Warningf(
+			"unsupported worker version skew detected, continuing anyway: cluster=%s node=%s",
+			clusterVersion,
+			wantVersion,
+		)
+	}
+
+	klog.Infof(
+		"running kubeadm upgrade node: role=%s clusterVersion=%s nodeVersion=%s kubeconfig=%s",
+		strings.TrimSpace(nctx.Config.Spec.ClusterRole),
+		clusterVersion,
+		wantVersion,
+		kubeconfigPath,
+	)
+
+	args := []string{
+		"upgrade",
+		"node",
+		"--kubeconfig",
+		kubeconfigPath,
+	}
+
+	runKubeadm := func(ctx context.Context) error {
+		_, err := nctx.SystemRunner.RunWithOptions(
+			ctx,
+			"kubeadm",
+			args,
+			system.RunOptions{
+				Timeout: 10 * time.Minute,
+				OnStdoutLine: func(line string) {
+					klog.Infof("[kubeadm] %s", line)
+				},
+				OnStderrLine: func(line string) {
+					klog.Infof("[kubeadm] %s", line)
+				},
+			},
+		)
+		return err
+	}
+
+	// COMPAT(kubeadm-upgrade-node-hostname)
+	// RemoveWhen: minimum supported Kubernetes version >= v1.35.0
+	// Replace this wrapper with direct runKubeadm(ctx).
+	if err := runKubeadmUpgradeNodeWithCompat(ctx, nctx, wantVersion, runKubeadm); err != nil {
+		return fmt.Errorf("run kubeadm upgrade node: %w", err)
+	}
+
+	return nil
+}

clitools/pkg/node/metadata.go

@@ -17,8 +17,8 @@ import (
 func ApplyLocalNodeMetadataIfPossible(ctx context.Context, nctx *NodeContext) error {
 	spec := nctx.Config.Spec
 
-	if len(spec.NodeAnnotations) == 0 && len(spec.NodeLabels) == 0 {
+	if len(spec.NodeLabels) == 0 {
-		klog.V(4).Infof("No annotations or labels was defined")
+		klog.V(4).Infof("No labels was defined")
 		return nil // nothing to do
 	}
 
@@ -53,9 +53,6 @@ func ApplyLocalNodeMetadataIfPossible(ctx context.Context, nctx *NodeContext) er
 	if node.Labels == nil {
 		node.Labels = make(map[string]string)
 	}
-	if node.Annotations == nil {
-		node.Annotations = make(map[string]string)
-	}
 
 	// Apply labels
 	for k, v := range spec.NodeLabels {
@@ -63,13 +60,8 @@ func ApplyLocalNodeMetadataIfPossible(ctx context.Context, nctx *NodeContext) er
 	}
 
 	// Additional Labels
-	if spec.EnableControlAgent {
+	if spec.EnableNodeControl {
-		node.Labels[monov1alpah1.ControlAgentKey] = controlAgentNodeSelectorValue
+		node.Labels[monov1alpah1.NodeControlKey] = "true"
-	}
-
-	// Apply annotations
-	for k, v := range spec.NodeAnnotations {
-		node.Annotations[k] = v
 	}
 
 	_, err = client.CoreV1().Nodes().Update(ctx, node, metav1.UpdateOptions{})
@@ -77,6 +69,6 @@ func ApplyLocalNodeMetadataIfPossible(ctx context.Context, nctx *NodeContext) er
 		return fmt.Errorf("update node metadata: %w", err)
 	}
 
-	klog.Infof("applied labels/annotations to node %q", nodeName)
+	klog.Infof("applied labels to node %q", nodeName)
 	return nil
 }

clitools/pkg/node/prereqs.go

@@ -8,9 +8,18 @@ import (
 	"strings"
 	"time"
 
+	"k8s.io/client-go/discovery"
+	"k8s.io/client-go/kubernetes"
+	"k8s.io/client-go/tools/clientcmd"
 	"k8s.io/klog/v2"
 )
 
+type kubeVersion struct {
+	Major int
+	Minor int
+	Patch int
+}
+
 func ValidateNodeIPAndAPIServerReachability(ctx context.Context, nct *NodeContext) error {
 	requireLocalIP := func(wantedIP string) error {
 		wantedIP = strings.TrimSpace(wantedIP)
@@ -189,3 +198,136 @@ func CheckForVersionSkew(ctx context.Context, nctx *NodeContext) error {
 
 	return nil
 }
+
+func versionEq(a, b string) bool {
+	return normalizeKubeVersion(a) == normalizeKubeVersion(b)
+}
+
+func versionLt(a, b string) (bool, error) {
+	av, err := parseKubeVersion(a)
+	if err != nil {
+		return false, err
+	}
+
+	bv, err := parseKubeVersion(b)
+	if err != nil {
+		return false, err
+	}
+
+	if av.Major != bv.Major {
+		return av.Major < bv.Major, nil
+	}
+	if av.Minor != bv.Minor {
+		return av.Minor < bv.Minor, nil
+	}
+	return av.Patch < bv.Patch, nil
+}
+
+func normalizeKubeVersion(v string) string {
+	v = strings.TrimSpace(v)
+	if v == "" {
+		return ""
+	}
+	if !strings.HasPrefix(v, "v") {
+		v = "v" + v
+	}
+	return v
+}
+
+func parseKubeVersion(s string) (kubeVersion, error) {
+	s = strings.TrimSpace(s)
+	s = strings.TrimPrefix(s, "v")
+
+	var v kubeVersion
+	n, err := fmt.Sscanf(s, "%d.%d.%d", &v.Major, &v.Minor, &v.Patch)
+	// Accepts "1.29" or "1.29.3"
+	if err != nil || n < 2 {
+		return kubeVersion{}, fmt.Errorf("invalid kubernetes version %q", s)
+	}
+	return v, nil
+}
+
+// Control-plane: keep this strict.
+// Accept same version, or a one-minor step where the node binary is newer than the current cluster.
+// That covers normal control-plane upgrade flow but blocks nonsense.
+func isSupportedControlPlaneSkew(clusterVersion, nodeVersion string) bool {
+	cv, err := parseKubeVersion(clusterVersion)
+	if err != nil {
+		return false
+	}
+	nv, err := parseKubeVersion(nodeVersion)
+	if err != nil {
+		return false
+	}
+
+	if cv.Major != nv.Major {
+		return false
+	}
+	if cv.Minor == nv.Minor {
+		return true
+	}
+	if nv.Minor == cv.Minor+1 {
+		return true
+	}
+	return false
+}
+
+// Worker: kubelet generally must not be newer than the apiserver.
+// Older kubelets are allowed within supported skew range.
+// Your requirement says unsupported worker skew should still proceed, so this
+// only classifies support status and must NOT be used to block this function.
+func isSupportedWorkerSkew(clusterVersion, nodeVersion string) bool {
+	cv, err := parseKubeVersion(clusterVersion)
+	if err != nil {
+		return false
+	}
+	nv, err := parseKubeVersion(nodeVersion)
+	if err != nil {
+		return false
+	}
+
+	if cv.Major != nv.Major {
+		return false
+	}
+
+	// kubelet newer than apiserver => unsupported
+	if nv.Minor > cv.Minor {
+		return false
+	}
+
+	// kubelet up to 3 minors older than apiserver => supported
+	if cv.Minor-nv.Minor <= 3 {
+		return true
+	}
+
+	return false
+}
+
+func getServerVersion(ctx context.Context, kubeconfigPath string) (string, error) {
+	restCfg, err := clientcmd.BuildConfigFromFlags("", kubeconfigPath)
+	if err != nil {
+		return "", fmt.Errorf("build kubeconfig %s: %w", kubeconfigPath, err)
+	}
+
+	// Keep this short. This is a probe, not a long-running client.
+	restCfg.Timeout = 5 * time.Second
+
+	clientset, err := kubernetes.NewForConfig(restCfg)
+	if err != nil {
+		return "", fmt.Errorf("create clientset: %w", err)
+	}
+
+	disc := clientset.Discovery()
+	return discoverServerVersion(ctx, disc)
+}
+
+func discoverServerVersion(ctx context.Context, disc discovery.DiscoveryInterface) (string, error) {
+	info, err := disc.ServerVersion()
+	if err != nil {
+		return "", err
+	}
+	if info == nil || strings.TrimSpace(info.GitVersion) == "" {
+		return "", errors.New("server version is empty")
+	}
+	return normalizeKubeVersion(info.GitVersion), nil
+}

clitools/pkg/render/agent.go

@@ -0,0 +1,284 @@
+package render
+
+import (
+	"fmt"
+	"strings"
+
+	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
+	rbacv1 "k8s.io/api/rbac/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+
+	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	buildinfo "example.com/monok8s/pkg/buildinfo"
+)
+
+type AgentConf struct {
+	Namespace        string
+	Image            string
+	ImagePullSecrets []string
+	Labels           map[string]string
+}
+
+func RenderAgentDaemonSets(conf AgentConf) (string, error) {
+	objs, err := buildAgentDaemonSetObjects(conf)
+	if err != nil {
+		return "", err
+	}
+
+	return renderObjects(objs)
+}
+
+func buildAgentDaemonSetObjects(conf AgentConf) ([]runtime.Object, error) {
+	if strings.TrimSpace(conf.Namespace) == "" {
+		return nil, fmt.Errorf("namespace is required")
+	}
+
+	conf.Labels = map[string]string{
+		"app.kubernetes.io/name":       monov1alpha1.NodeAgentName,
+		"app.kubernetes.io/component":  "agent",
+		"app.kubernetes.io/part-of":    "monok8s",
+		"app.kubernetes.io/managed-by": monov1alpha1.NodeControlName,
+	}
+
+	return []runtime.Object{
+		buildAgentServiceAccount(conf),
+		buildAgentClusterRole(conf),
+		buildAgentClusterRoleBinding(conf),
+		buildAgentDaemonSet(conf),
+	}, nil
+}
+
+func buildAgentNamespace(conf AgentConf) *corev1.Namespace {
+	return &corev1.Namespace{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "v1",
+			Kind:       "Namespace",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:   conf.Namespace,
+			Labels: copyStringMap(conf.Labels),
+		},
+	}
+}
+
+func buildAgentServiceAccount(conf AgentConf) *corev1.ServiceAccount {
+	return &corev1.ServiceAccount{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "v1",
+			Kind:       "ServiceAccount",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      monov1alpha1.NodeAgentName,
+			Namespace: conf.Namespace,
+			Labels:    copyStringMap(conf.Labels),
+		},
+	}
+}
+
+func buildAgentClusterRole(conf AgentConf) *rbacv1.ClusterRole {
+	wantRules := []rbacv1.PolicyRule{
+		{
+			APIGroups: []string{monov1alpha1.Group},
+			Resources: []string{"osupgrades"},
+			Verbs:     []string{"get"},
+		},
+		{
+			APIGroups: []string{monov1alpha1.Group},
+			Resources: []string{"osupgradeprogresses"},
+			Verbs:     []string{"get", "list", "watch", "create", "patch", "update"},
+		},
+		{
+			APIGroups: []string{monov1alpha1.Group},
+			Resources: []string{"osupgradeprogresses/status"},
+			Verbs:     []string{"get", "list", "watch", "create", "patch", "update"},
+		},
+		{
+			APIGroups: []string{""},
+			Resources: []string{"nodes"},
+			Verbs:     []string{"get", "list", "watch"},
+		},
+	}
+
+	return &rbacv1.ClusterRole{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "rbac.authorization.k8s.io/v1",
+			Kind:       "ClusterRole",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:   monov1alpha1.NodeAgentName,
+			Labels: copyStringMap(conf.Labels),
+		},
+		Rules: wantRules,
+	}
+}
+
+func buildAgentClusterRoleBinding(conf AgentConf) *rbacv1.ClusterRoleBinding {
+	return &rbacv1.ClusterRoleBinding{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "rbac.authorization.k8s.io/v1",
+			Kind:       "ClusterRoleBinding",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:   monov1alpha1.NodeAgentName,
+			Labels: copyStringMap(conf.Labels),
+		},
+		RoleRef: rbacv1.RoleRef{
+			APIGroup: rbacv1.GroupName,
+			Kind:     "ClusterRole",
+			Name:     monov1alpha1.NodeAgentName,
+		},
+		Subjects: []rbacv1.Subject{
+			{
+				Kind:      "ServiceAccount",
+				Name:      monov1alpha1.NodeAgentName,
+				Namespace: conf.Namespace,
+			},
+		},
+	}
+}
+
+func buildAgentDaemonSet(conf AgentConf) *appsv1.DaemonSet {
+	privileged := true
+	dsLabels := monov1alpha1.NodeAgentLabels()
+
+	image, pullPolicy := agentImage(conf)
+
+	return &appsv1.DaemonSet{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "apps/v1",
+			Kind:       "DaemonSet",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      monov1alpha1.NodeAgentName,
+			Namespace: conf.Namespace,
+			Labels:    copyStringMap(conf.Labels),
+		},
+		Spec: appsv1.DaemonSetSpec{
+			Selector: &metav1.LabelSelector{
+				MatchLabels: map[string]string{
+					"app.kubernetes.io/name": monov1alpha1.NodeAgentName,
+				},
+			},
+			Template: corev1.PodTemplateSpec{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: dsLabels,
+				},
+				Spec: corev1.PodSpec{
+					ServiceAccountName: monov1alpha1.NodeAgentName,
+					HostNetwork:        true,
+					HostPID:            true,
+					DNSPolicy:          corev1.DNSClusterFirstWithHostNet,
+					ImagePullSecrets:   imagePullSecrets(conf.ImagePullSecrets),
+					NodeSelector: map[string]string{
+						monov1alpha1.NodeControlKey: "true",
+					},
+					Tolerations: []corev1.Toleration{
+						{Operator: corev1.TolerationOpExists},
+					},
+					Containers: []corev1.Container{
+						{
+							Name:            "agent",
+							Image:           image,
+							ImagePullPolicy: pullPolicy,
+							Args:            []string{"agent", "--env-file", "$(CLUSTER_ENV_FILE)"},
+							Env: []corev1.EnvVar{
+								{
+									Name: "NODE_NAME",
+									ValueFrom: &corev1.EnvVarSource{
+										FieldRef: &corev1.ObjectFieldSelector{
+											APIVersion: "v1",
+											FieldPath:  "spec.nodeName",
+										},
+									},
+								},
+								{
+									Name:  "CLUSTER_ENV_FILE",
+									Value: "/host/opt/monok8s/config/cluster.env",
+								},
+								{
+									Name:  "FW_ENV_CONFIG_FILE",
+									Value: "/host/etc/fw_env.config",
+								},
+							},
+							SecurityContext: &corev1.SecurityContext{
+								Privileged: &privileged,
+							},
+							VolumeMounts: []corev1.VolumeMount{
+								{
+									Name:      "host-dev",
+									MountPath: "/dev",
+								},
+								{
+									Name:      "host-etc",
+									MountPath: "/host/etc",
+									ReadOnly:  true,
+								},
+								{
+									Name:      "host-config",
+									MountPath: "/host/opt/monok8s/config",
+									ReadOnly:  true,
+								},
+							},
+						},
+					},
+					Volumes: []corev1.Volume{
+						{
+							Name: "host-dev",
+							VolumeSource: corev1.VolumeSource{
+								HostPath: &corev1.HostPathVolumeSource{
+									Path: "/dev",
+									Type: hostPathType(corev1.HostPathDirectory),
+								},
+							},
+						},
+						{
+							Name: "host-etc",
+							VolumeSource: corev1.VolumeSource{
+								HostPath: &corev1.HostPathVolumeSource{
+									Path: "/etc",
+									Type: hostPathType(corev1.HostPathDirectory),
+								},
+							},
+						},
+						{
+							Name: "host-config",
+							VolumeSource: corev1.VolumeSource{
+								HostPath: &corev1.HostPathVolumeSource{
+									Path: "/opt/monok8s/config",
+									Type: hostPathType(corev1.HostPathDirectory),
+								},
+							},
+						},
+					},
+				},
+			},
+		},
+	}
+}
+
+func agentImage(conf AgentConf) (string, corev1.PullPolicy) {
+	if conf.Image != "" {
+		return conf.Image, corev1.PullIfNotPresent
+	}
+
+	return fmt.Sprintf("localhost/monok8s/node-control:%s", buildinfo.Version), corev1.PullNever
+}
+
+func copyStringMap(in map[string]string) map[string]string {
+	if len(in) == 0 {
+		return nil
+	}
+
+	out := make(map[string]string, len(in))
+	for k, v := range in {
+		out[k] = v
+	}
+
+	return out
+}
+
+func hostPathType(t corev1.HostPathType) *corev1.HostPathType {
+	return &t
+}

clitools/pkg/render/agent_apply.go

@@ -0,0 +1,203 @@
+package render
+
+import (
+	"context"
+	"fmt"
+	"reflect"
+
+	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
+	rbacv1 "k8s.io/api/rbac/v1"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/client-go/kubernetes"
+)
+
+func ApplyAgentDaemonSets(ctx context.Context, kubeClient kubernetes.Interface, conf AgentConf) error {
+	objs, err := buildAgentDaemonSetObjects(conf)
+	if err != nil {
+		return err
+	}
+
+	if err := applyAgentNamespace(ctx, kubeClient, buildAgentNamespace(conf)); err != nil {
+		return fmt.Errorf("apply namespace: %w", err)
+	}
+
+	for _, obj := range objs {
+		if err := applyAgentObject(ctx, kubeClient, obj); err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func applyAgentObject(ctx context.Context, kubeClient kubernetes.Interface, obj runtime.Object) error {
+	switch want := obj.(type) {
+	case *corev1.ServiceAccount:
+		return applyAgentServiceAccount(ctx, kubeClient, want)
+	case *rbacv1.ClusterRole:
+		return applyAgentClusterRole(ctx, kubeClient, want)
+	case *rbacv1.ClusterRoleBinding:
+		return applyAgentClusterRoleBinding(ctx, kubeClient, want)
+	case *appsv1.DaemonSet:
+		return applyAgentDaemonSet(ctx, kubeClient, want)
+	default:
+		return fmt.Errorf("unsupported agent object type %T", obj)
+	}
+}
+
+func applyAgentNamespace(ctx context.Context, kubeClient kubernetes.Interface, want *corev1.Namespace) error {
+	existing, err := kubeClient.CoreV1().Namespaces().Get(ctx, want.Name, metav1.GetOptions{})
+	if apierrors.IsNotFound(err) {
+		_, err = kubeClient.CoreV1().Namespaces().Create(ctx, want, metav1.CreateOptions{})
+		return err
+	}
+	if err != nil {
+		return err
+	}
+
+	labels, changed := mergeStringMapsInto(existing.Labels, want.Labels)
+	if !changed {
+		return nil
+	}
+
+	existing.Labels = labels
+	_, err = kubeClient.CoreV1().Namespaces().Update(ctx, existing, metav1.UpdateOptions{})
+	return err
+}
+
+func applyAgentServiceAccount(ctx context.Context, kubeClient kubernetes.Interface, want *corev1.ServiceAccount) error {
+	existing, err := kubeClient.CoreV1().ServiceAccounts(want.Namespace).Get(ctx, want.Name, metav1.GetOptions{})
+	if apierrors.IsNotFound(err) {
+		_, err = kubeClient.CoreV1().ServiceAccounts(want.Namespace).Create(ctx, want, metav1.CreateOptions{})
+		return err
+	}
+	if err != nil {
+		return err
+	}
+
+	changed := false
+	if !reflect.DeepEqual(existing.Labels, want.Labels) {
+		existing.Labels = want.Labels
+		changed = true
+	}
+
+	if !changed {
+		return nil
+	}
+
+	_, err = kubeClient.CoreV1().ServiceAccounts(want.Namespace).Update(ctx, existing, metav1.UpdateOptions{})
+	return err
+}
+
+func applyAgentClusterRole(ctx context.Context, kubeClient kubernetes.Interface, want *rbacv1.ClusterRole) error {
+	existing, err := kubeClient.RbacV1().ClusterRoles().Get(ctx, want.Name, metav1.GetOptions{})
+	if apierrors.IsNotFound(err) {
+		_, err = kubeClient.RbacV1().ClusterRoles().Create(ctx, want, metav1.CreateOptions{})
+		return err
+	}
+	if err != nil {
+		return err
+	}
+
+	changed := false
+	if !reflect.DeepEqual(existing.Labels, want.Labels) {
+		existing.Labels = want.Labels
+		changed = true
+	}
+	if !reflect.DeepEqual(existing.Rules, want.Rules) {
+		existing.Rules = want.Rules
+		changed = true
+	}
+
+	if !changed {
+		return nil
+	}
+
+	_, err = kubeClient.RbacV1().ClusterRoles().Update(ctx, existing, metav1.UpdateOptions{})
+	return err
+}
+
+func applyAgentClusterRoleBinding(ctx context.Context, kubeClient kubernetes.Interface, want *rbacv1.ClusterRoleBinding) error {
+	existing, err := kubeClient.RbacV1().ClusterRoleBindings().Get(ctx, want.Name, metav1.GetOptions{})
+	if apierrors.IsNotFound(err) {
+		_, err = kubeClient.RbacV1().ClusterRoleBindings().Create(ctx, want, metav1.CreateOptions{})
+		return err
+	}
+	if err != nil {
+		return err
+	}
+
+	// roleRef is immutable. If it differs, fail loudly instead of pretending we can patch it.
+	if !reflect.DeepEqual(existing.RoleRef, want.RoleRef) {
+		return fmt.Errorf("existing ClusterRoleBinding %q has different roleRef and must be recreated", want.Name)
+	}
+
+	changed := false
+	if !reflect.DeepEqual(existing.Labels, want.Labels) {
+		existing.Labels = want.Labels
+		changed = true
+	}
+	if !reflect.DeepEqual(existing.Subjects, want.Subjects) {
+		existing.Subjects = want.Subjects
+		changed = true
+	}
+
+	if !changed {
+		return nil
+	}
+
+	_, err = kubeClient.RbacV1().ClusterRoleBindings().Update(ctx, existing, metav1.UpdateOptions{})
+	return err
+}
+
+func applyAgentDaemonSet(ctx context.Context, kubeClient kubernetes.Interface, want *appsv1.DaemonSet) error {
+	existing, err := kubeClient.AppsV1().DaemonSets(want.Namespace).Get(ctx, want.Name, metav1.GetOptions{})
+	if apierrors.IsNotFound(err) {
+		_, err = kubeClient.AppsV1().DaemonSets(want.Namespace).Create(ctx, want, metav1.CreateOptions{})
+		return err
+	}
+	if err != nil {
+		return err
+	}
+
+	changed := false
+	if !reflect.DeepEqual(existing.Labels, want.Labels) {
+		existing.Labels = want.Labels
+		changed = true
+	}
+	if !reflect.DeepEqual(existing.Spec, want.Spec) {
+		existing.Spec = want.Spec
+		changed = true
+	}
+
+	if !changed {
+		return nil
+	}
+
+	_, err = kubeClient.AppsV1().DaemonSets(want.Namespace).Update(ctx, existing, metav1.UpdateOptions{})
+	return err
+}
+
+func mergeStringMapsInto(dst map[string]string, src map[string]string) (map[string]string, bool) {
+	if len(src) == 0 {
+		return dst, false
+	}
+
+	changed := false
+	if dst == nil {
+		dst = map[string]string{}
+		changed = true
+	}
+
+	for k, v := range src {
+		if dst[k] != v {
+			dst[k] = v
+			changed = true
+		}
+	}
+
+	return dst, changed
+}

clitools/pkg/render/controller.go

@@ -0,0 +1,324 @@
+package render
+
+import (
+	"fmt"
+
+	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
+	rbacv1 "k8s.io/api/rbac/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/util/intstr"
+
+	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	buildinfo "example.com/monok8s/pkg/buildinfo"
+)
+
+type ControllerConf struct {
+	Namespace        string
+	Image            string
+	ImagePullSecrets []string
+	Labels           map[string]string
+}
+
+func RenderControllerDeployments(conf ControllerConf) (string, error) {
+	if conf.Namespace == "" {
+		return "", fmt.Errorf("namespace is required")
+	}
+
+	conf.Labels = map[string]string{
+		"app.kubernetes.io/name":       monov1alpha1.ControllerName,
+		"app.kubernetes.io/component":  "controller",
+		"app.kubernetes.io/part-of":    "monok8s",
+		"app.kubernetes.io/managed-by": monov1alpha1.NodeControlName,
+	}
+
+	objs := []runtime.Object{
+		buildControllerServiceAccount(conf),
+		buildControllerClusterRole(conf),
+		buildControllerClusterRoleBinding(conf),
+		buildControllerDeployment(conf),
+	}
+
+	return renderObjects(objs)
+}
+
+func buildControllerServiceAccount(conf ControllerConf) *corev1.ServiceAccount {
+
+	automount := true
+
+	return &corev1.ServiceAccount{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "v1",
+			Kind:       "ServiceAccount",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      monov1alpha1.ControllerName,
+			Namespace: conf.Namespace,
+			Labels:    conf.Labels,
+		},
+		AutomountServiceAccountToken: &automount,
+	}
+}
+
+func buildControllerClusterRole(conf ControllerConf) *rbacv1.ClusterRole {
+	wantRules := []rbacv1.PolicyRule{
+		{
+			APIGroups: []string{monov1alpha1.Group},
+			Resources: []string{"osupgrades"},
+			Verbs:     []string{"get", "list", "watch"},
+		},
+		{
+			APIGroups: []string{monov1alpha1.Group},
+			Resources: []string{"osupgrades/status"},
+			Verbs:     []string{"get", "patch", "update"},
+		},
+		{
+			APIGroups: []string{monov1alpha1.Group},
+			Resources: []string{"osupgradeprogresses"},
+			Verbs:     []string{"get", "create"},
+		},
+		{
+			APIGroups: []string{monov1alpha1.Group},
+			Resources: []string{"osupgradeprogresses/status"},
+			Verbs:     []string{"update"},
+		},
+		{
+			APIGroups: []string{""},
+			Resources: []string{"nodes"},
+			Verbs:     []string{"get", "list"},
+		},
+	}
+
+	return &rbacv1.ClusterRole{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "rbac.authorization.k8s.io/v1",
+			Kind:       "ClusterRole",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:   monov1alpha1.ControllerName,
+			Labels: conf.Labels,
+		},
+		Rules: wantRules,
+	}
+}
+
+func buildControllerClusterRoleBinding(conf ControllerConf) *rbacv1.ClusterRoleBinding {
+
+	wantSubjects := []rbacv1.Subject{
+		{
+			Kind:      "ServiceAccount",
+			Name:      monov1alpha1.ControllerName,
+			Namespace: conf.Namespace,
+		},
+	}
+
+	wantRoleRef := rbacv1.RoleRef{
+		APIGroup: rbacv1.GroupName,
+		Kind:     "ClusterRole",
+		Name:     monov1alpha1.ControllerName,
+	}
+
+	return &rbacv1.ClusterRoleBinding{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "rbac.authorization.k8s.io/v1",
+			Kind:       "ClusterRoleBinding",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:   monov1alpha1.ControllerName,
+			Labels: conf.Labels,
+		},
+		Subjects: wantSubjects,
+		RoleRef:  wantRoleRef,
+	}
+}
+
+func buildControllerDeployment(conf ControllerConf) *appsv1.Deployment {
+	replicas := int32(1)
+
+	selectorLabels := map[string]string{
+		"app.kubernetes.io/name":      monov1alpha1.ControllerName,
+		"app.kubernetes.io/component": "controller",
+	}
+
+	podLabels := mergeStringMaps(conf.Labels, selectorLabels)
+
+	runAsNonRoot := true
+	allowPrivilegeEscalation := false
+	userGroup := int64(65532)
+
+	image, pullPolicy := controllerImage(conf)
+
+	return &appsv1.Deployment{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "apps/v1",
+			Kind:       "Deployment",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      monov1alpha1.ControllerName,
+			Namespace: conf.Namespace,
+			Labels:    conf.Labels,
+		},
+		Spec: appsv1.DeploymentSpec{
+			Replicas: &replicas,
+			Selector: &metav1.LabelSelector{
+				MatchLabels: selectorLabels,
+			},
+			Template: corev1.PodTemplateSpec{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: podLabels,
+				},
+				Spec: corev1.PodSpec{
+					ServiceAccountName: monov1alpha1.ControllerName,
+					ImagePullSecrets:   imagePullSecrets(conf.ImagePullSecrets),
+					Containers: []corev1.Container{
+						{
+							Name:            "controller",
+							Image:           image,
+							ImagePullPolicy: pullPolicy,
+							Args: []string{
+								"controller",
+								"--namespace",
+								conf.Namespace,
+							},
+							Env: []corev1.EnvVar{
+								{
+									Name: "POD_NAME",
+									ValueFrom: &corev1.EnvVarSource{
+										FieldRef: &corev1.ObjectFieldSelector{
+											APIVersion: "v1",
+											FieldPath:  "metadata.name",
+										},
+									},
+								},
+								{
+									Name: "POD_NAMESPACE",
+									ValueFrom: &corev1.EnvVarSource{
+										FieldRef: &corev1.ObjectFieldSelector{
+											APIVersion: "v1",
+											FieldPath:  "metadata.namespace",
+										},
+									},
+								},
+								{
+									Name: "NODE_NAME",
+									ValueFrom: &corev1.EnvVarSource{
+										FieldRef: &corev1.ObjectFieldSelector{
+											APIVersion: "v1",
+											FieldPath:  "spec.nodeName",
+										},
+									},
+								},
+							},
+							Ports: []corev1.ContainerPort{
+								{
+									Name:          "http",
+									ContainerPort: 8080,
+									Protocol:      corev1.ProtocolTCP,
+								},
+								{
+									Name:          "https",
+									ContainerPort: 8443,
+									Protocol:      corev1.ProtocolTCP,
+								},
+							},
+							LivenessProbe: &corev1.Probe{
+								ProbeHandler: corev1.ProbeHandler{
+									HTTPGet: &corev1.HTTPGetAction{
+										Path: "/healthz",
+										Port: intstr.FromString("http"),
+									},
+								},
+								InitialDelaySeconds: 5,
+								PeriodSeconds:       60,
+								TimeoutSeconds:      2,
+								FailureThreshold:    3,
+							},
+							ReadinessProbe: &corev1.Probe{
+								ProbeHandler: corev1.ProbeHandler{
+									HTTPGet: &corev1.HTTPGetAction{
+										Path: "/readyz",
+										Port: intstr.FromString("http"),
+									},
+								},
+								InitialDelaySeconds: 2,
+								PeriodSeconds:       5,
+								TimeoutSeconds:      2,
+								FailureThreshold:    3,
+							},
+							SecurityContext: &corev1.SecurityContext{
+								RunAsNonRoot:             &runAsNonRoot,
+								RunAsUser:                &userGroup,
+								RunAsGroup:               &userGroup,
+								AllowPrivilegeEscalation: &allowPrivilegeEscalation,
+							},
+						},
+					},
+					NodeSelector: controllerNodeSelector(conf),
+					Affinity:     controllerAffinity(conf),
+				},
+			},
+		},
+	}
+}
+
+func controllerImage(conf ControllerConf) (string, corev1.PullPolicy) {
+	if conf.Image != "" {
+		return conf.Image, corev1.PullIfNotPresent
+	}
+
+	return fmt.Sprintf("localhost/monok8s/node-control:%s", buildinfo.Version), corev1.PullNever
+}
+
+func controllerNodeSelector(conf ControllerConf) map[string]string {
+	if conf.Image != "" {
+		return nil
+	}
+
+	// Local image exists on managed nodes only.
+	return map[string]string{
+		monov1alpha1.NodeControlKey: "true",
+	}
+}
+
+func controllerAffinity(conf ControllerConf) *corev1.Affinity {
+	// Local image exists only on managed nodes, so in that mode we already use
+	// NodeSelector and should not fight placement with anti-affinity.
+	if conf.Image == "" {
+		return nil
+	}
+
+	return &corev1.Affinity{
+		PodAntiAffinity: &corev1.PodAntiAffinity{
+			PreferredDuringSchedulingIgnoredDuringExecution: []corev1.WeightedPodAffinityTerm{
+				{
+					Weight: 100,
+					PodAffinityTerm: corev1.PodAffinityTerm{
+						TopologyKey: corev1.LabelHostname,
+						LabelSelector: &metav1.LabelSelector{
+							MatchLabels: monov1alpha1.NodeAgentLabels(),
+						},
+					},
+				},
+			},
+		},
+	}
+}
+
+func mergeStringMaps(maps ...map[string]string) map[string]string {
+	var total int
+	for _, m := range maps {
+		total += len(m)
+	}
+	if total == 0 {
+		return nil
+	}
+
+	out := make(map[string]string, total)
+	for _, m := range maps {
+		for k, v := range m {
+			out[k] = v
+		}
+	}
+	return out
+}

clitools/pkg/render/helpers.go

@@ -0,0 +1,74 @@
+package render
+
+import (
+	"bytes"
+	"fmt"
+	"strings"
+
+	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"sigs.k8s.io/yaml"
+)
+
+func renderObjects(objs []runtime.Object) (string, error) {
+	var buf bytes.Buffer
+
+	for i, obj := range objs {
+		if i > 0 {
+			if _, err := fmt.Fprintln(&buf, "---"); err != nil {
+				return "", err
+			}
+		}
+
+		b, err := renderObjectYAML(obj)
+		if err != nil {
+			return "", err
+		}
+
+		if _, err := buf.Write(b); err != nil {
+			return "", err
+		}
+	}
+
+	return buf.String(), nil
+}
+
+func renderObjectYAML(obj runtime.Object) ([]byte, error) {
+	b, err := yaml.Marshal(obj)
+	if err != nil {
+		return nil, err
+	}
+
+	var m map[string]any
+	if err := yaml.Unmarshal(b, &m); err != nil {
+		return nil, err
+	}
+
+	delete(m, "status")
+
+	return yaml.Marshal(m)
+}
+
+func imagePullSecrets(names []string) []corev1.LocalObjectReference {
+	if len(names) == 0 {
+		return nil
+	}
+
+	refs := make([]corev1.LocalObjectReference, 0, len(names))
+	for _, name := range names {
+		name = strings.TrimSpace(name)
+		if name == "" {
+			continue
+		}
+
+		refs = append(refs, corev1.LocalObjectReference{
+			Name: name,
+		})
+	}
+
+	if len(refs) == 0 {
+		return nil
+	}
+
+	return refs
+}

clitools/pkg/render/monoks.go

@@ -1,4 +1,4 @@
-package templates
+package render
 
 import (
 	"bytes"
@@ -31,10 +31,12 @@ func RenderMonoKSConfig() (string, error) {
 	return buf.String(), nil
 }
 
-func RenderOSUpgrade() (string, error) {
+func RenderOSUpgrade(namespace string) (string, error) {
 	vals := templates.LoadTemplateValuesFromEnv()
 	cfg := templates.DefaultOSUpgrade(vals)
 
+	cfg.Namespace = namespace
+
 	s := runtime.NewScheme()
 	if err := monov1alpha1.AddToScheme(s); err != nil {
 		return "", err

clitools/pkg/render/sshd.go

@@ -0,0 +1,304 @@
+package render
+
+import (
+	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/resource"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/util/intstr"
+
+	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	"example.com/monok8s/pkg/templates"
+)
+
+const (
+	sshdName       = "sshd"
+	sshdConfigName = "sshd-authorized-keys"
+	sshdNodePort   = int32(30022)
+)
+
+func RenderSSHDDeployments(namespace, authKeys string) (string, error) {
+	vals := templates.LoadTemplateValuesFromEnv()
+
+	labels := map[string]string{
+		"app.kubernetes.io/name":       sshdName,
+		"app.kubernetes.io/component":  "host-access",
+		"app.kubernetes.io/part-of":    "monok8s",
+		"app.kubernetes.io/managed-by": monov1alpha1.NodeControlName,
+	}
+
+	objs := []runtime.Object{
+		buildSSHDConfigMap(authKeys, namespace, labels),
+		buildSSHDService(vals, namespace, labels),
+		buildSSHDDeployment(vals, namespace, labels),
+	}
+
+	return renderObjects(objs)
+}
+
+func buildSSHDConfigMap(
+	authorizedKeys string,
+	namespace string,
+	labels map[string]string,
+) *corev1.ConfigMap {
+
+	return &corev1.ConfigMap{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "v1",
+			Kind:       "ConfigMap",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      sshdConfigName,
+			Namespace: namespace,
+			Labels:    labels,
+		},
+		Data: map[string]string{
+			"authorized_keys": authorizedKeys,
+		},
+	}
+}
+
+func buildSSHDService(
+	tVals templates.TemplateValues,
+	namespace string,
+	labels map[string]string,
+) *corev1.Service {
+	selectorLabels := map[string]string{
+		monov1alpha1.NodeControlKey: "true",
+		"kubernetes.io/hostname":    tVals.NodeName,
+	}
+
+	return &corev1.Service{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "v1",
+			Kind:       "Service",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      sshdName,
+			Namespace: namespace,
+			Labels:    labels,
+		},
+		Spec: corev1.ServiceSpec{
+			Type:     corev1.ServiceTypeNodePort,
+			Selector: selectorLabels,
+			Ports: []corev1.ServicePort{
+				{
+					Name:       "ssh",
+					Protocol:   corev1.ProtocolTCP,
+					Port:       22,
+					TargetPort: intstr.FromInt32(22),
+					NodePort:   sshdNodePort,
+				},
+			},
+		},
+	}
+}
+
+func buildSSHDDeployment(
+	tVals templates.TemplateValues,
+	namespace string,
+	labels map[string]string,
+) *appsv1.Deployment {
+	replicas := int32(1)
+
+	selectorLabels := map[string]string{
+		monov1alpha1.NodeControlKey: "true",
+		"kubernetes.io/hostname":    tVals.NodeName,
+	}
+
+	podLabels := mergeStringMaps(labels, selectorLabels)
+
+	runAsUser := int64(0)
+	runAsNonRoot := false
+	privileged := true
+	allowPrivilegeEscalation := true
+	readOnlyRootFilesystem := false
+
+	return &appsv1.Deployment{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "apps/v1",
+			Kind:       "Deployment",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      sshdName,
+			Namespace: namespace,
+			Labels:    labels,
+		},
+		Spec: appsv1.DeploymentSpec{
+			Replicas: &replicas,
+			Selector: &metav1.LabelSelector{
+				MatchLabels: selectorLabels,
+			},
+			Template: corev1.PodTemplateSpec{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: podLabels,
+				},
+				Spec: corev1.PodSpec{
+					HostPID:      true,
+					NodeSelector: selectorLabels,
+					Containers: []corev1.Container{
+						{
+							Name:  sshdName,
+							Image: "alpine:latest",
+							Command: []string{
+								"/bin/sh",
+								"-ceu",
+								`
+apk add --no-cache openssh-server
+
+mkdir -p /run/sshd
+mkdir -p /root/.ssh
+
+cp /authorized-keys/authorized_keys /root/.ssh/authorized_keys
+chmod 700 /root/.ssh
+chmod 600 /root/.ssh/authorized_keys
+
+ssh-keygen -A
+
+exec /usr/sbin/sshd \
+  -D \
+  -e \
+  -p 22 \
+  -o PermitRootLogin=prohibit-password \
+  -o PasswordAuthentication=no \
+  -o KbdInteractiveAuthentication=no \
+  -o PubkeyAuthentication=yes \
+  -o AuthorizedKeysFile=/root/.ssh/authorized_keys
+`,
+							},
+							Ports: []corev1.ContainerPort{
+								{
+									Name:          "ssh",
+									ContainerPort: 22,
+									Protocol:      corev1.ProtocolTCP,
+								},
+							},
+							SecurityContext: &corev1.SecurityContext{
+								RunAsUser:                &runAsUser,
+								RunAsNonRoot:             &runAsNonRoot,
+								Privileged:               &privileged,
+								AllowPrivilegeEscalation: &allowPrivilegeEscalation,
+								ReadOnlyRootFilesystem:   &readOnlyRootFilesystem,
+							},
+							Resources: corev1.ResourceRequirements{
+								Requests: corev1.ResourceList{
+									corev1.ResourceCPU:    resource.MustParse("10m"),
+									corev1.ResourceMemory: resource.MustParse("32Mi"),
+								},
+								Limits: corev1.ResourceList{
+									corev1.ResourceCPU:    resource.MustParse("200m"),
+									corev1.ResourceMemory: resource.MustParse("128Mi"),
+								},
+							},
+							VolumeMounts: append(
+								[]corev1.VolumeMount{
+									{
+										Name:      "authorized-keys",
+										MountPath: "/authorized-keys",
+										ReadOnly:  true,
+									},
+								},
+								buildHostRootVolumeMounts()...,
+							),
+						},
+					},
+					Volumes: append(
+						[]corev1.Volume{
+							{
+								Name: "authorized-keys",
+								VolumeSource: corev1.VolumeSource{
+									ConfigMap: &corev1.ConfigMapVolumeSource{
+										LocalObjectReference: corev1.LocalObjectReference{
+											Name: sshdConfigName,
+										},
+										DefaultMode: ptrInt32(0600),
+									},
+								},
+							},
+						},
+						buildHostRootVolumes()...,
+					),
+				},
+			},
+		},
+	}
+}
+
+func buildHostRootVolumeMounts() []corev1.VolumeMount {
+	paths := []struct {
+		name      string
+		mountPath string
+		readOnly  bool
+	}{
+		{"host-bin", "/host/bin", true},
+		{"host-sbin", "/host/sbin", true},
+		{"host-lib", "/host/lib", true},
+		{"host-usr", "/host/usr", true},
+		{"host-etc", "/host/etc", false},
+		{"host-run", "/host/run", false},
+		{"host-proc", "/host/proc", false},
+		{"host-sys", "/host/sys", false},
+		{"host-dev", "/host/dev", false},
+		{"host-var", "/host/var", false},
+	}
+
+	mounts := make([]corev1.VolumeMount, 0, len(paths))
+
+	for _, p := range paths {
+		mounts = append(mounts, corev1.VolumeMount{
+			Name:      p.name,
+			MountPath: p.mountPath,
+			ReadOnly:  p.readOnly,
+		})
+	}
+
+	return mounts
+}
+
+func buildHostRootVolumes() []corev1.Volume {
+	hostPathDir := corev1.HostPathDirectory
+
+	paths := []struct {
+		name string
+		path string
+	}{
+		{"host-bin", "/bin"},
+		{"host-sbin", "/sbin"},
+		{"host-lib", "/lib"},
+		{"host-usr", "/usr"},
+		{"host-etc", "/etc"},
+		{"host-run", "/run"},
+		{"host-proc", "/proc"},
+		{"host-sys", "/sys"},
+		{"host-dev", "/dev"},
+
+		// /var is an rbind mount in monok8s and may be private.
+		// Mount the real backing path instead.
+		{"host-var", "/data/var"},
+	}
+
+	volumes := make([]corev1.Volume, 0, len(paths))
+
+	for _, p := range paths {
+		volumes = append(volumes, corev1.Volume{
+			Name: p.name,
+			VolumeSource: corev1.VolumeSource{
+				HostPath: &corev1.HostPathVolumeSource{
+					Path: p.path,
+					Type: &hostPathDir,
+				},
+			},
+		})
+	}
+
+	return volumes
+}
+
+func ptrInt32(v int32) *int32 {
+	return &v
+}
+
+func ptrHostPathType(v corev1.HostPathType) *corev1.HostPathType {
+	return &v
+}

clitools/pkg/templates/templates.go

@@ -7,7 +7,7 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
 
-const DefaultNamespace = "kube-system"
+const DefaultNamespace = "mono-system"
 
 func DefaultMonoKSConfig(v TemplateValues) monov1alpha1.MonoKSConfig {
 	return monov1alpha1.MonoKSConfig{
@@ -25,7 +25,7 @@ func DefaultMonoKSConfig(v TemplateValues) monov1alpha1.MonoKSConfig {
 
 			ClusterRole:        v.ClusterRole,
 			InitControlPlane:   v.InitControlPlane,
-			EnableControlAgent: v.EnableControlAgent,
+			EnableNodeControl: v.EnableNodeControl,
 
 			ClusterName:   v.ClusterName,
 			ClusterDomain: v.ClusterDomain,
@@ -51,7 +51,6 @@ func DefaultMonoKSConfig(v TemplateValues) monov1alpha1.MonoKSConfig {
 
 			SubjectAltNames: copyStringSlice(v.SubjectAltNames),
 			NodeLabels:      copyStringMap(v.NodeLabels),
-			NodeAnnotations: copyStringMap(v.NodeAnnotations),
 
 			Network: monov1alpha1.NetworkSpec{
 				Hostname:         firstNonEmpty(v.Hostname, v.NodeName),

clitools/pkg/templates/values.go

@@ -26,7 +26,7 @@ type TemplateValues struct {
 
 	ClusterRole       string // worker, control-plane
 	InitControlPlane  bool
-	EnableControlAgent bool
+	EnableNodeControl bool
 
 	AllowSchedulingOnControlPlane bool
 	SkipImageCheck                bool
@@ -39,7 +39,6 @@ type TemplateValues struct {
 
 	SubjectAltNames []string
 	NodeLabels      map[string]string
-	NodeAnnotations map[string]string
 }
 
 func defaultTemplateValues() TemplateValues {
@@ -61,7 +60,7 @@ func defaultTemplateValues() TemplateValues {
 
 		ClusterRole:       "control-plane",
 		InitControlPlane:  true,
-		EnableControlAgent: true,
+		EnableNodeControl: true,
 
 		AllowSchedulingOnControlPlane: true,
 		SkipImageCheck:                false,
@@ -78,9 +77,6 @@ func defaultTemplateValues() TemplateValues {
 		NodeLabels: map[string]string{
 			monov1alpha1.Label: "value",
 		},
-		NodeAnnotations: map[string]string{
-			monov1alpha1.Annotation: "value",
-		},
 	}
 }
 
@@ -88,7 +84,7 @@ func LoadTemplateValuesFromEnv() TemplateValues {
 	v := defaultTemplateValues()
 
 	v.Hostname = getenvDefault("MKS_HOSTNAME", v.Hostname)
-	v.NodeName = getenvDefault("MKS_NODE_NAME", v.Hostname)
+	v.NodeName = getenvDefault("MKS_NODE_NAME", getenvDefault("NODE_NAME", v.Hostname))
 
 	v.KubernetesVersion = getenvDefault("MKS_KUBERNETES_VERSION", v.KubernetesVersion)
 	v.ClusterName = getenvDefault("MKS_CLUSTER_NAME", v.ClusterName)
@@ -108,7 +104,7 @@ func LoadTemplateValuesFromEnv() TemplateValues {
 
 	v.ClusterRole = getenvDefault("MKS_CLUSTER_ROLE", v.ClusterRole)
 	v.InitControlPlane = getenvBoolDefault("MKS_INIT_CONTROL_PLANE", v.InitControlPlane)
-	v.EnableControlAgent = getenvBoolDefault("MKS_ENABLE_CONTROL_AGENT", v.EnableControlAgent)
+	v.EnableNodeControl = getenvBoolDefault("MKS_ENABLE_NODE_CONTROL", v.EnableNodeControl)
 
 	v.AllowSchedulingOnControlPlane = getenvBoolDefault("MKS_ALLOW_SCHEDULING_ON_CONTROL_PLANE", v.AllowSchedulingOnControlPlane)
 	v.SkipImageCheck = getenvBoolDefault("MKS_SKIP_IMAGE_CHECK", v.SkipImageCheck)
@@ -129,9 +125,6 @@ func LoadTemplateValuesFromEnv() TemplateValues {
 	if m := parseKeyValueMap(os.Getenv("MKS_NODE_LABELS")); len(m) > 0 {
 		v.NodeLabels = m
 	}
-	if m := parseKeyValueMap(os.Getenv("MKS_NODE_ANNOTATIONS")); len(m) > 0 {
-		v.NodeAnnotations = m
-	}
 
 	return v
 }

configs/cluster.env.default

@@ -30,8 +30,8 @@ MKS_CLUSTER_DOMAIN=cluster.local
 MKS_CLUSTER_ROLE=control-plane
 MKS_INIT_CONTROL_PLANE=yes
 
-# OSUpgrade agent
+# Enable if you want OTA OSUpgrade
-MKS_ENABLE_CONTROL_AGENT=yes
+MKS_ENABLE_NODE_CONTROL=yes
 
 # Boot configs
 # usb, emmc
@@ -56,8 +56,7 @@ MKS_CNI_PLUGIN=default
 
 # Node registration metadata
 # Comma-separated key=value pairs
-MKS_NODE_LABELS=topology.kubernetes.io/zone=lab,node.kubernetes.io/instance-type=mono-gateway
+MKS_NODE_LABELS=topology.kubernetes.io/zone=lab,node.kubernetes.io/instance-type=mono-gateway,mono.si/board=ls1046a
-MKS_NODE_ANNOTATIONS=mono.si/board=ls1046a,monok8s.io/image-version=dev
 
 # Optional
 # Extra API server SANs, comma-separated

devtools/build-all.sh

@@ -0,0 +1,248 @@
+#!/bin/bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+OUT_DIR="$(realpath "$SCRIPT_DIR/../out/")"
+
+cd "$SCRIPT_DIR/../"
+
+# ===== CONFIG TABLE =====
+# Add supported concrete builds here.
+# format: kube=crio
+CONFIGS=(
+  "v1.33.3=cri-o.arm64.v1.33.3"
+  "v1.34.1=cri-o.arm64.v1.34.1"
+  "v1.35.3=cri-o.arm64.v1.35.2"
+)
+
+# ===== HELPERS =====
+
+strip_v() {
+  echo "${1#v}"
+}
+
+minor_of_version() {
+  # input: 1.35.3 or v1.35.3
+  local v="${1#v}"
+  IFS='.' read -r major minor patch <<< "$v"
+  echo "$major.$minor"
+}
+
+version_sort() {
+  sort -t. -k1,1n -k2,2n -k3,3n
+}
+
+list_configs() {
+  echo "Available build targets:"
+  for c in "${CONFIGS[@]}"; do
+    local kube="${c%%=*}"
+    local crio="${c##*=}"
+    echo "  ${kube#v}  (CRI-O: $crio)"
+  done
+}
+
+list_minors() {
+  echo "Latest supported target per minor:"
+  latest_per_minor | while read -r kube crio; do
+    echo "  ${kube#v}  (CRI-O: $crio)"
+  done
+}
+
+latest_per_minor() {
+  # Output lines: <kube> <crio>
+  local minors
+  minors="$(
+    for c in "${CONFIGS[@]}"; do
+      kube="${c%%=*}"
+      minor_of_version "$kube"
+    done | sort -uV
+  )"
+
+  while read -r minor; do
+    [ -n "$minor" ] || continue
+    resolve_minor "$minor"
+  done <<< "$minors"
+}
+
+resolve_exact() {
+  local target="$1"
+  local target_v="v${target#v}"
+
+  for c in "${CONFIGS[@]}"; do
+    local kube="${c%%=*}"
+    local crio="${c##*=}"
+    if [[ "$kube" == "$target_v" ]]; then
+      echo "$kube $crio"
+      return 0
+    fi
+  done
+
+  return 1
+}
+
+resolve_minor() {
+  # input: 1.35 or v1.35
+  local want_minor
+  want_minor="$(minor_of_version "${1#v}.0")"
+
+  local matches=()
+  local c kube crio
+  for c in "${CONFIGS[@]}"; do
+    kube="${c%%=*}"
+    crio="${c##*=}"
+    if [[ "$(minor_of_version "$kube")" == "$want_minor" ]]; then
+      matches+=("${kube}=${crio}")
+    fi
+  done
+
+  if [[ ${#matches[@]} -eq 0 ]]; then
+    return 1
+  fi
+
+  local latest_kube
+  latest_kube="$(
+    for c in "${matches[@]}"; do
+      echo "${c%%=*#}" | sed 's/^v//'
+    done | version_sort | tail -n1
+  )"
+
+  resolve_exact "$latest_kube"
+}
+
+build_exact() {
+  local target="$1"
+  local resolved
+  if ! resolved="$(resolve_exact "$target")"; then
+    echo "❌ Unknown exact target: $target" >&2
+    echo >&2
+    list_configs >&2
+    exit 1
+  fi
+
+  local kube crio
+  read -r kube crio <<< "$resolved"
+
+  echo ">>> Building Kubernetes $kube with $crio"
+  make release CRIO_VERSION="$crio" KUBE_VERSION="$kube" 
+}
+
+build_minor() {
+  local minor="$1"
+  local resolved
+  if ! resolved="$(resolve_minor "$minor")"; then
+    echo "❌ No supported target found for minor: $minor" >&2
+    echo >&2
+    list_minors >&2
+    exit 1
+  fi
+
+  local kube crio
+  read -r kube crio <<< "$resolved"
+
+  echo ">>> Minor $minor resolved to latest supported target ${kube#v}"
+  echo ">>> Building Kubernetes $kube with $crio"
+  make release CRIO_VERSION="$crio" KUBE_VERSION="$kube" 
+}
+
+build_range() {
+  # input like 1.33-1.35
+  local range="$1"
+  local start="${range%-*}"
+  local end="${range#*-}"
+
+  local start_minor="${start#v}"
+  local end_minor="${end#v}"
+
+  local any=0
+  while read -r minor; do
+    [ -n "$minor" ] || continue
+
+    # simple lexical-safe because format is N.N and sort -V was used
+    if [[ "$(printf '%s\n%s\n' "$start_minor" "$minor" | sort -V | head -n1)" != "$start_minor" ]]; then
+      continue
+    fi
+    if [[ "$(printf '%s\n%s\n' "$minor" "$end_minor" | sort -V | head -n1)" != "$minor" ]]; then
+      continue
+    fi
+
+    any=1
+    build_minor "$minor"
+  done < <(
+    for c in "${CONFIGS[@]}"; do
+      kube="${c%%=*}"
+      minor_of_version "$kube"
+    done | sort -uV
+  )
+
+  if [[ $any -eq 0 ]]; then
+    echo "❌ No supported minors found in range: $range" >&2
+    echo >&2
+    list_minors >&2
+    exit 1
+  fi
+}
+
+build_all() {
+  local c kube crio
+  for c in "${CONFIGS[@]}"; do
+    kube="${c%%=*}"
+    crio="${c##*=}"
+    echo ">>> Building Kubernetes $kube with $crio"
+    make release CRIO_VERSION="$crio" KUBE_VERSION="$kube" 
+  done
+}
+
+usage() {
+  cat <<'EOF'
+Usage:
+  ./devtools/build-all.sh
+  ./devtools/build-all.sh list
+  ./devtools/build-all.sh list-minors
+  ./devtools/build-all.sh 1.35.3
+  ./devtools/build-all.sh 1.35
+  ./devtools/build-all.sh 1.33-1.35
+
+Behavior:
+  no args       Build all configured targets
+  list          List all exact configured targets
+  list-minors   List latest supported target per minor
+  X.Y.Z         Build exact version
+  X.Y           Build latest supported patch in that minor
+  X.Y-A.B       Build latest supported patch for each minor in range
+EOF
+}
+
+# ===== ENTRY =====
+
+if [[ $# -eq 0 ]]; then
+  echo "No target specified -> building all configured targets"
+  build_all
+  exit 0
+fi
+
+case "$1" in
+  list)
+    list_configs
+    ;;
+  list-minors)
+    list_minors
+    ;;
+  -*|--help|help)
+    usage
+    ;;
+  *-*)
+    build_range "$1"
+    ;;
+  *.*.*)
+    build_exact "$1"
+    ;;
+  *.*)
+    build_minor "$1"
+    ;;
+  *)
+    echo "❌ Unrecognized target: $1" >&2
+    echo >&2
+    usage >&2
+    exit 1
+    ;;
+esac

devtools/create-join-token.sh

@@ -0,0 +1,196 @@
+#!/bin/sh
+set -eu
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+ROOT_DIR="$(realpath "$SCRIPT_DIR/..")"
+LIB_DIR="$ROOT_DIR/scripts"
+CLUSTER_ENV_WORK="${CLUSTER_ENV_WORK:-$ROOT_DIR/configs/cluster.env.work}"
+
+KUBECTL="${KUBECTL:-kubectl}"
+TTL_HOURS="${TTL_HOURS:-24}"
+WAIT_SECONDS="${WAIT_SECONDS:-30}"
+
+need() {
+    command -v "$1" >/dev/null 2>&1 || {
+        echo "missing required command: $1" >&2
+        exit 1
+    }
+}
+
+rfc3339_after_hours() {
+    hours="$1"
+
+    # GNU date
+    if date -u -d "+${hours} hours" '+%Y-%m-%dT%H:%M:%SZ' >/dev/null 2>&1; then
+        date -u -d "+${hours} hours" '+%Y-%m-%dT%H:%M:%SZ'
+        return
+    fi
+
+    # BSD/macOS date
+    if date -u -v+"${hours}"H '+%Y-%m-%dT%H:%M:%SZ' >/dev/null 2>&1; then
+        date -u -v+"${hours}"H '+%Y-%m-%dT%H:%M:%SZ'
+        return
+    fi
+
+    echo "cannot compute expiration time with this date(1). Set EXPIRATION manually." >&2
+    exit 1
+}
+
+decode_base64_to_file() {
+    input="$1"
+    output="$2"
+
+    if printf '%s' "$input" | base64 -d >"$output" 2>/dev/null; then
+        return
+    fi
+
+    if printf '%s' "$input" | base64 -D >"$output" 2>/dev/null; then
+        return
+    fi
+
+    if printf '%s' "$input" | openssl base64 -d -A >"$output" 2>/dev/null; then
+        return
+    fi
+
+    echo "failed to decode certificate-authority-data" >&2
+    exit 1
+}
+
+need "$KUBECTL"
+need openssl
+need awk
+need sed
+
+TOKEN_ID="${TOKEN_ID:-$(openssl rand -hex 3)}"
+TOKEN_SECRET="${TOKEN_SECRET:-$(openssl rand -hex 8)}"
+TOKEN="${TOKEN_ID}.${TOKEN_SECRET}"
+SECRET_NAME="bootstrap-token-${TOKEN_ID}"
+
+if [ "${TTL_HOURS}" = "0" ]; then
+    EXPIRATION=""
+else
+    EXPIRATION="${EXPIRATION:-$(rfc3339_after_hours "$TTL_HOURS")}"
+fi
+
+echo "Creating bootstrap token Secret: ${SECRET_NAME}" >&2
+
+{
+    cat <<EOF
+apiVersion: v1
+kind: Secret
+metadata:
+  name: ${SECRET_NAME}
+  namespace: kube-system
+type: bootstrap.kubernetes.io/token
+stringData:
+  description: "Join token created with kubectl"
+  token-id: "${TOKEN_ID}"
+  token-secret: "${TOKEN_SECRET}"
+  usage-bootstrap-authentication: "true"
+  usage-bootstrap-signing: "true"
+  auth-extra-groups: "system:bootstrappers:kubeadm:default-node-token"
+EOF
+
+    if [ -n "$EXPIRATION" ]; then
+        printf '  expiration: "%s"\n' "$EXPIRATION"
+    fi
+} | "$KUBECTL" apply -f -
+
+TMPDIR="$(mktemp -d)"
+trap 'rm -rf "$TMPDIR"' EXIT INT TERM
+
+CA_FILE="$TMPDIR/ca.crt"
+
+CA_DATA="$("$KUBECTL" config view --raw --minify --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')"
+
+if [ -z "$CA_DATA" ]; then
+    echo "could not find certificate-authority-data in current kubeconfig" >&2
+    echo "token was created, but cannot print a safe kubeadm join command" >&2
+    echo "token: ${TOKEN}"
+    exit 0
+fi
+
+decode_base64_to_file "$CA_DATA" "$CA_FILE"
+
+CA_HASH="$(
+    openssl x509 -in "$CA_FILE" -pubkey -noout |
+        openssl pkey -pubin -outform der 2>/dev/null |
+        openssl dgst -sha256 -hex |
+        awk '{print $2}'
+)"
+
+SERVER="$("$KUBECTL" config view --raw --minify -o jsonpath='{.clusters[0].cluster.server}')"
+JOIN_ENDPOINT="$(printf '%s\n' "$SERVER" | sed -E 's#^https?://##')"
+
+echo "Waiting for cluster-info signature for token ${TOKEN_ID}..." >&2
+
+i=0
+signed="false"
+while [ "$i" -lt "$WAIT_SECONDS" ]; do
+    template="{{ index .data \"jws-kubeconfig-${TOKEN_ID}\" }}"
+    sig="$("$KUBECTL" -n kube-public get configmap cluster-info -o "go-template=${template}" 2>/dev/null || true)"
+
+    if [ -n "$sig" ]; then
+        signed="true"
+        break
+    fi
+
+    i=$((i + 1))
+    sleep 1
+done
+
+echo
+echo "Token:"
+echo "  ${TOKEN}"
+
+if [ -n "$EXPIRATION" ]; then
+    echo
+    echo "Expires:"
+    echo "  ${EXPIRATION}"
+fi
+
+echo
+echo "Join command:"
+echo "  kubeadm join ${JOIN_ENDPOINT} --token ${TOKEN} --discovery-token-ca-cert-hash sha256:${CA_HASH}"
+
+TMP_ENV="$(mktemp)"
+trap 'rm -f "$TMP_ENV"; rm -rf "$TMPDIR"' EXIT INT TERM
+
+cat >"$TMP_ENV" <<EOF
+MKS_API_SERVER_ENDPOINT=${JOIN_ENDPOINT}
+MKS_BOOTSTRAP_TOKEN=${TOKEN}
+MKS_DISCOVERY_TOKEN_CA_CERT_HASH=sha256:${CA_HASH}
+EOF
+
+echo
+echo "cluster-config:"
+cat "$TMP_ENV"
+
+if [ ! -x "$LIB_DIR/merge-env.sh" ]; then
+    echo "merge-env.sh not found or not executable: $LIB_DIR/merge-env.sh" >&2
+    exit 1
+fi
+
+"$LIB_DIR/merge-env.sh" "$TMP_ENV" "$CLUSTER_ENV_WORK"
+
+echo
+echo "Merged into:"
+echo "  $CLUSTER_ENV_WORK"
+echo
+echo "Try"
+cat <<EOF
+make cluster-config \\
+    MKS_HOSTNAME=monok8s-worker \\
+    MKS_CLUSTER_ROLE=worker \\
+    MKS_INIT_CONTROL_PLANE=no \\
+    MKS_MGMT_ADDRESS=10.0.0.10/24 \\
+    MKS_APISERVER_ADVERTISE_ADDRESS=10.0.0.10 \\
+    MKS_CNI_PLUGIN=none
+EOF
+
+
+if [ "$signed" != "true" ]; then
+    echo >&2
+    echo "warning: cluster-info was not signed within ${WAIT_SECONDS}s." >&2
+    echo "If kubeadm join fails discovery, check that kube-controller-manager enables bootstrapsigner." >&2
+fi

devtools/serve-images.sh

@@ -0,0 +1,6 @@
+#/bin/bash
+
+SCRIPT_DIR="$(dirname "${BASH_SOURCE[0]}")"
+OUT_DIR="$(realpath "$SCRIPT_DIR"/../out/)"
+
+python3 -m http.server 8000 --bind 0.0.0.0 --directory "$OUT_DIR"

devtools/setup-bulid-host.sh

@@ -0,0 +1,73 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+if [ "$(id -u)" -ne 0 ]; then
+    echo "Run as root, e.g. sudo $0" >&2
+    exit 1
+fi
+
+. /etc/os-release
+
+if [ "${ID:-}" != "debian" ]; then
+    echo "This script is intended for Debian. Detected ID=${ID:-unknown}" >&2
+    exit 1
+fi
+
+echo "==> Removing conflicting Docker packages, if present"
+apt-get remove -y \
+    docker.io \
+    docker-compose \
+    docker-doc \
+    podman-docker \
+    containerd \
+    runc || true
+
+echo "==> Installing minimal repo setup tools"
+apt-get update
+apt-get install -y --no-install-recommends \
+    ca-certificates \
+    curl
+
+echo "==> Adding Docker official APT repo"
+install -m 0755 -d /etc/apt/keyrings
+
+curl -fsSL https://download.docker.com/linux/debian/gpg \
+    -o /etc/apt/keyrings/docker.asc
+
+chmod a+r /etc/apt/keyrings/docker.asc
+
+cat > /etc/apt/sources.list.d/docker.sources <<EOF
+Types: deb
+URIs: https://download.docker.com/linux/debian
+Suites: ${VERSION_CODENAME}
+Components: stable
+Architectures: $(dpkg --print-architecture)
+Signed-By: /etc/apt/keyrings/docker.asc
+EOF
+
+echo "==> Installing build/test packages"
+apt-get update
+apt-get install -y --no-install-recommends \
+    docker-ce \
+    docker-buildx-plugin \
+    qemu-user-static \
+    binfmt-support \
+    make
+
+echo "==> Enabling Docker"
+systemctl enable --now docker
+
+echo "==> Registering binfmt handlers"
+systemctl restart binfmt-support || true
+
+echo "==> Docker version"
+docker --version
+
+echo "==> Buildx version"
+docker buildx version || true
+
+echo "==> Done"
+echo
+echo "Optional: allow your normal user to run docker without sudo:"
+echo "  sudo usermod -aG docker \$USER"
+echo "Then log out and back in."

devtools/test-upgrade.sh

@@ -0,0 +1,54 @@
+#!/bin/bash
+
+SCRIPT_DIR="$(dirname "${BASH_SOURCE[0]}")"
+OUT_DIR="$(realpath "$SCRIPT_DIR"/../out/)"
+
+set -e
+
+DEFAULT_BASE_URL="http://localhost:8000"
+DEFAULT_TARGET_VERSION="v1.34.1"
+STABLE_VERSION="v1.34.1"
+NAME="my-upgrade-1"
+
+if [ -r /dev/tty ]; then
+    printf "Enter the base url (%s): " "$DEFAULT_BASE_URL" > /dev/tty
+    read -r BASE_URL < /dev/tty
+
+    printf "Enter the target version (%s): " "$DEFAULT_TARGET_VERSION" > /dev/tty
+    read -r TARGET_VERSION < /dev/tty
+else
+    echo "No TTY available for interactive input" >&2
+    exit 1
+fi
+
+BASE_URL="${BASE_URL:-$DEFAULT_BASE_URL}"
+TARGET_VERSION="${TARGET_VERSION:-$DEFAULT_TARGET_VERSION}"
+
+echo "apiVersion: monok8s.io/v1alpha1"
+echo "kind: OSUpgrade"
+echo "metadata:"
+echo "  name: \"$NAME\""
+echo "spec:"
+echo "  desiredVersion: \"$TARGET_VERSION\""
+echo "  nodeSelector: {}"
+echo "  catalog:"
+echo "    inline: |"
+echo "      stable: $STABLE_VERSION"
+echo "      images:"
+
+for c in "$OUT_DIR"/catalog-*.txt; do
+    version=$(grep 'version:' "$c" | awk '{print $3}')
+    url=$(grep 'url:' "$c" | sed 's/.*"\(.*\)"/\1/')
+    checksum=$(grep 'checksum:' "$c" | awk '{print $2}')
+    size=$(grep 'size:' "$c" | awk '{print $2}')
+
+    filename=$(basename "$url")
+
+    echo "        - version: $version"
+    echo "          url: $BASE_URL/$filename"
+    echo "          checksum: sha256:$checksum"
+    echo "          size: $size"
+done
+
+echo "      blocked:"
+echo "        - v1.34.0"

docker/kernel-build.Dockerfile

@@ -23,6 +23,7 @@ WORKDIR /build/nxplinux
 
 COPY kernel-extra.config /tmp/kernel-extra.config
 COPY kernel-build/dts/*.dts ./arch/arm64/boot/dts/freescale/
+COPY kernel-build/ensure-kconfig.sh /build/
 
 RUN grep -q "^dtb-\\\$(CONFIG_ARCH_LAYERSCAPE) += ${DEVICE_TREE_TARGET}.dtb$" \
       arch/arm64/boot/dts/freescale/Makefile \
@@ -33,7 +34,7 @@ RUN grep -q "^dtb-\\\$(CONFIG_ARCH_LAYERSCAPE) += ${DEVICE_TREE_TARGET}.dtb$" \
 RUN make ARCH="${ARCH}" CROSS_COMPILE="${CROSS_COMPILE}" defconfig lsdk.config \
  && ./scripts/kconfig/merge_config.sh -m .config /tmp/kernel-extra.config \
  && make ARCH="${ARCH}" CROSS_COMPILE="${CROSS_COMPILE}" olddefconfig \
- && grep '^CONFIG_NF_TABLES=' .config \
+ && /build/ensure-kconfig.sh .config /tmp/kernel-extra.config \
  && make ARCH="${ARCH}" CROSS_COMPILE="${CROSS_COMPILE}" -j"$(nproc)"
 
 # artifact collection

docs/cilium.md

@@ -0,0 +1,16 @@
+# Worker node
+
+```yaml
+apiVersion: cilium.io/v2
+kind: CiliumNodeConfig
+metadata:
+  namespace: kube-system
+  name: monok8s-worker
+spec:
+  nodeSelector:
+    matchLabels:
+      node.kubernetes.io/instance-type: mono-gateway
+  defaults:
+    devices: "eth1"
+    direct-routing-device: "eth1"
+```

docs/installing-ssh-pod.md

@@ -0,0 +1,204 @@
+# Installing the recovery SSHD pod
+
+This page explains how to install a temporary SSH server pod for break-glass recovery.
+
+Use this when normal Kubernetes access is degraded, for example after the API server certificate expires or rotates and you need to retrieve updated host-side credentials.
+
+The SSHD pod is intended for recovery and debugging only. Remove it when you are done.
+
+## What this does
+
+The recovery pod starts an SSH server on the selected node and authorizes your local SSH public key.
+
+The pod also mounts selected host paths under `/host`, so you can inspect the host filesystem and run some host-side recovery commands through `chroot`.
+
+For example:
+
+```sh
+chroot /host /bin/sh -lc 'rc-status'
+chroot /host /bin/sh -lc 'rc-service crio status'
+chroot /host /bin/sh -lc 'rc-service kubelet status'
+```
+
+## Requirements
+
+You need:
+
+- A working `kubectl` connection to the cluster.
+- Access to the `node-agent` DaemonSet in the `mono-system` namespace.
+- A local SSH public key, usually `~/.ssh/id_rsa.pub` or `~/.ssh/id_ed25519.pub`.
+
+Use a public key file only. Do not pass your private key.
+
+## Generate the SSHD manifest
+
+To print the recovery SSHD manifest:
+
+```bash
+kubectl exec -i -n mono-system ds/node-agent -- \
+  ctl create sshd --authkeys /dev/stdin < ~/.ssh/id_rsa.pub
+```
+
+This reads your local public key and places it into the generated pod's `authorized_keys`.
+
+If you use Ed25519 keys, use:
+
+```bash
+kubectl exec -i -n mono-system ds/node-agent -- \
+  ctl create sshd --authkeys /dev/stdin < ~/.ssh/id_ed25519.pub
+```
+
+## Generate and apply the manifest
+
+To create the recovery SSHD resources in one step:
+
+```bash
+kubectl exec -i -n mono-system ds/node-agent -- \
+  ctl create sshd --authkeys /dev/stdin < ~/.ssh/id_rsa.pub \
+  | kubectl apply -f -
+```
+
+For Ed25519:
+
+```bash
+kubectl exec -i -n mono-system ds/node-agent -- \
+  ctl create sshd --authkeys /dev/stdin < ~/.ssh/id_ed25519.pub \
+  | kubectl apply -f -
+```
+
+## Why `-i` is used instead of `-it`
+
+Use `-i`, not `-it`, when piping the SSH public key.
+
+The `-t` option allocates a pseudo-TTY. A pseudo-TTY can modify piped input, which is not what you want when passing an SSH public key through stdin.
+
+Correct:
+
+```bash
+kubectl exec -i -n mono-system ds/node-agent -- \
+  ctl create sshd --authkeys /dev/stdin < ~/.ssh/id_rsa.pub
+```
+
+Avoid:
+
+```bash
+kubectl exec -it -n mono-system ds/node-agent -- \
+  ctl create sshd --authkeys /dev/stdin < ~/.ssh/id_rsa.pub
+```
+
+## Check that the pod is running
+
+After applying the manifest, check the pod:
+
+```bash
+kubectl get pods -n mono-system -l app.kubernetes.io/name=sshd
+```
+
+Check the service:
+
+```bash
+kubectl get svc -n mono-system -l app.kubernetes.io/name=sshd
+```
+
+If the pod does not start, inspect it:
+
+```bash
+kubectl describe pod -n mono-system -l app.kubernetes.io/name=sshd
+```
+
+## Connect through SSH
+
+The exact SSH command depends on how the generated service exposes the pod.
+
+If the service uses a NodePort such as `30022`, connect with:
+
+```bash
+ssh -p 30022 root@<node-ip>
+```
+
+Replace `<node-ip>` with the node's reachable IP address.
+
+## Access the host environment
+
+Inside the SSH session, the host filesystem is available under `/host`.
+
+Useful checks:
+
+```sh
+ls -la /host
+chroot /host /bin/sh -lc 'rc-status'
+chroot /host /bin/sh -lc 'rc-service crio status'
+chroot /host /bin/sh -lc 'rc-service kubelet status'
+```
+
+Restart CRI-O:
+
+```sh
+chroot /host /bin/sh -lc 'rc-service crio restart'
+```
+
+Restart kubelet:
+
+```sh
+chroot /host /bin/sh -lc 'rc-service kubelet restart'
+```
+
+You can also inspect host processes from the pod because the recovery pod uses the host PID namespace:
+
+```sh
+ps aux | grep -E 'kubelet|crio'
+```
+
+## Notes for monok8s host mounts
+
+The recovery pod does not mount host `/` directly.
+
+On monok8s, `/` and `/var` may be private mounts. Mounting them directly as host paths can fail with errors such as:
+
+```text
+path "/" is mounted on "/" but it is not a shared or slave mount
+```
+
+or:
+
+```text
+path "/var" is mounted on "/var" but it is not a shared or slave mount
+```
+
+Instead, the recovery pod assembles a minimal host root under `/host` from individual host paths.
+
+For `/var`, it uses the backing path:
+
+```text
+/data/var -> /host/var
+```
+
+This avoids the private bind-mount issue.
+
+## Remove the recovery pod
+
+When recovery is complete, remove the generated resources.
+
+If the resources use the default SSHD labels:
+
+```bash
+kubectl delete deployment -n mono-system -l app.kubernetes.io/name=sshd
+kubectl delete service -n mono-system -l app.kubernetes.io/name=sshd
+kubectl delete configmap -n mono-system -l app.kubernetes.io/name=sshd
+```
+
+If your generated manifest uses a fixed resource name, you can also remove them by name:
+
+```bash
+kubectl delete deployment -n mono-system sshd
+kubectl delete service -n mono-system sshd
+kubectl delete configmap -n mono-system sshd-authorized-keys
+```
+
+## Security warning
+
+This pod is powerful.
+
+It runs with root-level recovery access and can inspect or modify host files through `/host`. Treat it as a temporary break-glass tool, not a normal service.
+
+Do not leave it running after recovery.

docs/ota.md

@@ -1,19 +1,54 @@
-## Upgrade process
+# OS OTA Upgrades
 
-We use an agent to watch the OSUpgrade CRD to handle this. Our image versions follows upstream.
+MonoK8s upgrades are driven through two custom resources:
 
-To issue an upgrade. Simply use
+- `OSUpgrade`: the user-facing upgrade request.
+- `OSUpgradeProgress`: the per-node upgrade state watched and executed by the node agent.
 
+The node agent does the actual upgrade work. It watches `OSUpgradeProgress` resources assigned to its node, downloads the selected image, writes it to the inactive rootfs partition, updates status, and reboots when ready.
+
+The controller is optional but strongly recommended. It watches `OSUpgrade` resources and creates the matching `OSUpgradeProgress` resources for the target nodes.
+
+## Install the controller
+
+By default, each managed node only runs the node agent. The node agent does **not** watch `OSUpgrade` directly; it only watches `OSUpgradeProgress`.
+
+You can create `OSUpgradeProgress` resources by hand, but normal users should not need to. Install the controller instead, then create `OSUpgrade` resources.
+
+Install the controller from the existing node-agent image:
+
+```bash
+kubectl exec -i -n mono-system ds/node-agent -- \
+  ctl create controller --image REPO/IMAGE:TAG | kubectl apply -f -
+```
+
+### `--image`
+
+`--image` is optional.
+
+If omitted, the generated Deployment uses the local controller image that is already shipped with managed nodes. In that mode, the controller Deployment is scheduled only onto managed nodes because the image is expected to exist locally.
+
+If provided, the generated Deployment uses that image directly. This is useful when you host the controller image in your own registry.
+
+There is no official public image repository yet, so external controller images must currently be managed by the operator.
+
+## Create an upgrade
+
+Create an `OSUpgrade` resource to request an upgrade:
+
+```bash
 kubectl apply -f upgrade.yaml
+```
+
+Example:
 
-Example yaml
 ```yaml
 apiVersion: monok8s.io/v1alpha1
 kind: OSUpgrade
 metadata:
-  name: "my-ugrade-2"
+  name: my-upgrade-2
 spec:
-  version: "v1.35.3"
+  version: v1.35.3
   nodeSelector: {}
   catalog:
     inline: |
@@ -34,24 +69,61 @@ spec:
         - version: v1.35.1
           url: http://localhost:8000/rootfs.ext4.zst
           checksum: sha256:99af82a263deca44ad91d21d684f0fa944d5d0456a1da540f1c644f8aa59b14b
-          size: 1858076672 # expanded image size in bytes, use "zstd -lv image.zst to check"
+          size: 1858076672 # expanded image size in bytes; check with: zstd -lv image.zst
       blocked:
         - v1.34.0
 ```
 
-catalog also accepts URL or ConfigMap※
+### `spec.version`
+
+`spec.version` is the requested target version.
+
+It may be either:
+
+- an explicit version, such as `v1.35.3`
+- `stable`, if the catalog defines a `stable` version
+
+### `spec.nodeSelector`
+
+`spec.nodeSelector` selects the nodes that should receive the upgrade.
+
+An empty selector means all eligible managed nodes.
+
+### `spec.catalog`
+
+The catalog tells the agent where to find available OS images.
+
+The catalog can be provided inline:
+
 ```yaml
 catalog:
-  URL: https://example.com/images.yaml
+  inline: |
-
+    stable: v1.35.1
-catalog:
+    images:
-  ConfigMap: images-cm
+      - version: v1.35.1
+        url: https://example.invalid/images/monok8s-v1.35.1.img.zst
+        checksum: sha256:abc
+        size: 1858076672
 ```
 
-※ ConfigMap requires additional RBAC permissions which is not enabled by default. You can edit
+It can also be loaded from a URL:
-the control-agent's ClusterRole and add `configmaps: get` to allow this.
+
+```yaml
+catalog:
+  url: https://example.com/images.yaml
+```
+
+Or from a ConfigMap:
+
+```yaml
+catalog:
+  configMap: images-cm
+```
+
+ConfigMap catalogs require extra RBAC. This permission is not enabled by default. To use a ConfigMap catalog, edit the relevant ClusterRole and allow `get` on `configmaps`.
+
+Catalog content should look like this:
 
-Contents should look like this
 ```yaml
 stable: v1.35.1
 images:
@@ -70,64 +142,114 @@ images:
   - version: v1.35.1
     url: http://localhost:8000/rootfs.ext4.zst
     checksum: sha256:99af82a263deca44ad91d21d684f0fa944d5d0456a1da540f1c644f8aa59b14b
-    size: 1858076672 # expanded image size in bytes, use "zstd -lv image.zst to check"
+    size: 1858076672 # expanded image size in bytes; check with: zstd -lv image.zst
 blocked:
   - v1.34.0
 ```
 
-### Monitoring the upgrades
+## Monitor upgrades
 
-kubectl get osugrades
+List upgrade requests:
-```
+
-NAME            DESIRED    RESOLVED   PHASE       TARGETS   OK   FAIL   AGE
+```bash
-my-upgrade-3    stable     v1.35.4    RollingOut  3         1    0      1m
+kubectl get osupgrades
-my-upgrade-2    v1.35.3    v1.35.3    Accepted    2         0    0      1m
-my-downgrade-1  v1.33.2    v1.33.2    Rejected    2         0    2      1m
 ```
 
-kubectl get osupgradeprogress
+Example output:
+
+```text
+NAME            DESIRED    RESOLVED   PHASE
+my-upgrade-3    stable     v1.35.4    Pending
+my-upgrade-2    v1.35.3    v1.35.3    Accepted
+my-downgrade-1  v1.33.2    v1.33.2    Rejected
 ```
+
+List per-node progress:
+
+```bash
+kubectl get osupgradeprogresses
+```
+
+Example output:
+
+```text
 NAME                NODE        SOURCE        CURRENT  TARGET   STATUS
-osupgrade-abc123f   node-1      my-upgrade-2  v1.34.1  v1.35.3  downloading
+osupgrade-abc123f   node-1      my-upgrade-2  v1.34.1  v1.35.3  Downloading
-osupgrade-cde456g   node-2      my-upgrade-2  v1.35.3  v1.35.3  completed
+osupgrade-cde456g   node-2      my-upgrade-2  v1.35.3  v1.35.3  Completed
 ```
 
+Inspect one node's progress:
 
+```bash
 kubectl describe osupgradeprogress osupgrade-abc123f
+```
+
+Example resource:
+
 ```yaml
 apiVersion: monok8s.io/v1alpha1
 kind: OSUpgradeProgress
 metadata:
-  name: "osupgrade-abc123f"
+  name: osupgrade-abc123f
 spec:
   sourceRef:
     name: my-upgrade-2
   nodeName: node-1
 status:
-  currentVersion: "v1.34.1"
+  currentVersion: v1.34.1
-  targetVersion: "v1.35.3"
+  targetVersion: v1.35.3
   phase: Downloading
   startedAt: null
   completedAt: null
   lastUpdatedAt: null
   retryCount: 0
-  inactivePartition: "B"
+  inactivePartition: B
   failureReason: ""
   message: ""
 ```
 
+## Retry a failed upgrade
+
+If an upgrade fails, for example because the image download failed, edit `spec.retryNonce` on the affected `OSUpgradeProgress` resource.
+
+Any changed value is enough. The field is only used to tell the node agent that the user intentionally requested a retry.
+
+Example:
+
+```bash
+kubectl patch osupgradeprogress osupgrade-abc123f \
+  --type merge \
+  -p '{"spec":{"retryNonce":"retry-1"}}'
+```
+
+If the same node fails again and you want to retry again, change the nonce to a new value:
+
+```bash
+kubectl patch osupgradeprogress osupgrade-abc123f \
+  --type merge \
+  -p '{"spec":{"retryNonce":"retry-2"}}'
+```
+
 ## Development notes
 
-### Flashing manually into partition B
+### Flash an image manually into partition B
 
-**Use nmap ncat**. Otherwise we'll have all kinds of fabulous issues sending it.
+Use nmap's `ncat`. Other tools may work, but they are more likely to cause annoying stream or connection behavior.
 
-Sending side
+On the sending machine:
-```
+
-pv "out/rootfs.ext4.zst" | ncat 10.0.0.10 1234 --send-only
+```bash
+pv out/rootfs.ext4.zst | ncat 10.0.0.10 1234 --send-only
 ```
 
-Receiving side
+On the receiving machine:
-```
+
-ncat -l 1234 --recv-only | zstd -d -c | dd of=/dev/sda3 bs=4M status=progress && sync && echo "SUCCESS"
+```bash
+ncat -l 1234 --recv-only | \
+  zstd -d -c | \
+  dd of=/dev/sda3 bs=4M status=progress && \
+  sync && \
+  echo "SUCCESS"
 ```
+
+Be careful with the target partition. The example writes to `/dev/sda3`, which is assumed to be rootfs B in that setup. Verify the partition layout before running this on real hardware.

docs/uboot.md

@@ -20,11 +20,11 @@ run bootusb
 
 ## Run fw_printenv and fw_setenv from kubectl
 ```
-# Avoid using daemonset/control-agent if you have multiple nodes
+# Avoid using daemonset/node-agent if you have multiple nodes
-kubectl exec -n kube-system control-agent-abcdef1 -- /ctl internal fw-setenv --key foo --value bar
+kubectl exec -n kube-system node-agent-abcdef1 -- /ctl internal fw-setenv --key foo --value bar
 
 # fw_printenv
-kubectl exec -n kube-system ds/control-agent -- /ctl internal fw-printenv --key foo
+kubectl exec -n kube-system ds/node-agent -- /ctl internal fw-printenv --key foo
 ```
 
 ## Original uboot env from mono

initramfs/rootfs-extra/init

@@ -22,6 +22,9 @@ mount_retry() {
 
     i=0
     while :; do
+		# BusyBox mount just needs a normal -o option string here.
+		# The important bit is that overlayfs itself requires lowerdir/upperdir/workdir,
+		# and workdir must live on the same filesystem as upperdir.
         if mount -o "$opts" -t "$fstype" "$dev" "$target"; then
             return 0
         fi
@@ -32,6 +35,30 @@ mount_retry() {
     done
 }
 
+mount_data_overlay() {
+    dir="$1"
+
+    case "$dir" in
+        /*) ;;
+        *) panic "overlay dir must be absolute: $dir" ;;
+    esac
+
+    lower="/newroot$dir"
+    state="/newroot/data${dir}-overlay"
+    upper="$state/upper"
+    work="$state/work"
+
+    [ -d "$lower" ] || mkdir -p "$lower"
+
+    mkdir -p "$upper" "$work"
+
+    log "Mounting overlay for $dir"
+
+    mount_or_panic -t overlay overlay \
+        -o "lowerdir=$lower,upperdir=$upper,workdir=$work" \
+        "$lower"
+}
+
 wait_for_path() {
     path="$1"
     i=0
@@ -207,6 +234,16 @@ mount_or_panic -t proc proc /proc
 mount_or_panic -t sysfs sysfs /sys
 mount_or_panic -t tmpfs tmpfs /run
 
+mkdir -p /sys/fs/bpf
+
+if ! mountpoint -q /sys/fs/bpf; then
+    mount_or_panic -t bpf bpffs /sys/fs/bpf
+fi
+
+mount_or_panic --make-rshared /sys
+mount_or_panic --make-rshared /run
+mount_or_panic --make-shared /sys/fs/bpf
+
 echo 1 > /proc/sys/kernel/printk
 
 mkdir -p /dev/pts
@@ -264,17 +301,11 @@ mount_retry "$ROOT_DEV" /newroot ext4 ro
 mount_retry "$DATA_DEV" /newroot/data ext4 rw
 
 mkdir -p /newroot/data/var
-mkdir -p /newroot/data/etc-overlay/upper
-mkdir -p /newroot/data/etc-overlay/work
-
 mount_or_panic --bind /newroot/data/var /newroot/var
+mount_or_panic --make-rshared /newroot/var
 
-# BusyBox mount just needs a normal -o option string here.
+mount_data_overlay /etc
-# The important bit is that overlayfs itself requires lowerdir/upperdir/workdir,
+mount_data_overlay /opt/cni
-# and workdir must live on the same filesystem as upperdir.
-mount_or_panic -t overlay overlay \
-    -o "lowerdir=/newroot/etc,upperdir=/newroot/data/etc-overlay/upper,workdir=/newroot/data/etc-overlay/work" \
-    /newroot/etc
 
 if [ "$BOOT_PART" = "A" ]; then
     ALT_PART="$(find_sibling_part_on_same_disk "$ROOT_DEV" rootfsB || true)"

kernel-build/ensure-kconfig.sh

@@ -0,0 +1,127 @@
+#!/bin/sh
+set -eu
+
+CONFIG_FILE="${1:-}"
+EXPECTED_FILE="${2:-}"
+
+if [ -z "$CONFIG_FILE" ] || [ -z "$EXPECTED_FILE" ]; then
+    echo "usage: $0 <resolved-.config> <expected-fragment.config>" >&2
+    exit 2
+fi
+
+if [ ! -f "$CONFIG_FILE" ]; then
+    echo "error: config file not found: $CONFIG_FILE" >&2
+    exit 2
+fi
+
+if [ ! -f "$EXPECTED_FILE" ]; then
+    echo "error: expected config fragment not found: $EXPECTED_FILE" >&2
+    exit 2
+fi
+
+failed=0
+
+normalize_expected_line() {
+    line="$1"
+
+    case "$line" in
+        CONFIG_*=y|CONFIG_*=m)
+            echo "$line"
+            ;;
+        CONFIG_*=n)
+            sym="${line%%=*}"
+            echo "# $sym is not set"
+            ;;
+        "# CONFIG_"*" is not set")
+            echo "$line"
+            ;;
+        CONFIG_*=*)
+            echo "$line"
+            ;;
+        *)
+            return 1
+            ;;
+    esac
+}
+
+is_disabled_expected() {
+    expected="$1"
+
+    case "$expected" in
+        "# CONFIG_"*" is not set")
+            return 0
+            ;;
+        *)
+            return 1
+            ;;
+    esac
+}
+
+symbol_from_expected() {
+    expected="$1"
+
+    case "$expected" in
+        CONFIG_*=*)
+            echo "${expected%%=*}"
+            ;;
+        "# CONFIG_"*" is not set")
+            printf '%s\n' "$expected" | sed 's/^# \(CONFIG_[^ ]*\) is not set$/\1/'
+            ;;
+        *)
+            return 1
+            ;;
+    esac
+}
+
+check_expected_line() {
+    expected="$1"
+    sym="$(symbol_from_expected "$expected")"
+
+    actual="$(grep -E "^${sym}=|^# ${sym} is not set$" "$CONFIG_FILE" || true)"
+
+    if [ "$actual" = "$expected" ]; then
+        return 0
+    fi
+
+    # For disabled symbols, absence from the final .config is acceptable.
+    # Some Kconfig symbols do not exist on this arch/tree, and missing still means "not enabled".
+    if is_disabled_expected "$expected" && [ -z "$actual" ]; then
+        return 0
+    fi
+
+    echo "kconfig mismatch: $sym" >&2
+    echo "  expected: $expected" >&2
+    if [ -n "$actual" ]; then
+        echo "  actual:   $actual" >&2
+    else
+        echo "  actual:   <missing>" >&2
+    fi
+
+    failed=1
+}
+
+while IFS= read -r raw || [ -n "$raw" ]; do
+    # Strip leading/trailing whitespace.
+    line="$(printf '%s\n' "$raw" | sed 's/^[[:space:]]*//; s/[[:space:]]*$//')"
+
+    # Ignore blanks.
+    [ -z "$line" ] && continue
+
+    # Ignore normal comments, but keep '# CONFIG_FOO is not set'.
+    case "$line" in
+        "# CONFIG_"*" is not set") ;;
+        "#"*) continue ;;
+    esac
+
+    expected="$(normalize_expected_line "$line" || true)"
+    [ -z "${expected:-}" ] && continue
+
+    check_expected_line "$expected"
+done < "$EXPECTED_FILE"
+
+if [ "$failed" -ne 0 ]; then
+    echo "error: resolved kernel config does not satisfy $EXPECTED_FILE" >&2
+    exit 1
+fi
+
+echo "kernel config satisfies $EXPECTED_FILE"

kernel-extra.config

@@ -3,81 +3,39 @@
 ###############################################################################
 
 CONFIG_HWMON=y
-# Hardware monitoring framework. Needed so sensor drivers can expose temps/fans.
-
 CONFIG_I2C=y
-# Core I2C subsystem. Required by your RTC/fan controller drivers.
-
 CONFIG_SENSORS_EMC2305=y
-# EMC2305 fan controller driver. Built-in so fan control is available early.
-
 CONFIG_RTC_DRV_PCF2127=y
-# RTC driver for PCF2127. Built-in so timekeeping is available early.
 
 
 ###############################################################################
 # Namespaces
-# These are fundamental container primitives. Keep these built-in.
 ###############################################################################
 
 CONFIG_NAMESPACES=y
-# Master switch for Linux namespaces.
-
 CONFIG_UTS_NS=y
-# Isolates hostname/domainname per container.
-
 CONFIG_IPC_NS=y
-# Isolates SysV IPC and POSIX message queues between containers.
-
 CONFIG_PID_NS=y
-# Gives containers their own PID tree (so processes inside see their own PID 1).
-
 CONFIG_NET_NS=y
-# Gives containers their own network stack, interfaces, routing, etc.
-
 CONFIG_USER_NS=y
-# User namespaces. Useful for modern container behavior and future flexibility.
-# Not every setup strictly needs this on day one, but I would enable it.
 
 
 ###############################################################################
 # Cgroups / resource control
-# Required for kubelet/CRI-O to manage resource isolation.
 ###############################################################################
 
 CONFIG_CGROUPS=y
-# Master switch for cgroups.
-
 CONFIG_CGROUP_BPF=y
-# Allows BPF programs to be attached to cgroups. Not required for first boot,
+CONFIG_CGROUP_NET_CLASSID=y
-# but modern systems increasingly expect this.
-
 CONFIG_CGROUP_FREEZER=y
-# Allows freezing/thawing process groups. Useful for container lifecycle control.
-
 CONFIG_CGROUP_PIDS=y
-# Limits number of processes in a cgroup.
-
 CONFIG_CGROUP_DEVICE=y
-# Controls device access from containers.
-
 CONFIG_CPUSETS=y
-# CPU affinity partitioning by cgroup.
-
 CONFIG_MEMCG=y
-# Memory cgroup support. Critical for container memory accounting/limits.
-
 CONFIG_BLK_CGROUP=y
-# Block IO control/accounting for cgroups.
-
 CONFIG_CGROUP_SCHED=y
-# Scheduler integration for cgroups.
-
 CONFIG_FAIR_GROUP_SCHED=y
-# Fair scheduler group support for cgroups.
-
 CONFIG_CFS_BANDWIDTH=y
-# CPU quota/limit support. Important for kubelet resource enforcement.
 
 
 ###############################################################################
@@ -85,23 +43,20 @@ CONFIG_CFS_BANDWIDTH=y
 ###############################################################################
 
 CONFIG_KEYS=y
-# Kernel key retention service. Commonly relied on by container/userland tooling.
-
 CONFIG_TMPFS=y
-# Tmpfs support. Containers and runtimes rely on this heavily.
-
 CONFIG_TMPFS_XATTR=y
-# Extended attributes on tmpfs. Useful for container runtime behavior.
-
 CONFIG_TMPFS_POSIX_ACL=y
-# POSIX ACLs on tmpfs. Good compatibility feature for userland.
-
 CONFIG_OVERLAY_FS=y
-# Overlay filesystem. This is the big one for container image/layer storage.
-# Module is fine; CRI-O can load/use it after boot. No need to bloat FIT image.
-
 CONFIG_FS_POSIX_ACL=y
-# General POSIX ACL support. Good to have for overlay/tmpfs behavior.
+CONFIG_FHANDLE=y
+CONFIG_AUTOFS_FS=y
+
+CONFIG_PROC_FS=y
+CONFIG_SYSFS=y
+CONFIG_DEVTMPFS=y
+CONFIG_DEVTMPFS_MOUNT=y
+
+CONFIG_BLK_DEV_LOOP=y
 
 
 ###############################################################################
@@ -109,171 +64,144 @@ CONFIG_FS_POSIX_ACL=y
 ###############################################################################
 
 CONFIG_INET=y
-# IPv4 stack.
-
 CONFIG_IPV6=y
-# IPv6 stack. You may be tempted to disable it, but Kubernetes/container stacks
-# increasingly assume it exists. Keep it on unless you have a hard reason not to.
-
 CONFIG_UNIX=y
-# Unix domain sockets. Containers and runtimes absolutely rely on this.
 
 CONFIG_TUN=m
-# TUN/TAP device support. Commonly used by networking tools/VPN/CNI-related flows.
-# Module is fine.
-
 CONFIG_DUMMY=m
-# Dummy network interface. Sometimes useful for CNI/network setups and testing.
+CONFIG_VETH=y
+CONFIG_BRIDGE=y
+CONFIG_BRIDGE_NETFILTER=y
+CONFIG_VXLAN=y
+
+# Enables IPv4/IPv6 policy routing and multiple routing tables.
+# Required by CNIs such as Cilium for ip-rule based routing.
+CONFIG_IP_ADVANCED_ROUTER=y
+CONFIG_IP_MULTIPLE_TABLES=y
+CONFIG_IPV6_MULTIPLE_TABLES=y
 
 
 ###############################################################################
-# Netfilter / packet filtering / NAT
+# Netfilter base
-# This is where container networking gets messy. Better to enable a sane baseline.
 ###############################################################################
 
 CONFIG_NETFILTER=y
-# Netfilter core framework. Module is okay if your setup loads it before use.
-
 CONFIG_NETFILTER_ADVANCED=y
-# Exposes more advanced netfilter options and modules.
+CONFIG_NETFILTER_NETLINK=y
+CONFIG_NETFILTER_XTABLES=y
+
+# Linux 6.17+ gates legacy iptables/xtables support behind these options.
+# Without these, IP_NF_* / IP6_NF_* options may silently fall back to =m
+# or disappear after olddefconfig.
+CONFIG_NETFILTER_XTABLES_LEGACY=y
 
 CONFIG_NF_CONNTRACK=y
-# Connection tracking. Critical for NAT, Kubernetes service traffic, and many CNIs.
-
 CONFIG_NF_NAT=y
-# NAT framework. Required for masquerading and pod egress in many setups.
-
-CONFIG_NF_TABLES=y
-# nftables framework. Modern Linux packet filtering backend.
-
-CONFIG_NFT_CT=y
-# nftables conntrack expressions.
-
-CONFIG_NFT_COUNTER=y
-# nftables packet/byte counters
-
-CONFIG_NFT_CHAIN_NAT=y
-# nftables NAT chain support.
-
-CONFIG_NFT_MASQ=y
-# nftables masquerade support. Often needed for pod egress NAT.
-
-CONFIG_NFT_REDIR=y
-# nftables redirect target.
-
-CONFIG_NFT_NAT=y
-# nftables NAT support.
-
-CONFIG_NF_NAT_IPV4=y
-# IPv4 NAT helper support. Some kernels still expose this separately.
-
-CONFIG_NF_NAT_IPV6=y
-# IPv6 NAT helper support.
-
 CONFIG_NF_CT_NETLINK=y
-# userspace netlink access to the conntrack table; kube-proxy uses this for conntrack listing/cleanup
-
-CONFIG_NF_CT_NETLINK_TIMEOUT=y
-# userspace netlink support for conntrack timeout objects
-
-CONFIG_NF_CT_NETLINK_HELPER=y
-# userspace netlink support for conntrack helper objects
-
-CONFIG_IP_NF_IPTABLES=y
-# iptables compatibility for IPv4. Still useful because lots of CNI/plugin code
-# still expects iptables even on nft-backed systems.
-
-CONFIG_IP_NF_NAT=y
-# IPv4 NAT support for iptables compatibility.
-
-CONFIG_IP6_NF_IPTABLES=y
-# ip6tables compatibility.
-
-CONFIG_IP6_NF_FILTER=y
-# IPv6 "filter" table (same as above but for IPv6)
 
 CONFIG_NF_REJECT_IPV4=y
-# core IPv4 reject logic used by netfilter/iptables/nftables
+CONFIG_NF_REJECT_IPV6=y
 
+# Do not re-add these stale / absent symbols for this NXP 6.18 tree:
+#
+#   CONFIG_NF_NAT_IPV4
+#   CONFIG_NF_NAT_IPV6
+#   CONFIG_NFT_CHAIN_NAT
+#   CONFIG_NFT_COUNTER
+#   CONFIG_NETFILTER_XT_TARGET_REJECT
+#
+# Use the currently valid symbols instead:
+#
+#   CONFIG_NF_NAT
+#   CONFIG_IP_NF_NAT
+#   CONFIG_IP6_NF_NAT
+#   CONFIG_IP_NF_TARGET_REJECT
+#   CONFIG_IP6_NF_TARGET_REJECT
+#   CONFIG_NFT_REJECT
+#   CONFIG_NFT_REJECT_INET
+#
+# Also avoid enabling these unless there is a real need:
+#
+#   CONFIG_NF_CT_NETLINK_TIMEOUT
+#   CONFIG_NF_CT_NETLINK_HELPER
+#
+# They exist in this tree, but pull in extra dependencies and are not required
+# for basic Kubernetes/Cilium bring-up.
+
+
+###############################################################################
+# nftables backend
+###############################################################################
+
+CONFIG_NF_TABLES=y
+CONFIG_NF_TABLES_INET=y
+
+CONFIG_NFT_CT=y
+CONFIG_NFT_MASQ=y
+CONFIG_NFT_REDIR=y
+CONFIG_NFT_NAT=y
 CONFIG_NFT_REJECT=y
-# nftables equivalent of REJECT (needed for nf_tables backend compatibility)
+CONFIG_NFT_REJECT_INET=y
 
+###############################################################################
+# nftables FIB expression
+#
+# Required by CNI hostport nftables rules such as:
+#   fib daddr type local goto hostports
+###############################################################################
+
+CONFIG_NFT_FIB=y
+CONFIG_NFT_FIB_INET=y
+CONFIG_NFT_FIB_IPV4=y
+CONFIG_NFT_FIB_IPV6=y
+
+###############################################################################
+# IPv4 iptables compatibility
+###############################################################################
+
+CONFIG_IP_NF_IPTABLES=y
+CONFIG_IP_NF_IPTABLES_LEGACY=y
 CONFIG_IP_NF_FILTER=y
-# IPv4 "filter" table (INPUT/FORWARD/OUTPUT chains for iptables)
+CONFIG_IP_NF_MANGLE=y
-
+CONFIG_IP_NF_NAT=y
 CONFIG_IP_NF_TARGET_REJECT=y
-# IPv4-specific REJECT target for legacy iptables
 
+
+###############################################################################
+# IPv6 iptables compatibility
+###############################################################################
+
+CONFIG_IP6_NF_IPTABLES=y
+CONFIG_IP6_NF_IPTABLES_LEGACY=y
+CONFIG_IP6_NF_FILTER=y
+CONFIG_IP6_NF_MANGLE=y
+CONFIG_IP6_NF_NAT=y
 CONFIG_IP6_NF_TARGET_REJECT=y
-# IPv6-specific REJECT target for legacy iptables
 
-CONFIG_IP_SET=m
-# IP sets. Useful for some network policies / firewalling toolchains.
 
-CONFIG_NETFILTER_NETLINK_ACCT=y
+###############################################################################
-# netfilter accounting subsystem used for nfacct-based kube-proxy metrics
+# xtables matches / targets
-
+###############################################################################
-CONFIG_NETFILTER_XT_MATCH_NFACCT=y
-# iptables nfacct match that hooks rules into the netfilter accounting subsystem
 
 CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=y
-# xtables match for address types. Often used in iptables rules.
-
-CONFIG_NETFILTER_XT_TARGET_REJECT=y
-# iptables REJECT target (actively reject packets instead of silently dropping)
-
 CONFIG_NETFILTER_XT_MATCH_COMMENT=y
-# Allows comments in iptables rules. Not critical, but harmless and useful.
-
 CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y
-# xtables conntrack matching.
+CONFIG_NETFILTER_XT_MATCH_MARK=y
-
-CONFIG_NETFILTER_XT_MATCH_STATISTIC=y
-# iptables "statistic" match used for probabilistic packet matching / load balancing
-
 CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y
-# Match multiple ports in one rule.
+CONFIG_NETFILTER_XT_MATCH_SOCKET=y
-
+CONFIG_NETFILTER_XT_MATCH_STATISTIC=y
 CONFIG_NETFILTER_XT_MATCH_TCPMSS=y
-# Useful for TCP MSS clamping in some network paths.
-
-CONFIG_NETFILTER_XT_TARGET_MASQUERADE=y
-# iptables MASQUERADE target. Very commonly needed for pod outbound NAT.
-
-CONFIG_NETFILTER_XT_TARGET_REDIRECT=y
-# Redirect target.
-
-CONFIG_NETFILTER_XT_TARGET_MARK=y
-# Packet marking support. Useful for advanced networking/routing rules.
 
 CONFIG_NETFILTER_XT_TARGET_CT=y
-# Connection tracking target for xtables.
+CONFIG_NETFILTER_XT_TARGET_MARK=y
+CONFIG_NETFILTER_XT_TARGET_MASQUERADE=y
+CONFIG_NETFILTER_XT_TARGET_REDIRECT=y
+CONFIG_NETFILTER_XT_TARGET_TPROXY=y
 
-# Optional. Good only if you know you need transparent proxying.
+CONFIG_NETFILTER_NETLINK_ACCT=y
-# Not required for initial CRI-O bring-up.
+CONFIG_NETFILTER_XT_MATCH_NFACCT=y
-# CONFIG_NETFILTER_XT_TARGET_TPROXY=m
 
-
+CONFIG_IP_SET=m
-###############################################################################
-# Bridge / container interface plumbing
-###############################################################################
-
-CONFIG_VETH=y
-# Virtual Ethernet pairs. This is how container interfaces are commonly connected
-# to the host/network namespace.
-
-CONFIG_BRIDGE=y
-# Ethernet bridge support. Needed by bridge-based CNIs.
-
-CONFIG_BRIDGE_NETFILTER=y
-# Allows bridged traffic to pass through netfilter/iptables/nftables hooks.
-# Important for Kubernetes networking behavior.
-
-# Optional / version-dependent:
-# Some kernels expose additional ebtables/bridge netfilter pieces separately.
-# Keep this if your kernel has it, but don't panic if it doesn't.
-CONFIG_BRIDGE_NF_EBTABLES=y
-# Bridge filtering via ebtables compatibility. Sometimes useful, not always critical.
 
 
 ###############################################################################
@@ -281,24 +209,10 @@ CONFIG_BRIDGE_NF_EBTABLES=y
 ###############################################################################
 
 CONFIG_SECCOMP=y
-# Secure computing mode. Lets runtimes restrict syscall surface.
-
 CONFIG_SECCOMP_FILTER=y
-# BPF-based seccomp filters. This is the useful seccomp mode for containers.
-
-# AppArmor / SELinux are optional depending on distro/security model.
-# Alpine often won't use AppArmor by default; that's fine for first bring-up.
-
-# If your kernel tree has these and you care later:
-# CONFIG_SECURITY=y
-# CONFIG_SECURITYFS=y
 
 CONFIG_SECURITY_NETWORK=y
-# enables Security Module (LSM) hooks for network operations. CoreDNS needs this
-
 CONFIG_SECURITY_PATH=y
-# Recommended for container isolation
-
 CONFIG_SECURITY_NETWORK_XFRM=y
 
 
@@ -307,59 +221,28 @@ CONFIG_SECURITY_NETWORK_XFRM=y
 ###############################################################################
 
 CONFIG_POSIX_MQUEUE=y
-# POSIX message queues. Containers/apps sometimes rely on this.
-
 CONFIG_EPOLL=y
-# Event polling. Usually already enabled; standard modern userspace feature.
-
 CONFIG_SIGNALFD=y
-# File-descriptor-based signal delivery. Common Linux userspace feature.
-
 CONFIG_TIMERFD=y
-# File-descriptor timers. Common Linux userspace feature.
-
 CONFIG_EVENTFD=y
-# Event notification file descriptors. Common Linux userspace feature.
-
 CONFIG_MEMFD_CREATE=y
-# Anonymous memory-backed file creation. Widely used by modern software.
 
-CONFIG_FHANDLE=y
+
-# File handle support. Useful for container/runtime operations.
+###############################################################################
+# Disable unused platform/virtualization pieces
+###############################################################################
 
 CONFIG_DMIID=n
-# Optional on embedded boards; usually not needed unless your tree selects it.
 
-
-###############################################################################
-# Storage / block / other practical container bits
-###############################################################################
-
-CONFIG_BLK_DEV_LOOP=y
-# Loop devices. Often useful for image/layer tooling or debugging.
-# Could be =m too, but built-in is harmless and often convenient.
-
-CONFIG_AUTOFS_FS=y
-# Automount filesystem support. Not strictly required for CRI-O, but harmless.
-
-CONFIG_PROC_FS=y
-# /proc support. Essential.
-
-CONFIG_SYSFS=y
-# /sys support. Essential.
-
-CONFIG_DEVTMPFS=y
-# Kernel-managed /dev population support.
-
-CONFIG_DEVTMPFS_MOUNT=y
-# Automatically mount devtmpfs. Very practical on small/custom systems.
-
-### Disable XEN because it breaks our build and we don't need it
 CONFIG_XEN=n
 CONFIG_XEN_DOM0=n
 CONFIG_VHOST_XEN=n
 
-### For Disk IO diagnostics
+
+###############################################################################
+# Disk IO diagnostics
+###############################################################################
+
 CONFIG_TASK_DELAY_ACCT=y
 CONFIG_TASK_IO_ACCOUNTING=y
 CONFIG_TASKSTATS=y

macos/flashusb.sh

@@ -2,7 +2,9 @@
 set -eu
 
 ### User-configurable vars
-IMG_DIR="./out"
+SCRIPT_DIR="$(dirname "${BASH_SOURCE[0]}")"
+OUT_DIR="$( realpath "$SCRIPT_DIR"/../out/ )"
+IMG_DIR="$OUT_DIR"
 CONFIG_PART_SUFFIX="s1"           # config partition after flashing
 ENV_SEARCH_DIR="./out"            # optional; leave empty to use selected image dir
 

makefile

@@ -1,4 +1,5 @@
 include build.env
+-include build.env.work
 export
 
 TAG ?= dev
@@ -26,9 +27,8 @@ CLITOOLS_BIN := bin/ctl-linux-$(ARCH)-$(TAG)
 CONFIGS_DIR          := configs
 SCRIPTS_DIR          := scripts
 CLUSTER_ENV_DEFAULT  := $(CONFIGS_DIR)/cluster.env.default
+CLUSTER_ENV_WORK     := $(CONFIGS_DIR)/cluster.env.work
 CLUSTER_ENV          := $(OUT_DIR)/cluster.env
-NODE_ENV_DEFAULT     := configs/node.env.default
-NODE_ENV             := $(OUT_DIR)/node.env
 
 BOARD_ITB     := $(OUT_DIR)/board.itb
 INITRAMFS     := $(OUT_DIR)/initramfs.cpio.gz
@@ -38,7 +38,7 @@ PART_IMAGE    := $(OUT_DIR)/monok8s-$(KUBE_VERSION)-$(TAG).ext4.zst
 KERNEL_IMAGE  := $(OUT_DIR)/Image.gz
 
 BUILD_BASE_STAMP        := $(OUT_DIR)/.build-base-$(TAG).stamp
-DOWNLOAD_PACKAGES_STAMP := $(PACKAGES_DIR)/.download-packages.stamp
+DOWNLOAD_PACKAGES_STAMP := $(PACKAGES_DIR)/.stamp-$(KUBE_VERSION)-$(CRIO_VERSION)
 
 ALPINE_SERIES := $(word 1,$(subst ., ,$(ALPINE_VER))).$(word 2,$(subst ., ,$(ALPINE_VER)))
 
@@ -147,7 +147,11 @@ $(BUILD_BASE_STAMP): $(BUILD_BASE_DEPS) | $(OUT_DIR)
 		--build-arg APT_PROXY=$(APT_PROXY) \
 		--build-arg TAG=$(TAG) \
 		-t $(DOCKER_IMAGE_ROOT)/build-base:$(TAG) .
-	@iid=$$(docker image inspect monok8s/build-base:$(TAG) | jq -r '.[].Id' | cut -d':' -f2 | cut -c -8); \
+	@iid=$$(docker image inspect \
+		--format '{{.Id}}' \
+		$(DOCKER_IMAGE_ROOT)/build-base:$(TAG) \
+		| cut -d':' -f2 \
+		| cut -c -8); \
 	docker tag monok8s/build-base:$(TAG) monok8s/build-base:$$iid; \
 	touch $@
 
@@ -177,10 +181,14 @@ $(INITRAMFS): $(INITRAMFS_DEPS) $(DOWNLOAD_PACKAGES_STAMP) | $(OUT_DIR)
 	test -f $@
 
 $(CLITOOLS_BIN): $(CLITOOLS_SRCS)
-	$(MAKE) -C clitools build-agent
+	$(MAKE) -C clitools build-local VERSION="$(TAG)"
 
 vpp: $(BUILD_BASE_STAMP) $(VPP_TAR) $(DPDK_TAR) $(FMLIB_TAR) $(FMC_TAR) $(NXP_TAR)
-	@build_base_tag=$$(docker image inspect $(DOCKER_IMAGE_ROOT)/build-base:$(TAG) | jq -r '.[0].Id' | cut -d':' -f2 | cut -c -8); \
+	@build_base_tag=$$(docker image inspect \
+		--format '{{.Id}}' \
+		$(DOCKER_IMAGE_ROOT)/build-base:$(TAG) \
+		| cut -d':' -f2 \
+		| cut -c -8); \
 	@mkdir -p $(OUT_DIR)/vpp
 	docker build \
 		-f docker/vpp.Dockerfile \
@@ -219,7 +227,11 @@ $(BOARD_ITB): $(ITB_DEPS) | $(OUT_DIR)
 	test -f $@
 
 $(RELEASE_IMAGE): $(RELEASE_DEPS) $(DOWNLOAD_PACKAGES_STAMP) | $(OUT_DIR)
-	@build_base_tag=$$(docker image inspect $(DOCKER_IMAGE_ROOT)/build-base:$(TAG) | jq -r '.[0].Id' | cut -d':' -f2 | cut -c -8); \
+	@build_base_tag=$$(docker image inspect \
+		--format '{{.Id}}' \
+		$(DOCKER_IMAGE_ROOT)/build-base:$(TAG) \
+		| cut -d':' -f2 \
+		| cut -c -8); \
 	docker build \
 		-f docker/alpine.Dockerfile \
 		--no-cache \
@@ -264,8 +276,15 @@ $(RELEASE_IMAGE): $(RELEASE_DEPS) $(DOWNLOAD_PACKAGES_STAMP) | $(OUT_DIR)
 
 # ---- config targets ------------------------------------------------------------
 
-cluster-config: $(CLUSTER_ENV_DEFAULT) $(SCRIPTS_DIR)/merge-env.sh | $(OUT_DIR)
+cluster-config: $(CLUSTER_ENV_DEFAULT) $(CLUSTER_ENV_WORK) $(SCRIPTS_DIR)/merge-env.sh | $(OUT_DIR)
+	@rm -f $(CLUSTER_ENV)
 	sh $(SCRIPTS_DIR)/merge-env.sh $(CLUSTER_ENV_DEFAULT) $(CLUSTER_ENV)
+	@if [ -f "$(CLUSTER_ENV_WORK)" ]; then \
+		echo "Merging $(CLUSTER_ENV_WORK) into $(CLUSTER_ENV)"; \
+		sh $(SCRIPTS_DIR)/merge-env.sh $(CLUSTER_ENV_WORK) $(CLUSTER_ENV); \
+	else \
+		echo "No $(CLUSTER_ENV_WORK), using defaults only"; \
+	fi
 
 cluster-defconfig: $(CLUSTER_ENV_DEFAULT) | $(OUT_DIR)
 	cp $(CLUSTER_ENV_DEFAULT) $(CLUSTER_ENV)

scripts/merge-env.sh

@@ -6,6 +6,25 @@ OUTPUT="${2:?output file required}"
 
 mkdir -p "$(dirname "$OUTPUT")"
 
+TMP="$(mktemp)"
+BASE_CREATED=0
+
+if [ -f "$OUTPUT" ]; then
+    BASE="$OUTPUT"
+else
+    BASE="$(mktemp)"
+    BASE_CREATED=1
+    : > "$BASE"
+fi
+
+cleanup() {
+    rm -f "$TMP"
+    if [ "$BASE_CREATED" = "1" ]; then
+        rm -f "$BASE"
+    fi
+}
+trap cleanup EXIT INT TERM
+
 awk '
 function trim(s) {
     sub(/^[[:space:]]+/, "", s)
@@ -13,33 +32,76 @@ function trim(s) {
     return s
 }
 
-BEGIN {
+function parse_key(line,    eq, key) {
-    for (k in ENVIRON) {
-        env[k] = ENVIRON[k]
-    }
-}
-
-/^[[:space:]]*#/ || /^[[:space:]]*$/ {
-    print
-    next
-}
-
-{
-    line = $0
     eq = index(line, "=")
-
     if (eq == 0) {
-        print line
+        return ""
-        next
     }
 
     key = trim(substr(line, 1, eq - 1))
-    val = substr(line, eq + 1)
 
-    if (key in env) {
+    if (key !~ /^[A-Za-z_][A-Za-z0-9_]*$/) {
-        print key "=" env[key]
+        return ""
-    } else {
+    }
+
+    return key
+}
+
+function merged_line(key, line) {
+    if (key ~ /^MKS_/ && key in ENVIRON) {
+        return key "=" ENVIRON[key]
+    }
+
+    return line
+}
+
+# First file: INPUT
+phase == 1 {
+    line = $0
+    key = parse_key(line)
+
+    if (key != "") {
+        incoming[key] = merged_line(key, line)
+
+        if (!(key in input_seen)) {
+            input_order[++input_count] = key
+            input_seen[key] = 1
+        }
+    }
+
+    next
+}
+
+# Second file: existing OUTPUT / BASE
+phase == 2 {
+    line = $0
+    key = parse_key(line)
+
+    if (key != "" && key in incoming) {
+        print incoming[key]
+        written[key] = 1
+        next
+    }
+
     print line
+
+    if (key != "") {
+        written[key] = 1
+    }
+
+    next
+}
+
+END {
+    for (i = 1; i <= input_count; i++) {
+        key = input_order[i]
+
+        if (!(key in written)) {
+            print incoming[key]
+            written[key] = 1
+        }
     }
 }
-' "$INPUT" > "$OUTPUT"
+' phase=1 "$INPUT" phase=2 "$BASE" > "$TMP"
+
+mv "$TMP" "$OUTPUT"