Added missing kconfig

.gitignore

@@ -1,3 +1,6 @@
+build.env.work
+cluster.env.work
+.DS_Store
 clitools/bin
 packages/
 out/

README.md

@@ -1,84 +1,195 @@
 # monok8s
-An Alpine-based Kubernetes cluster image for Mono's Gateway Development Kit
 
-https://docs.mono.si/gateway-development-kit/getting-started
+This is an Alpine-based Kubernetes image for Mono's Gateway Development Kit.
 
-## Features
- * A/B deployment
- * Read-only RootFS
- * k8s style OS upgrade (see Upgrading)
+It gives you a ready-to-boot Kubernetes control-plane image so you can get your device running first, then learn and customize from there.
 
-## IMPORTANT NOTES
- * This is not your everyday linux image! It is best suited for users that is already familiar
-   with k8s. For first-timers, you may want to try the default config that gives you a ready-to-used
-   cluster then get yourself started from there
+Project/device docs: <https://docs.mono.si/gateway-development-kit/getting-started>
 
- * The 3 RJ45 ports are label in eth1, eth2, eth0 respectively by the kernel (left to right)
-   So `ip addr eth0` is your right most port
+---
 
-### Table of Contents
-1. Flashing
-   - [USB](docs/flashing-usb.md)
-   - [Over network (tftp)](docs/flashing-network.md)
-2. [Upgrading](docs/ota.md)
-3. Getting shell access to the host
-   - UART. The thing you did in first time flashing.
-   - [Install an ssh pod](docs/installing-ssh-pod.md) (Recommended)
+## What you get
 
-## Build
-Find the latest package versions and update build.env
- * [kernel](https://github.com/nxp-qoriq/linux/archive/refs/tags/)
- * [busybox](https://github.com/mirror/busybox/archive/refs/tags/)
- * [CRI-O](https://github.com/cri-o/cri-o/releases)
- * [Kubelet](https://kubernetes.io/releases/download/)
- * [UBoot](https://github.com/u-boot/)
+The default image boots into a small Kubernetes control-plane environment with:
 
-Then run
+- Alpine Linux
+- Kubernetes initialized through `kubeadm`
+- read-only root filesystem layout
+- A/B rootfs layout for safer OS upgrades
+- a Kubernetes-style OS upgrade path through `OSUpgrade`
+
+You do **not** need to know Go or understand the internal build system to try the image.
+
+---
+
+## Before you start
+
+You need:
+
+- a Linux build machine or VM
+- Docker
+- `make`
+- basic command-line comfort
+
+If you are building on a fresh Debian machine, you can install the usual build dependencies with:
+
+```bash
+devtools/setup-build-host.sh
 ```
+
+Or install the minimum packages yourself:
+
+```bash
+sudo apt-get update
+sudo apt-get install -y docker.io make qemu-user-static binfmt-support
+```
+
+Make sure your user can run Docker, or use `sudo` where needed.
+
+---
+
+## Fast path: build an image
+
+Download the project tarball, extract it, then run:
+
+```bash
 make release
 ```
 
-The default configuration will boot as a first time control-plane. Adjust to your own needs.
+When the build finishes, check the `out/` directory for the generated image artifacts.
 
-For control-plane
-```
-make cluster-config \
-  MKS_HOSTNAME=monok8s-master \
-  MKS_CLUSTER_ROLE=control-plane \
-  MKS_INIT_CONTROL_PLANE=true \
-  MKS_MGMT_ADDRESS=10.0.0.10/24 \
-  MKS_APISERVER_ADVERTISE_ADDRESS=10.0.0.10
-```
+That is the main path most users should try first.
 
-For worker
-```
-make cluster-config \
-  MKS_HOSTNAME=monok8s-worker \
-  MKS_CLUSTER_ROLE=worker \
-  MKS_INIT_CONTROL_PLANE=no \
-  MKS_MGMT_ADDRESS=10.0.0.10/24 \
-  MKS_APISERVER_ADVERTISE_ADDRESS=10.0.0.10 \
-  MKS_API_SERVER_ENDPOINT=10.0.0.1:6443 \
-  MKS_CNI_PLUGIN=none \
-  MKS_BOOTSTRAP_TOKEN=abcd12.ef3456789abcdef0 \
-  MKS_DISCOVERY_TOKEN_CA_CERT_HASH=sha256:9f1c2b3a4d5e6f7890abc1234567890abcdef1234567890abcdef1234567890ab
-```
+---
 
-Check inside [configs/cluster.env.default](configs/cluster.env.default) for configuration details
+## Flash the image
+
+After building, flash the generated image to your device.
+
+Start with one of these guides:
+
+- [Flash over USB](docs/flashing-usb.md)
+- [Flash over network / TFTP](docs/flashing-network.md)
+
+USB flashing is usually the easiest path when you are setting up the device for the first time.
+
+---
+
+## First boot
+
+The default configuration is intended to boot as a first-time Kubernetes control-plane node.
+
+Default-style control-plane configuration looks like this:
 
-### Making sub stages
 ```bash
-make build-base # The image that builds the kernel and everything
-make kernel     # Builds our kernel from NXP
-make initramfs
-make itb        # Builds out/board.itb (contains the kernel and the initramfs)
+make cluster-config \
+	MKS_HOSTNAME=monok8s-master \
+	MKS_CLUSTER_ROLE=control-plane \
+	MKS_INIT_CONTROL_PLANE=true \
+	MKS_MGMT_ADDRESS=10.0.0.10/24 \
+	MKS_APISERVER_ADVERTISE_ADDRESS=10.0.0.10
 ```
 
-## NOTES
-### The device's dts files are located at here
-https://github.com/we-are-mono/OpenWRT-ASK/tree/mono-25.12.0-rc3/target/linux/layerscape/files/arch/arm64/boot/dts/freescale
-* We need both `mono-gateway-dk-sdk.dts` and `mono-gateway-dk.dts` since the sdk one includes the non-sdk one.
-* The actual dts being used is the `mono-gateway-dk-sdk.dts`
+If you are just trying the image for the first time, start with the default control-plane setup. Worker-node setup is still incomplete.
 
-## DISCLAIMER
-USE AT YOUR OWN RISKS. I leverage ChatGPT heavily for this.
+For all available configuration values, see:
+
+- [configs/cluster.env.default](configs/cluster.env.default)
+
+For worker node
+```
+make cluster-config \
+	MKS_HOSTNAME=monok8s-worker \
+	MKS_CLUSTER_ROLE=worker \
+	MKS_INIT_CONTROL_PLANE=no \
+	MKS_MGMT_ADDRESS=10.0.0.10/24 \
+	MKS_APISERVER_ADVERTISE_ADDRESS=10.0.0.10 \
+	MKS_API_SERVER_ENDPOINT=10.0.0.1:6443 \
+	MKS_CNI_PLUGIN=none \
+	MKS_BOOTSTRAP_TOKEN=abcd12.ef3456789abcdef0 \
+	MKS_DISCOVERY_TOKEN_CA_CERT_HASH=sha256:9f1c2b3a4d5e6f7890abc1234567890abcdef1234567890abcdef1234567890ab
+```
+
+---
+
+## Getting shell access
+
+For first-time setup, UART is the most direct option because it is already part of the flashing process.
+
+After the device is running, the recommended path is:
+
+- [Install an SSH pod](docs/installing-ssh-pod.md)
+
+---
+
+## Upgrading
+
+monok8s includes a Kubernetes-style OS upgrade flow using the `OSUpgrade` custom resource.
+
+See:
+
+- [OTA upgrade guide](docs/ota.md)
+
+The currently tested upgrade chain is:
+
+- `1.33.3 -> 1.33.10`
+- `1.33.10 -> 1.34.6`
+- `1.34.6 -> 1.35.3`
+
+Tested worker node upgrade chain:
+
+- `1.33.3 -> 1.34.1`
+- `1.33.1 -> 1.35.3`
+
+---
+
+## Current status
+
+This project is usable for experimenting with a single control-plane device image, but it is still a development project.
+
+Working today:
+
+- initramfs boot flow
+- Alpine boot
+- Kubernetes control-plane bootstrap
+- Kubernetes worker-node
+- default bridge CNI
+- Cilium
+
+Still in progress:
+
+- VPP/DPAA networking experiments
+
+---
+
+## Common build issue
+
+### `chroot: failed to run command '/bin/sh': Exec format error`
+
+This usually means the build host cannot run ARM64 binaries.
+
+On Debian, install ARM64 emulation support:
+
+```bash
+sudo apt-get install -y qemu-user-static binfmt-support
+```
+
+Then run the build again:
+
+```bash
+make release
+```
+
+---
+
+## Notes
+
+This is not a general-purpose Linux distribution. It is a device image for experimenting with Kubernetes on Mono's Gateway Development Kit.
+
+The safest path is:
+
+1. build the default image,
+2. flash it,
+3. boot the control-plane,
+4. confirm Kubernetes is running,
+5. customize only after the base image works.

alpine/build-rootfs.sh

@@ -2,9 +2,11 @@
 
 set -euo pipefail
 
+source /utils.sh
+
 /preload-k8s-images.sh || exit 1
 
-export CTL_BIN_LAYER=$( skopeo inspect docker-daemon:localhost/monok8s/control-agent:dev | jq -r '.Layers[0] | sub("^sha256:"; "")' )
+export CTL_BIN_LAYER=$( skopeo inspect docker-daemon:localhost/monok8s/node-control:$TAG | jq -r '.Layers[0] | sub("^sha256:"; "")' )
 
 mkdir -p \
   "$ROOTFS/dev" \
@@ -17,6 +19,7 @@ mkdir -p \
   "$ROOTFS/build" \
   "$ROOTFS/var/cache/apk" \
   "$ROOTFS/usr/lib/monok8s/crds" \
+  "$ROOTFS/usr/lib/monok8s/migrations.d/k8s" \
   "$ROOTFS/opt/monok8s/config"
 
 mount --bind /var/cache/apk "$ROOTFS/var/cache/apk"
@@ -30,6 +33,15 @@ cp /etc/resolv.conf "$ROOTFS/etc/resolv.conf"
 cp /build/crio.tar.gz "$ROOTFS/build/"
 cp /build/crds/*.yaml "$ROOTFS/usr/lib/monok8s/crds"
 
+KUBE_MINOR=$(printf '%s\n' "$KUBE_VERSION" | sed -E 's/^v?([0-9]+\.[0-9]+).*/\1/')
+MIG_SRC="/build/migrations/k8s/$KUBE_MINOR"
+MIG_DST="$ROOTFS/usr/lib/monok8s/migrations.d/k8s/$KUBE_MINOR"
+
+if [ -d "$MIG_SRC" ]; then
+    mkdir -p "$MIG_DST"
+    cp -a "$MIG_SRC"/. "$MIG_DST"/
+fi
+
 chroot "$ROOTFS" /bin/sh -c "ln -s /var/cache/apk /etc/apk/cache"
 # chroot "$ROOTFS" /bin/sh -c "apk update"
 chroot "$ROOTFS" /bin/sh -c "apk add bash curl"
@@ -64,7 +76,7 @@ DISK_SIZE="${DISK_SIZE:-8G}"
 
 ROOTFS_IMG="${ROOTFS_IMG:-rootfs.ext4}"
 ROOTFS_IMG_ZST="${ROOTFS_IMG_ZST:-rootfs.ext4.zst}"
-ROOTFS_PART_SIZE_MIB="${ROOTFS_PART_SIZE_MIB:-2048}"
+ROOTFS_PART_SIZE_MIB="${ROOTFS_PART_SIZE_MIB:-2560}"
 
 FAKE_DEV="/tmp/dev"
 MNT_ROOTFS_IMG="/mnt/rootfs-img"
@@ -107,6 +119,8 @@ mkdir -p "$FAKE_DEV" "$MNT_ROOTFS_IMG" "$MNT_DATA"
 
 echo "##################################################### Packaging RootFS $(du -sh "$ROOTFS" | awk '{print $1}')"
 
+ensure_loop_ready
+
 ###############################################################################
 # 1. Build reusable rootfs ext4 image once
 ###############################################################################
@@ -183,8 +197,8 @@ truncate -s "$DISK_SIZE" "$IMG"
 
 sgdisk -o "$IMG" \
   -n 1:2048:+64M  -t 1:0700 -c 1:config \
-  -n 2:0:+2G      -t 2:8300 -c 2:rootfsA \
-  -n 3:0:+2G      -t 3:8300 -c 3:rootfsB \
+  -n 2:0:+2560M   -t 2:8300 -c 2:rootfsA \
+  -n 3:0:+2560M   -t 3:8300 -c 3:rootfsB \
   -n 4:0:0        -t 4:8300 -c 4:data
 
 losetup -D

alpine/install-packages.sh

@@ -10,7 +10,8 @@ apk add alpine-base \
 # For diagnotics
 apk add \
   iproute2 iproute2-ss curl bind-tools procps strace tcpdump lsof jq binutils \
-  openssl conntrack-tools ethtool findmnt kmod coreutils util-linux zstd libcap-utils
+  openssl conntrack-tools ethtool findmnt kmod coreutils util-linux zstd libcap-utils \
+  iotop sysstat
 echo '[ -x /bin/bash ] && exec /bin/bash -l' >> "/root/.profile"
 
 # Compat layer for kubelet for now. Will look into building it myself later. If needed

alpine/migrations/k8s/1.35/10-drop-removed-kubelet-pause-flag.sh

@@ -0,0 +1,35 @@
+#!/bin/sh
+set -eu
+
+# Kubernetes removed the kubelet flag:
+#   --pod-infra-container-image
+#
+# Timeline:
+# - Deprecated before v1.27
+# - Removed in newer kubelet versions (>=1.27+)
+# - kubeadm may still write it into:
+#     /var/lib/kubelet/kubeadm-flags.env
+#
+# This causes kubelet to fail with:
+#   "unknown flag: --pod-infra-container-image"
+#
+# References:
+# - https://github.com/kubernetes/kubeadm/issues/3281
+# - https://github.com/kubernetes/kubernetes/pull/122739
+# - https://kubernetes.io/blog/2022/04/07/upcoming-changes-in-kubernetes-1-24/
+#
+# Root cause:
+# - Sandbox (pause) image is now managed by CRI (containerd/CRI-O),
+#   not kubelet flags.
+#
+# Fix:
+# - Strip the flag from kubeadm-flags.env during upgrade
+
+FILE=/var/lib/kubelet/kubeadm-flags.env
+
+[ -f "$FILE" ] || exit 0
+grep -q -- '--pod-infra-container-image=' "$FILE" || exit 0
+
+sed -i 's/ --pod-infra-container-image=[^"]*//g' "$FILE"
+
+echo "Removed deprecated kubelet flag --pod-infra-container-image from $FILE"

alpine/preload-k8s-images.sh

@@ -26,7 +26,7 @@ FUSE_OVERLAYFS="${FUSE_OVERLAYFS:-/usr/bin/fuse-overlayfs}"
 #   )
 EXTRA_IMAGES=(
 	"${EXTRA_IMAGES[@]:-}"
-	"docker-daemon:localhost/monok8s/control-agent:$TAG"
+	"docker-daemon:localhost/monok8s/node-control:$TAG"
 )
 
 # Keep archive cache version/arch scoped so downloads do not get mixed.

alpine/rootfs-extra/etc/containers/storage.conf

@@ -1,9 +1 @@
-[storage]
-driver = "overlay"
-runroot = "/run/containers/storage"
-graphroot = "/var/lib/containers/storage"
-
-[storage.options]
-additionalimagestores = [
-  "/usr/lib/monok8s/imagestore"
-]
+# Generated file. DO NOT MODIFY.

alpine/rootfs-extra/etc/local.d/monok8s.start

@@ -1,4 +1,41 @@
 #!/bin/sh
+set -eu
+
+exec >>/var/log/monok8s/boot.log 2>&1
 
 export PATH="/usr/local/bin:/usr/local/sbin:$PATH"
-/usr/local/bin/ctl init --env-file /opt/monok8s/config/cluster.env >>/var/log/monok8s/bootstrap.log 2>&1 &
+
+MIGRATIONS_LIB=/usr/lib/monok8s/lib/migrations.sh
+CONFIG_DIR=/opt/monok8s/config
+BOOT_STATE=/run/monok8s/boot-state.env
+BOOTPART_FILE="$CONFIG_DIR/.bootpart"
+MIGRATION_STATE_DIR="$CONFIG_DIR/migration-state"
+
+mkdir -p /dev/hugepages
+mountpoint -q /dev/hugepages || mount -t hugetlbfs none /dev/hugepages
+echo 256 > /proc/sys/vm/nr_hugepages
+
+CUR=$(grep '^BOOT_PART=' "$BOOT_STATE" | cut -d= -f2-)
+LAST=$(cat "$BOOTPART_FILE" 2>/dev/null || true)
+
+slot_changed=0
+if [ "$CUR" != "$LAST" ]; then
+	slot_changed=1
+	echo "Slot changed ($LAST -> $CUR)"
+fi
+
+# shellcheck source=/dev/null
+. "$MIGRATIONS_LIB"
+
+if [ "$slot_changed" -eq 1 ]; then
+	monok8s_cleanup_runtime_state
+fi
+
+K8S_MINOR="$(monok8s_detect_k8s_minor || true)"
+if [ -n "$K8S_MINOR" ]; then
+	monok8s_run_migration_dir \
+	"/usr/lib/monok8s/migrations.d/k8s/$K8S_MINOR" \
+	"$MIGRATION_STATE_DIR/k8s/$K8S_MINOR"
+fi
+
+/usr/lib/monok8s/lib/supervised-init.sh &

alpine/rootfs-extra/usr/lib/monok8s/lib/migrations.sh

@@ -0,0 +1,58 @@
+#!/bin/sh
+
+monok8s_detect_k8s_minor() {
+	local env_file=${1:-}
+	local version major_minor
+
+	version=$(/usr/local/bin/ctl version -k 2>/dev/null || true)
+
+	[ -n "$version" ] || return 1
+
+	version=${version#v}
+	major_minor=$(printf '%s\n' "$version" | cut -d. -f1,2)
+
+	[ -n "$major_minor" ] || return 1
+	printf '%s\n' "$major_minor"
+}
+
+monok8s_cleanup_runtime_state() {
+	echo "Cleaning runtime state"
+
+	rm -rf \
+		/var/lib/containers \
+		/var/lib/cni \
+		/var/lib/kubelet/pods \
+		/var/lib/kubelet/plugins \
+		/var/lib/kubelet/plugins_registry \
+		/var/lib/kubelet/device-plugins \
+		/run/containers \
+		/run/netns
+
+	mkdir -p \
+		/var/lib/containers \
+		/var/lib/kubelet \
+		/var/lib/cni
+}
+
+monok8s_run_migration_dir() {
+	dir=$1
+	state_dir=$2
+
+	[ -d "$dir" ] || return 0
+	mkdir -p "$state_dir"
+
+	for script in "$dir"/*.sh; do
+		[ -e "$script" ] || continue
+
+		name=$(basename "$script")
+		stamp="$state_dir/$name.done"
+
+		if [ -e "$stamp" ]; then
+			continue
+		fi
+
+		echo "Running migration: $script"
+		sh "$script"
+		: > "$stamp"
+	done
+}

alpine/rootfs-extra/usr/lib/monok8s/lib/supervised-init.sh

@@ -0,0 +1,57 @@
+#!/bin/sh
+set -eu
+
+CONFIG_DIR=/opt/monok8s/config
+LOG=/var/log/monok8s/bootstrap.log
+STATE_DIR=/run/monok8s
+FAIL_COUNT_FILE="$STATE_DIR/bootstrap-fail-count"
+LOCK_DIR="$STATE_DIR/supervised-init.lock"
+
+# For debugging
+HOLD_FILE="$CONFIG_DIR/bootstrap.hold"
+
+mkdir -p "$STATE_DIR" /var/log/monok8s
+
+if ! mkdir "$LOCK_DIR" 2>/dev/null; then
+	echo "[$(date -Is)] supervised-init already running" >> "$LOG"
+	exit 0
+fi
+
+trap 'rmdir "$LOCK_DIR"' EXIT INT TERM
+
+fail_count=0
+if [ -f "$FAIL_COUNT_FILE" ]; then
+	fail_count="$(cat "$FAIL_COUNT_FILE" 2>/dev/null || echo 0)"
+	case "$fail_count" in
+		''|*[!0-9]*) fail_count=0 ;;
+	esac
+fi
+
+while true; do
+	if [ -f "$HOLD_FILE" ]; then
+		echo "[$(date -Is)] bootstrap held by $HOLD_FILE" >> "$LOG"
+		sleep 300
+		continue
+	fi
+
+	echo "[$(date -Is)] starting ctl init" >> "$LOG"
+
+	if /usr/local/bin/ctl init --env-file "$CONFIG_DIR/cluster.env" >> "$LOG" 2>&1; then
+		echo "[$(date -Is)] ctl init succeeded" >> "$LOG"
+		rm -f "$FAIL_COUNT_FILE"
+		exit 0
+	fi
+
+	fail_count=$((fail_count + 1))
+	echo "$fail_count" > "$FAIL_COUNT_FILE"
+
+	echo "[$(date -Is)] ctl init failed, count=$fail_count" >> "$LOG"
+
+	case "$fail_count" in
+		1) sleep 10 ;;
+		2) sleep 30 ;;
+		3) sleep 60 ;;
+		4) sleep 120 ;;
+		*) sleep 300 ;;
+	esac
+done

alpine/utils.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+
+ensure_loop_ready() {
+    # The loop kernel module is host-side. This only works if the container
+    # has permission and modprobe exists; otherwise the host must load it.
+    if ! grep -qw loop /proc/modules 2>/dev/null; then
+        modprobe loop 2>/dev/null || true
+    fi
+
+    # /dev/loop-control: char device 10:237
+    if [ ! -e /dev/loop-control ]; then
+        echo "Creating missing /dev/loop-control" >&2
+        mknod /dev/loop-control c 10 237 || {
+            echo "ERROR: cannot create /dev/loop-control" >&2
+            echo "Run container with --privileged, or pass --device=/dev/loop-control and loop devices." >&2
+            exit 1
+        }
+        chmod 600 /dev/loop-control || true
+    fi
+
+    if [ ! -c /dev/loop-control ]; then
+        echo "ERROR: /dev/loop-control exists but is not a character device" >&2
+        ls -l /dev/loop-control >&2 || true
+        exit 1
+    fi
+
+    # Create a reasonable pool of loop block devices.
+    # loopN block devices are major 7, minor N.
+    for i in $(seq 0 31); do
+        if [ ! -e "/dev/loop$i" ]; then
+            echo "Creating missing /dev/loop$i" >&2
+            mknod "/dev/loop$i" b 7 "$i" || {
+                echo "ERROR: cannot create /dev/loop$i" >&2
+                echo "Run container with --privileged, or pre-create/pass loop devices." >&2
+                exit 1
+            }
+            chmod 660 "/dev/loop$i" || true
+        fi
+
+        if [ ! -b "/dev/loop$i" ]; then
+            echo "ERROR: /dev/loop$i exists but is not a block device" >&2
+            ls -l "/dev/loop$i" >&2 || true
+            exit 1
+        fi
+    done
+
+    # Smoke test: ask losetup for a free loop device.
+    if ! losetup -f >/dev/null 2>&1; then
+        echo "ERROR: losetup cannot find/use a loop device" >&2
+        echo "Debug info:" >&2
+        ls -l /dev/loop-control /dev/loop* >&2 || true
+        grep -w loop /proc/modules >&2 || true
+        echo >&2
+        echo "Docker likely needs --privileged, or at minimum CAP_SYS_ADMIN plus loop devices." >&2
+        exit 1
+    fi
+}

build.env

@@ -3,16 +3,33 @@ DOCKER_IMAGE_ROOT=monok8s
 # Image tag
 TAG=dev
 
-# The Linux kernel, from NXP
-NXP_VERSION=lf-6.18.2-1.0.0
+# NXP's Linux Factory
+LINUX_FACTORY=6.12.49-2.2.0
+NXP_VERSION=lf-$(LINUX_FACTORY)
+FMLIB_VERSION=lf-$(LINUX_FACTORY)
+FMC_VERSION=lf-$(LINUX_FACTORY)
+DPDK_VERSION=lf-$(LINUX_FACTORY)
+VPP_VERSION=lf-$(LINUX_FACTORY)
+VPP_UPSTREAM_VERSION=23.10
+
+# ASK's deps
+MONO_ASK_VERSION=mt-$(LINUX_FACTORY)
+LIBNFNETLINK_VERSION=1.0.2
+LIBMNL_VERSION=1.0.5
+LIBNFCT_VERSION=1.1.0
+LIBCLI_VERSION=1.10.7
+# Check the package version for Debian trixies (what ASK uses)
+LIBXML2_VERSION=2.11.7
+TCLAP_VERSION=1.2.5
+LIBPCAP_VERSION=1.10.4
+
 CRIO_VERSION=cri-o.arm64.v1.35.2
-KUBE_VERSION=v1.34.1
+KUBE_VERSION=v1.35.3
 
 # Mono's tutorial said fsl-ls1046a-rdb.dtb but our shipped board is not that one
 # We need fsl-ls1046a-rdb-sdk.dtb here
 DEVICE_TREE_TARGET=mono-gateway-dk-sdk
 
-
 # Arch, should always be arm64 for our board. This is here in case branching off to other devices
 ARCH=arm64
 
@@ -32,3 +49,11 @@ ALPINE_HOSTNAME=monok8s-hostname
 
 # Upper case [A-Z]_ only, used for naming env vars
 BUILD_TAG=MONOK8S
+
+# Optional apt cache
+#   example: apt-cacher-ng.eco-system.svc.cluster.local:3142
+APT_PROXY=
+
+# remote image repository prefix to push to
+# e.g. ghcr.io/monok8s
+IMAGE_REPOSITORY=

clitools/README.md

@@ -1,30 +1,28 @@
-## For development workflow
-
-For running `controll-gen`
+Installing `controller-gen`
 ```
 export PATH="$(go env GOPATH)/bin:$PATH"
 go install sigs.k8s.io/controller-tools/cmd/controller-gen@latest
 ```
 
-Run this on device
+## For development workflow
+
+Run this on the gateway device
 ```bash
 while true; do nc -l -p 1234 -e sh; done
 ```
 
-Run this script on the dev machine
+Run this script on your dev machine
 ```bash
-#!/bin/bash
 make build
-
 SIZE=$(wc -c < ./bin/ctl-linux-aarch64-dev)
-
 (
   echo 'base64 -d > /var/ctl <<'"'"'EOF'"'"''
   pv -s "$SIZE" < ./bin/ctl-linux-aarch64-dev | base64
   echo 'EOF'
+  echo "export DEBUG=1"
   echo 'chmod +x /var/ctl'
-  echo '/var/ctl create config > /var/abc.yaml'
-  echo "/var/ctl internal run-step $1 -c /var/abc.yaml 2>&1"
+  echo "echo Running /var/ctl $@"
+  echo "/var/ctl $@ 2>&1"
 ) | nc 10.0.0.10 1234
 ```
 
@@ -33,21 +31,14 @@ And use it like this
 ./send.sh start_crio
 ```
 
+### Join token
 
 Create join token from control plane
 ```
 kubeadm token create --print-join-command
 ```
 
-Export the token inside the device
-```
-export DEBUG=1
-export HOSTNAME=monok8s-master-1
-export BOOTSTRAP_TOKEN=
-export TOKEN_CACERT_HASH=
-```
-
-Generate using kubectl
+Generate join token using kubectl
 ```
 TOKEN_NAME=bootstrap-token-iwotl0
 
@@ -61,6 +52,7 @@ HASH=$(kubectl -n kube-public get configmap cluster-info -o jsonpath='{.data.kub
 | openssl rsa -pubin -outform der 2>/dev/null \
 | openssl dgst -sha256 -hex \
 | awk '{print "sha256:" $2}')
+
 echo "export API_SERVER_ENDPOINT=${API_SERVER}"
 echo "export BOOTSTRAP_TOKEN=${TOKEN}"
 echo "export TOKEN_CACERT_HASH=${HASH}"

clitools/devtools/run.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+SCRIPT_DIR="$(dirname "${BASH_SOURCE[0]}")"
+PROJ_ROOT="$( realpath "$SCRIPT_DIR"/../ )"
+OUT_DIR="$PROJ_ROOT"/out
+
+if [ "$1" == "controller" ]; then
+	if [ -f "$OUT_DIR/tls.key" ] && [ -f "$OUT_DIR/tls.crt" ]; then
+		echo "Use existing certs"
+	else
+		echo "Generating self signed certs"
+		openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
+		  -keyout "$OUT_DIR"/tls.key -out "$OUT_DIR"/tls.crt \
+		  -subj "/CN=127.0.0.1" \
+		  -addext "subjectAltName=IP:127.0.0.1,DNS:localhost"
+	fi
+	go run "$PROJ_ROOT"/cmd/ctl $@ --tls-cert-file "$OUT_DIR"/tls.crt --tls-private-key-file "$OUT_DIR"/tls.key
+else
+	go run "$PROJ_ROOT"/cmd/ctl $@
+fi

clitools/docker/crdgen.Dockerfile

@@ -0,0 +1,14 @@
+ARG BASE_IMAGE=localhost/monok8s/ctl-build-base:dev
+FROM ${BASE_IMAGE} AS build
+
+WORKDIR /src
+
+RUN GOBIN=/usr/local/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.20.1
+
+COPY . .
+
+RUN mkdir -p /out && \
+    controller-gen crd paths=./pkg/apis/... output:crd:dir=/out
+
+FROM scratch
+COPY --from=build /out/ /

clitools/docker/ctl-agent.Dockerfile

@@ -1,16 +1,41 @@
+ARG BASE_IMAGE=localhost/monok8s/ctl-build-base:dev
+
+FROM --platform=$BUILDPLATFORM ${BASE_IMAGE} AS build
+
+ARG VERSION=dev
+ARG TARGETOS
+ARG TARGETARCH
+
+WORKDIR /src
+
+COPY . .
+
+RUN test -f pkg/buildinfo/buildinfo_gen.go
+
+RUN mkdir -p /out && \
+    GOOS=${TARGETOS} GOARCH=${TARGETARCH} CGO_ENABLED=0 \
+    go build -trimpath -ldflags="-s -w" \
+      -o /out/ctl ./cmd/ctl
+
+
 FROM alpine:latest AS cacerts
+
+
 FROM scratch
 
 ARG VERSION
+ARG TARGETOS
+ARG TARGETARCH
 ENV VERSION=${VERSION}
 
 WORKDIR /
 
-COPY bin/ctl-linux-aarch64-${VERSION} ./ctl
-COPY out/fw_printenv ./
-COPY out/fw_setenv ./
+COPY --from=build /out/ctl /ctl
 COPY --from=cacerts /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
 
+COPY out/uboot-tools/${TARGETOS}_${TARGETARCH}/fw_printenv /fw_printenv
+COPY out/uboot-tools/${TARGETOS}_${TARGETARCH}/fw_setenv /fw_setenv
+
 ENV PATH=/
 
 ENTRYPOINT ["/ctl"]

clitools/docker/ctl-build-base.Dockerfile

@@ -0,0 +1,8 @@
+FROM golang:1.26-alpine
+
+WORKDIR /src
+
+RUN apk add --no-cache git build-base
+
+COPY go.mod go.sum ./
+RUN go mod download

clitools/docker/download-packages.Dockerfile

@@ -0,0 +1,26 @@
+FROM alpine:3.23.0 AS base
+
+# We seperate the packages so this line can be cached upstream
+RUN apk add --no-cache curl ca-certificates
+RUN apk add --no-cache git
+
+# ---- uboot ----
+FROM base AS uboot
+ARG UBOOT_VERSION
+
+WORKDIR /work
+
+RUN apk add --no-cache git tar gzip
+
+RUN git clone \
+      --depth 1 \
+      --branch "${UBOOT_VERSION}" \
+      --filter=blob:none \
+      https://github.com/u-boot/u-boot.git src
+
+RUN mkdir -p /out && \
+    tar -C /work/src -zcf "/out/uboot-${UBOOT_VERSION}.tar.gz" .
+
+# ---- final exported artifact set ----
+FROM scratch
+COPY --from=uboot /out/ /

clitools/docker/dpdk.Dockerfile

@@ -0,0 +1,51 @@
+ARG TAG=dev
+ARG DOCKER_IMAGE_ROOT=monok8s
+
+FROM alpine:3.22 AS build
+
+RUN apk add --no-cache \
+  bash \
+  build-base \
+  linux-headers \
+  meson \
+  ninja \
+  pkgconf \
+  python3 \
+  py3-elftools \
+  coreutils \
+  file \
+  git \
+  bsd-compat-headers
+
+RUN mkdir /src
+
+WORKDIR /src
+
+ARG DPDK_TAR
+ARG DPDK_VERSION
+
+COPY ${DPDK_TAR} /tmp/
+
+RUN set -eux; \
+    mkdir -p /src/dpdk; \
+    tar -xf "/tmp/$(basename "${DPDK_TAR}")" -C /src/dpdk --strip-components=1
+
+RUN set -eux; \
+    meson setup /src/dpdk/build /src/dpdk \
+      --buildtype=release \
+      -Dplatform=dpaa \
+      -Dtests=false \
+      -Ddisable_drivers=crypto/*,compress/*,baseband/*,dma/*,event/*,regex/*,ml/*,gpu/*,raw/*,net/pcap,net/tap,net/vhost,net/virtio,net/ixgbe,net/i40e,net/txgbe,net/ring,net/af_packet; \
+    meson configure /src/dpdk/build | tee /tmp/meson-config.txt; \
+    grep -Ei 'dpaa|platform|disable_drivers' /tmp/meson-config.txt || true; \
+    ninja -C /src/dpdk/build; \
+    DESTDIR=/out ninja -C /src/dpdk/build install
+
+RUN set -eux; \
+    mkdir -p /artifact/bin; \
+    test -x /src/dpdk/build/app/dpdk-testpmd; \
+    cp -r /src/dpdk/build/app/dpdk-testpmd /artifact/bin/
+
+FROM scratch AS export
+COPY --from=build /out/ /
+COPY --from=build /artifact/ /

clitools/docker/uboot-tools.Dockerfile

@@ -1,4 +1,4 @@
-FROM alpine:3.22 AS build
+FROM alpine:3.23 AS build
 
 RUN apk add --no-cache \
     build-base \

clitools/go.mod

@@ -1,33 +1,39 @@
 module example.com/monok8s
 
-go 1.24.0
+go 1.26.0
 
 require (
+	github.com/emicklei/go-restful/v3 v3.12.2
 	github.com/klauspost/compress v1.18.5
-	github.com/spf13/cobra v1.9.1
-	golang.org/x/sys v0.31.0
+	github.com/spf13/cobra v1.10.2
+	golang.org/x/sys v0.39.0
 	gopkg.in/yaml.v3 v3.0.1
-	k8s.io/api v0.34.0
-	k8s.io/apiextensions-apiserver v0.34.0
-	k8s.io/apimachinery v0.34.0
+	k8s.io/api v0.35.0
+	k8s.io/apiextensions-apiserver v0.35.0
+	k8s.io/apimachinery v0.35.0
+	k8s.io/apiserver v0.35.0
 	k8s.io/cli-runtime v0.34.0
-	k8s.io/client-go v0.34.0
+	k8s.io/client-go v0.35.0
+	k8s.io/code-generator v0.35.0
 	k8s.io/klog/v2 v2.130.1
+	sigs.k8s.io/controller-tools v0.20.1
 	sigs.k8s.io/yaml v1.6.0
 )
 
 require (
 	github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect
+	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/blang/semver/v4 v4.0.0 // indirect
+	github.com/cespare/xxhash/v2 v2.3.0 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
-	github.com/emicklei/go-restful/v3 v3.12.2 // indirect
+	github.com/fatih/color v1.18.0 // indirect
 	github.com/fxamacker/cbor/v2 v2.9.0 // indirect
 	github.com/go-errors/errors v1.4.2 // indirect
-	github.com/go-logr/logr v1.4.2 // indirect
+	github.com/go-logr/logr v1.4.3 // indirect
 	github.com/go-openapi/jsonpointer v0.21.0 // indirect
 	github.com/go-openapi/jsonreference v0.20.2 // indirect
 	github.com/go-openapi/swag v0.23.0 // indirect
-	github.com/gogo/protobuf v1.3.2 // indirect
+	github.com/gobuffalo/flect v1.0.3 // indirect
 	github.com/google/btree v1.1.3 // indirect
 	github.com/google/gnostic-models v0.7.0 // indirect
 	github.com/google/uuid v1.6.0 // indirect
@@ -37,30 +43,42 @@ require (
 	github.com/json-iterator/go v1.1.12 // indirect
 	github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect
 	github.com/mailru/easyjson v0.7.7 // indirect
+	github.com/mattn/go-colorable v0.1.13 // indirect
+	github.com/mattn/go-isatty v0.0.20 // indirect
 	github.com/moby/term v0.5.0 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect
 	github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
 	github.com/peterbourgon/diskv v2.0.1+incompatible // indirect
-	github.com/pkg/errors v0.9.1 // indirect
-	github.com/spf13/pflag v1.0.6 // indirect
+	github.com/prometheus/client_golang v1.23.2 // indirect
+	github.com/prometheus/client_model v0.6.2 // indirect
+	github.com/prometheus/common v0.66.1 // indirect
+	github.com/prometheus/procfs v0.16.1 // indirect
+	github.com/spf13/pflag v1.0.10 // indirect
 	github.com/x448/float16 v0.8.4 // indirect
 	github.com/xlab/treeprint v1.2.0 // indirect
-	go.yaml.in/yaml/v2 v2.4.2 // indirect
+	go.opentelemetry.io/otel v1.36.0 // indirect
+	go.opentelemetry.io/otel/trace v1.36.0 // indirect
+	go.yaml.in/yaml/v2 v2.4.3 // indirect
 	go.yaml.in/yaml/v3 v3.0.4 // indirect
-	golang.org/x/net v0.38.0 // indirect
-	golang.org/x/oauth2 v0.27.0 // indirect
-	golang.org/x/sync v0.12.0 // indirect
-	golang.org/x/term v0.30.0 // indirect
-	golang.org/x/text v0.23.0 // indirect
+	golang.org/x/mod v0.31.0 // indirect
+	golang.org/x/net v0.48.0 // indirect
+	golang.org/x/oauth2 v0.30.0 // indirect
+	golang.org/x/sync v0.19.0 // indirect
+	golang.org/x/term v0.38.0 // indirect
+	golang.org/x/text v0.32.0 // indirect
 	golang.org/x/time v0.9.0 // indirect
-	google.golang.org/protobuf v1.36.5 // indirect
-	gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
+	golang.org/x/tools v0.40.0 // indirect
+	google.golang.org/protobuf v1.36.8 // indirect
+	gopkg.in/evanphx/json-patch.v4 v4.13.0 // indirect
 	gopkg.in/inf.v0 v0.9.1 // indirect
-	k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b // indirect
-	k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 // indirect
-	sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 // indirect
+	gopkg.in/yaml.v2 v2.4.0 // indirect
+	k8s.io/component-base v0.35.0 // indirect
+	k8s.io/gengo/v2 v2.0.0-20250922181213-ec3ebc5fd46b // indirect
+	k8s.io/kube-openapi v0.0.0-20250910181357-589584f1c912 // indirect
+	k8s.io/utils v0.0.0-20251002143259-bc988d571ff4 // indirect
+	sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 // indirect
 	sigs.k8s.io/kustomize/api v0.20.1 // indirect
 	sigs.k8s.io/kustomize/kyaml v0.20.1 // indirect
 	sigs.k8s.io/randfill v1.0.0 // indirect

clitools/go.sum

@@ -1,7 +1,19 @@
+cel.dev/expr v0.24.0 h1:56OvJKSH3hDGL0ml5uSxZmz3/3Pq4tJ+fb1unVLAFcY=
+cel.dev/expr v0.24.0/go.mod h1:hLPLo1W4QUmuYdA72RBX06QTs6MXw941piREPl3Yfiw=
 github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 h1:L/gRVlceqvL25UVaW/CKtUDjefjrs0SPonmDGUVOYP0=
 github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
+github.com/Masterminds/semver/v3 v3.4.0 h1:Zog+i5UMtVoCU8oKka5P7i9q9HgrJeGzI9SA1Xbatp0=
+github.com/Masterminds/semver/v3 v3.4.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM=
+github.com/antlr4-go/antlr/v4 v4.13.0 h1:lxCg3LAv+EUK6t1i0y1V6/SLeUi0eKEKdhQAlS8TVTI=
+github.com/antlr4-go/antlr/v4 v4.13.0/go.mod h1:pfChB/xh/Unjila75QW7+VU4TSnWnnk9UTnmpPaOR2g=
+github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
+github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
 github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM=
 github.com/blang/semver/v4 v4.0.0/go.mod h1:IbckMUScFkM3pff0VJDNKRiT6TG/YpiHIM2yvyW5YoQ=
+github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
+github.com/cenkalti/backoff/v4 v4.3.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
+github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
+github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
 github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
 github.com/creack/pty v1.1.18 h1:n56/Zwd5o6whRC5PMGretI4IdRLlmBXYNjScPaBgsbY=
@@ -11,12 +23,20 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/emicklei/go-restful/v3 v3.12.2 h1:DhwDP0vY3k8ZzE0RunuJy8GhNpPL6zqLkDf9B/a0/xU=
 github.com/emicklei/go-restful/v3 v3.12.2/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
+github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM=
+github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU=
+github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
+github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
+github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
+github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
 github.com/fxamacker/cbor/v2 v2.9.0 h1:NpKPmjDBgUfBms6tr6JZkTHtfFGcMKsw3eGcmD/sapM=
 github.com/fxamacker/cbor/v2 v2.9.0/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ=
 github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA=
 github.com/go-errors/errors v1.4.2/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og=
-github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
-github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
+github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
+github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
+github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
+github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
 github.com/go-openapi/jsonpointer v0.19.6/go.mod h1:osyAmYz/mB/C3I+WsTTSgw1ONzaLJoLCyoi6/zppojs=
 github.com/go-openapi/jsonpointer v0.21.0 h1:YgdVicSA9vH5RiHs9TZW5oyafXZFc6+2Vc1rr/O9oNQ=
 github.com/go-openapi/jsonpointer v0.21.0/go.mod h1:IUyH9l/+uyhIYQ/PXVA41Rexl+kOkAPDdXEYns6fzUY=
@@ -27,29 +47,31 @@ github.com/go-openapi/swag v0.23.0 h1:vsEVJDUo2hPJ2tu0/Xc+4noaxyEffXNIs3cOULZ+Gr
 github.com/go-openapi/swag v0.23.0/go.mod h1:esZ8ITTYEsH1V2trKHjAN8Ai7xHb8RV+YSZ577vPjgQ=
 github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI=
 github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8=
-github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
-github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
+github.com/gobuffalo/flect v1.0.3 h1:xeWBM2nui+qnVvNM4S3foBhCAL2XgPU+a7FdpelbTq4=
+github.com/gobuffalo/flect v1.0.3/go.mod h1:A5msMlrHtLqh9umBSnvabjsMrCcCpAyzglnDvkbYKHs=
 github.com/google/btree v1.1.3 h1:CVpQJjYgC4VbzxeGVHfvZrv1ctoYCAI8vbl07Fcxlyg=
 github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4=
+github.com/google/cel-go v0.26.0 h1:DPGjXackMpJWH680oGY4lZhYjIameYmR+/6RBdDGmaI=
+github.com/google/cel-go v0.26.0/go.mod h1:A9O8OU9rdvrK5MQyrqfIxo1a0u4g3sF8KB6PUIaryMM=
 github.com/google/gnostic-models v0.7.0 h1:qwTtogB15McXDaNqTZdzPJRHvaVJlAl+HVQnLmJEJxo=
 github.com/google/gnostic-models v0.7.0/go.mod h1:whL5G0m6dmc5cPxKc5bdKdEN3UjI7OUGxBlw57miDrQ=
 github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
 github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
 github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
-github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db h1:097atOisP2aRj7vFgYQBbFN4U4JNXUNYpxael3UzMyo=
-github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db/go.mod h1:vavhavw2zAxS5dIdcRluK6cSGGPlZynqzFM8NdvU144=
+github.com/google/pprof v0.0.0-20250403155104-27863c87afa6 h1:BHT72Gu3keYf3ZEu2J0b1vyeLSOYI8bm5wbJM/8yDe8=
+github.com/google/pprof v0.0.0-20250403155104-27863c87afa6/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA=
 github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
 github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79 h1:+ngKgrYPPJrOjhax5N+uePQ0Fh1Z7PheYoUI/0nzkPA=
 github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.3 h1:5ZPtiqj0JL5oKWmcsq4VMaAW5ukBEgSGXEN89zeH1Jo=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.3/go.mod h1:ndYquD05frm2vACXE1nsccT4oJzjhw2arTS2cpUD1PI=
 github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
 github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
 github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
 github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
 github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
 github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
-github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
-github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
 github.com/klauspost/compress v1.18.5 h1:/h1gH5Ce+VWNLSWqPzOVn6XBO+vJbCNGvjoaGBFW2IE=
 github.com/klauspost/compress v1.18.5/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
 github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
@@ -59,10 +81,17 @@ github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
 github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
 github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
+github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
+github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
 github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de h1:9TO3cAIGXtEhnIaL+V+BEER86oLrvS+kWobKpbJuye0=
 github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de/go.mod h1:zAbeS9B/r2mtpb6U+EI2rYA5OAXxsYw6wTamcNW+zcE=
 github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0=
 github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
+github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
+github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
+github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
+github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
+github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
 github.com/moby/term v0.5.0 h1:xt8Q1nalod/v7BqbG21f8mQPqH+xAaC9C3N3wfWbVP0=
 github.com/moby/term v0.5.0/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y=
 github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
@@ -75,25 +104,38 @@ github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 h1:n6/
 github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00/go.mod h1:Pm3mSP3c5uWn86xMLZ5Sa7JB9GsEZySvHYXCTK4E9q4=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
-github.com/onsi/ginkgo/v2 v2.21.0 h1:7rg/4f3rB88pb5obDgNZrNHrQ4e6WpjonchcpuBRnZM=
-github.com/onsi/ginkgo/v2 v2.21.0/go.mod h1:7Du3c42kxCUegi0IImZ1wUQzMBVecgIHjR1C+NkhLQo=
-github.com/onsi/gomega v1.35.1 h1:Cwbd75ZBPxFSuZ6T+rN/WCb/gOc6YgFBXLlZLhC7Ds4=
-github.com/onsi/gomega v1.35.1/go.mod h1:PvZbdDc8J6XJEpDK4HCuRBm8a6Fzp9/DmhC9C7yFlog=
+github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE=
+github.com/nxadm/tail v1.4.8/go.mod h1:+ncqLTQzXmGhMZNUePPaPqPvBxHAIsmXswZKocGu+AU=
+github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE=
+github.com/onsi/ginkgo v1.16.5/go.mod h1:+E8gABHa3K6zRBolWtd+ROzc/U5bkGt0FwiG042wbpU=
+github.com/onsi/ginkgo/v2 v2.27.2 h1:LzwLj0b89qtIy6SSASkzlNvX6WktqurSHwkk2ipF/Ns=
+github.com/onsi/ginkgo/v2 v2.27.2/go.mod h1:ArE1D/XhNXBXCBkKOLkbsb2c81dQHCRcF5zwn/ykDRo=
+github.com/onsi/gomega v1.38.3 h1:eTX+W6dobAYfFeGC2PV6RwXRu/MyT+cQguijutvkpSM=
+github.com/onsi/gomega v1.38.3/go.mod h1:ZCU1pkQcXDO5Sl9/VVEGlDyp+zm0m1cmeG5TOzLgdh4=
 github.com/peterbourgon/diskv v2.0.1+incompatible h1:UBdAOUP5p4RWqPBg048CAvpKN+vxiaj6gdUUzhl4XmI=
 github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
-github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
-github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
-github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII=
-github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o=
+github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o=
+github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg=
+github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
+github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
+github.com/prometheus/common v0.66.1 h1:h5E0h5/Y8niHc5DlaLlWLArTQI7tMrsfQjHV+d9ZoGs=
+github.com/prometheus/common v0.66.1/go.mod h1:gcaUsgf3KfRSwHY4dIMXLPV0K/Wg1oZ8+SbZk/HH/dA=
+github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg=
+github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is=
+github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
+github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
 github.com/sergi/go-diff v1.2.0 h1:XU+rvMAioB0UC3q1MFrIQy4Vo5/4VsRDQQXHsEya6xQ=
 github.com/sergi/go-diff v1.2.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM=
-github.com/spf13/cobra v1.9.1 h1:CXSaggrXdbHK9CF+8ywj8Amf7PBRmPCOJugH954Nnlo=
-github.com/spf13/cobra v1.9.1/go.mod h1:nDyEzZ8ogv936Cinf6g1RU9MRY64Ir93oCnqb9wxYW0=
-github.com/spf13/pflag v1.0.6 h1:jFzHGLGAlb3ruxLB8MhbI6A8+AQX/2eW4qeyNZXNp2o=
-github.com/spf13/pflag v1.0.6/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
+github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
+github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
+github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+github.com/stoewer/go-strcase v1.3.0 h1:g0eASXYtp+yvN9fK8sH94oCIk0fau9uV1/ZdJ0AVEzs=
+github.com/stoewer/go-strcase v1.3.0/go.mod h1:fAH5hQ5pehh+j3nZfvwdk2RgEgQjAoM8wodgtPmh1xo=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
 github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
@@ -104,92 +146,115 @@ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
 github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
 github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
-github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
-github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
+github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
+github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
 github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
 github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg=
 github.com/xlab/treeprint v1.2.0 h1:HzHnuAF1plUN2zGlAFHbSQP2qJ0ZAD3XF5XD7OesXRQ=
 github.com/xlab/treeprint v1.2.0/go.mod h1:gj5Gd3gPdKtR1ikdDK6fnFLdmIS0X30kTTuNd/WEJu0=
-github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
-github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
+go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=
+go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
+go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0 h1:F7Jx+6hwnZ41NSFTO5q4LYDtJRXBf2PD0rNBkeB/lus=
+go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0/go.mod h1:UHB22Z8QsdRDrnAtX4PntOl36ajSxcdUMt1sF7Y6E7Q=
+go.opentelemetry.io/otel v1.36.0 h1:UumtzIklRBY6cI/lllNZlALOF5nNIzJVb16APdvgTXg=
+go.opentelemetry.io/otel v1.36.0/go.mod h1:/TcFMXYjyRNh8khOAO9ybYkqaDBb/70aVwkNML4pP8E=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.34.0 h1:OeNbIYk/2C15ckl7glBlOBp5+WlYsOElzTNmiPW/x60=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.34.0/go.mod h1:7Bept48yIeqxP2OZ9/AqIpYS94h2or0aB4FypJTc8ZM=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.34.0 h1:tgJ0uaNS4c98WRNUEx5U3aDlrDOI5Rs+1Vifcw4DJ8U=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.34.0/go.mod h1:U7HYyW0zt/a9x5J1Kjs+r1f/d4ZHnYFclhYY2+YbeoE=
+go.opentelemetry.io/otel/metric v1.36.0 h1:MoWPKVhQvJ+eeXWHFBOPoBOi20jh6Iq2CcCREuTYufE=
+go.opentelemetry.io/otel/metric v1.36.0/go.mod h1:zC7Ks+yeyJt4xig9DEw9kuUFe5C3zLbVjV2PzT6qzbs=
+go.opentelemetry.io/otel/sdk v1.36.0 h1:b6SYIuLRs88ztox4EyrvRti80uXIFy+Sqzoh9kFULbs=
+go.opentelemetry.io/otel/sdk v1.36.0/go.mod h1:+lC+mTgD+MUWfjJubi2vvXWcVxyr9rmlshZni72pXeY=
+go.opentelemetry.io/otel/trace v1.36.0 h1:ahxWNuqZjpdiFAyrIoQ4GIiAIhxAunQR6MUoKrsNd4w=
+go.opentelemetry.io/otel/trace v1.36.0/go.mod h1:gQ+OnDZzrybY4k4seLzPAWNwVBBVlF2szhehOBB/tGA=
+go.opentelemetry.io/proto/otlp v1.5.0 h1:xJvq7gMzB31/d406fB8U5CBdyQGw4P399D1aQWU/3i4=
+go.opentelemetry.io/proto/otlp v1.5.0/go.mod h1:keN8WnHxOy8PG0rQZjJJ5A2ebUoafqWp0eVQ4yIXvJ4=
 go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
 go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
-go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI=
-go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU=
+go.yaml.in/yaml/v2 v2.4.3 h1:6gvOSjQoTB3vt1l+CU+tSyi/HOjfOjRLJ4YwYZGwRO0=
+go.yaml.in/yaml/v2 v2.4.3/go.mod h1:zSxWcmIDjOzPXpjlTTbAsKokqkDNAVtZO0WOMiT90s8=
 go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
 go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
-golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
-golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
-golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
-golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
-golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
-golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
-golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
-golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
-golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
-golang.org/x/net v0.38.0 h1:vRMAPTMaeGqVhG5QyLJHqNDwecKTomGeqbnfZyKlBI8=
-golang.org/x/net v0.38.0/go.mod h1:ivrbrMbzFq5J41QOQh0siUuly180yBYtLp+CKbEaFx8=
-golang.org/x/oauth2 v0.27.0 h1:da9Vo7/tDv5RH/7nZDz1eMGS/q1Vv1N/7FCrBhI9I3M=
-golang.org/x/oauth2 v0.27.0/go.mod h1:onh5ek6nERTohokkhCD/y2cV4Do3fxFHFuAejCkRWT8=
-golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.12.0 h1:MHc5BpPuC30uJk597Ri8TV3CNZcTLu6B6z4lJy+g6Jw=
-golang.org/x/sync v0.12.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
-golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
-golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 h1:2dVuKD2vS7b0QIHQbpyTISPd0LeHDbnYEryqj5Q1ug8=
+golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56/go.mod h1:M4RDyNAINzryxdtnbRXRL/OHtkFuWGRjvuhBJpk2IlY=
+golang.org/x/mod v0.31.0 h1:HaW9xtz0+kOcWKwli0ZXy79Ix+UW/vOfmWI5QVd2tgI=
+golang.org/x/mod v0.31.0/go.mod h1:43JraMp9cGx1Rx3AqioxrbrhNsLl2l/iNAvuBkrezpg=
+golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
+golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
+golang.org/x/oauth2 v0.30.0 h1:dnDm7JmhM45NNpd8FDDeLhK6FwqbOf4MLCM9zb1BOHI=
+golang.org/x/oauth2 v0.30.0/go.mod h1:B++QgG3ZKulg6sRPGD/mqlHQs5rB3Ml9erfeDY7xKlU=
+golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
+golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
 golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
-golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
-golang.org/x/term v0.30.0 h1:PQ39fJZ+mfadBm0y5WlL4vlM7Sx1Hgf13sMIY2+QS9Y=
-golang.org/x/term v0.30.0/go.mod h1:NYYFdzHoI5wRh/h5tDMdMqCqPJZEuNqVR5xJLd/n67g=
-golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
-golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
-golang.org/x/text v0.23.0 h1:D71I7dUrlY+VX0gQShAThNGHFxZ13dGLBHQLVl1mJlY=
-golang.org/x/text v0.23.0/go.mod h1:/BLNzu4aZCJ1+kcD0DNRotWKage4q2rGVAg4o22unh4=
+golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
+golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
+golang.org/x/term v0.38.0 h1:PQ5pkm/rLO6HnxFR7N2lJHOZX6Kez5Y1gDSJla6jo7Q=
+golang.org/x/term v0.38.0/go.mod h1:bSEAKrOT1W+VSu9TSCMtoGEOUcKxOKgl3LE5QEF/xVg=
+golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
+golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
 golang.org/x/time v0.9.0 h1:EsRrnYcQiGH+5FfbgvV4AP7qEZstoyrHB0DzarOQ4ZY=
 golang.org/x/time v0.9.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
-golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
-golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
-golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
-golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
-golang.org/x/tools v0.26.0 h1:v/60pFQmzmT9ExmjDv2gGIfi3OqfKoEP6I5+umXlbnQ=
-golang.org/x/tools v0.26.0/go.mod h1:TPVVj70c7JJ3WCazhD8OdXcZg/og+b9+tH/KxylGwH0=
-golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-google.golang.org/protobuf v1.36.5 h1:tPhr+woSbjfYvY6/GPufUoYizxw1cF/yFoxJ2fmpwlM=
-google.golang.org/protobuf v1.36.5/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
+golang.org/x/tools v0.40.0 h1:yLkxfA+Qnul4cs9QA3KnlFu0lVmd8JJfoq+E41uSutA=
+golang.org/x/tools v0.40.0/go.mod h1:Ik/tzLRlbscWpqqMRjyWYDisX8bG13FrdXp3o4Sr9lc=
+golang.org/x/tools/go/expect v0.1.1-deprecated h1:jpBZDwmgPhXsKZC6WhL20P4b/wmnpsEAGHaNy0n/rJM=
+golang.org/x/tools/go/expect v0.1.1-deprecated/go.mod h1:eihoPOH+FgIqa3FpoTwguz/bVUSGBlGQU67vpBeOrBY=
+golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated h1:1h2MnaIAIXISqTFKdENegdpAgUXz6NrPEsbIeWaBRvM=
+golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated/go.mod h1:RVAQXBGNv1ib0J382/DPCRS/BPnsGebyM1Gj5VSDpG8=
+google.golang.org/genproto/googleapis/api v0.0.0-20250303144028-a0af3efb3deb h1:p31xT4yrYrSM/G4Sn2+TNUkVhFCbG9y8itM2S6Th950=
+google.golang.org/genproto/googleapis/api v0.0.0-20250303144028-a0af3efb3deb/go.mod h1:jbe3Bkdp+Dh2IrslsFCklNhweNTBgSYanP1UXhJDhKg=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20250528174236-200df99c418a h1:v2PbRU4K3llS09c7zodFpNePeamkAwG3mPrAery9VeE=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20250528174236-200df99c418a/go.mod h1:qQ0YXyHHx3XkvlzUtpXDkS29lDSafHMZBAZDc03LQ3A=
+google.golang.org/grpc v1.72.2 h1:TdbGzwb82ty4OusHWepvFWGLgIbNo1/SUynEN0ssqv8=
+google.golang.org/grpc v1.72.2/go.mod h1:wH5Aktxcg25y1I3w7H69nHfXdOG3UiadoBtjh3izSDM=
+google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc=
+google.golang.org/protobuf v1.36.8/go.mod h1:fuxRtAxBytpl4zzqUh6/eyUujkJdNiuEkXntxiD/uRU=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
-gopkg.in/evanphx/json-patch.v4 v4.12.0 h1:n6jtcsulIzXPJaxegRbvFNNrZDjbij7ny3gmSPG+6V4=
-gopkg.in/evanphx/json-patch.v4 v4.12.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWMG4EsWvDvM72M=
+gopkg.in/evanphx/json-patch.v4 v4.13.0 h1:czT3CmqEaQ1aanPc5SdlgQrrEIb8w/wwCvWWnfEbYzo=
+gopkg.in/evanphx/json-patch.v4 v4.13.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWMG4EsWvDvM72M=
 gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
 gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
+gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
+gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
+gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
+gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
-k8s.io/api v0.34.0 h1:L+JtP2wDbEYPUeNGbeSa/5GwFtIA662EmT2YSLOkAVE=
-k8s.io/api v0.34.0/go.mod h1:YzgkIzOOlhl9uwWCZNqpw6RJy9L2FK4dlJeayUoydug=
-k8s.io/apiextensions-apiserver v0.34.0 h1:B3hiB32jV7BcyKcMU5fDaDxk882YrJ1KU+ZSkA9Qxoc=
-k8s.io/apiextensions-apiserver v0.34.0/go.mod h1:hLI4GxE1BDBy9adJKxUxCEHBGZtGfIg98Q+JmTD7+g0=
-k8s.io/apimachinery v0.34.0 h1:eR1WO5fo0HyoQZt1wdISpFDffnWOvFLOOeJ7MgIv4z0=
-k8s.io/apimachinery v0.34.0/go.mod h1:/GwIlEcWuTX9zKIg2mbw0LRFIsXwrfoVxn+ef0X13lw=
+k8s.io/api v0.35.0 h1:iBAU5LTyBI9vw3L5glmat1njFK34srdLmktWwLTprlY=
+k8s.io/api v0.35.0/go.mod h1:AQ0SNTzm4ZAczM03QH42c7l3bih1TbAXYo0DkF8ktnA=
+k8s.io/apiextensions-apiserver v0.35.0 h1:3xHk2rTOdWXXJM+RDQZJvdx0yEOgC0FgQ1PlJatA5T4=
+k8s.io/apiextensions-apiserver v0.35.0/go.mod h1:E1Ahk9SADaLQ4qtzYFkwUqusXTcaV2uw3l14aqpL2LU=
+k8s.io/apimachinery v0.35.0 h1:Z2L3IHvPVv/MJ7xRxHEtk6GoJElaAqDCCU0S6ncYok8=
+k8s.io/apimachinery v0.35.0/go.mod h1:jQCgFZFR1F4Ik7hvr2g84RTJSZegBc8yHgFWKn//hns=
+k8s.io/apiserver v0.35.0 h1:CUGo5o+7hW9GcAEF3x3usT3fX4f9r8xmgQeCBDaOgX4=
+k8s.io/apiserver v0.35.0/go.mod h1:QUy1U4+PrzbJaM3XGu2tQ7U9A4udRRo5cyxkFX0GEds=
 k8s.io/cli-runtime v0.34.0 h1:N2/rUlJg6TMEBgtQ3SDRJwa8XyKUizwjlOknT1mB2Cw=
 k8s.io/cli-runtime v0.34.0/go.mod h1:t/skRecS73Piv+J+FmWIQA2N2/rDjdYSQzEE67LUUs8=
-k8s.io/client-go v0.34.0 h1:YoWv5r7bsBfb0Hs2jh8SOvFbKzzxyNo0nSb0zC19KZo=
-k8s.io/client-go v0.34.0/go.mod h1:ozgMnEKXkRjeMvBZdV1AijMHLTh3pbACPvK7zFR+QQY=
+k8s.io/client-go v0.35.0 h1:IAW0ifFbfQQwQmga0UdoH0yvdqrbwMdq9vIFEhRpxBE=
+k8s.io/client-go v0.35.0/go.mod h1:q2E5AAyqcbeLGPdoRB+Nxe3KYTfPce1Dnu1myQdqz9o=
+k8s.io/code-generator v0.35.0 h1:TvrtfKYZTm9oDF2z+veFKSCcgZE3Igv0svY+ehCmjHQ=
+k8s.io/code-generator v0.35.0/go.mod h1:iS1gvVf3c/T71N5DOGYO+Gt3PdJ6B9LYSvIyQ4FHzgc=
+k8s.io/component-base v0.35.0 h1:+yBrOhzri2S1BVqyVSvcM3PtPyx5GUxCK2tinZz1G94=
+k8s.io/component-base v0.35.0/go.mod h1:85SCX4UCa6SCFt6p3IKAPej7jSnF3L8EbfSyMZayJR0=
+k8s.io/gengo/v2 v2.0.0-20250922181213-ec3ebc5fd46b h1:gMplByicHV/TJBizHd9aVEsTYoJBnnUAT5MHlTkbjhQ=
+k8s.io/gengo/v2 v2.0.0-20250922181213-ec3ebc5fd46b/go.mod h1:CgujABENc3KuTrcsdpGmrrASjtQsWCT7R99mEV4U/fM=
 k8s.io/klog/v2 v2.130.1 h1:n9Xl7H1Xvksem4KFG4PYbdQCQxqc/tTUyrgXaOhHSzk=
 k8s.io/klog/v2 v2.130.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE=
-k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b h1:MloQ9/bdJyIu9lb1PzujOPolHyvO06MXG5TUIj2mNAA=
-k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b/go.mod h1:UZ2yyWbFTpuhSbFhv24aGNOdoRdJZgsIObGBUaYVsts=
-k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 h1:hwvWFiBzdWw1FhfY1FooPn3kzWuJ8tmbZBHi4zVsl1Y=
-k8s.io/utils v0.0.0-20250604170112-4c0f3b243397/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
-sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 h1:gBQPwqORJ8d8/YNZWEjoZs7npUVDpVXUUOFfW6CgAqE=
-sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8/go.mod h1:mdzfpAEoE6DHQEN0uh9ZbOCuHbLK5wOm7dK4ctXE9Tg=
+k8s.io/kube-openapi v0.0.0-20250910181357-589584f1c912 h1:Y3gxNAuB0OBLImH611+UDZcmKS3g6CthxToOb37KgwE=
+k8s.io/kube-openapi v0.0.0-20250910181357-589584f1c912/go.mod h1:kdmbQkyfwUagLfXIad1y2TdrjPFWp2Q89B3qkRwf/pQ=
+k8s.io/utils v0.0.0-20251002143259-bc988d571ff4 h1:SjGebBtkBqHFOli+05xYbK8YF1Dzkbzn+gDM4X9T4Ck=
+k8s.io/utils v0.0.0-20251002143259-bc988d571ff4/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
+sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.31.2 h1:jpcvIRr3GLoUoEKRkHKSmGjxb6lWwrBlJsXc+eUYQHM=
+sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.31.2/go.mod h1:Ve9uj1L+deCXFrPOk1LpFXqTg7LCFzFso6PA48q/XZw=
+sigs.k8s.io/controller-tools v0.20.1 h1:gkfMt9YodI0K85oT8rVi80NTXO/kDmabKR5Ajn5GYxs=
+sigs.k8s.io/controller-tools v0.20.1/go.mod h1:b4qPmjGU3iZwqn34alUU5tILhNa9+VXK+J3QV0fT/uU=
+sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 h1:IpInykpT6ceI+QxKBbEflcR5EXP7sU1kvOlxwZh5txg=
+sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730/go.mod h1:mdzfpAEoE6DHQEN0uh9ZbOCuHbLK5wOm7dK4ctXE9Tg=
 sigs.k8s.io/kustomize/api v0.20.1 h1:iWP1Ydh3/lmldBnH/S5RXgT98vWYMaTUL1ADcr+Sv7I=
 sigs.k8s.io/kustomize/api v0.20.1/go.mod h1:t6hUFxO+Ph0VxIk1sKp1WS0dOjbPCtLJ4p8aADLwqjM=
 sigs.k8s.io/kustomize/kyaml v0.20.1 h1:PCMnA2mrVbRP3NIB6v9kYCAc38uvFLVs8j/CD567A78=

clitools/hack/boilerplate.go.txt

@@ -0,0 +1 @@
+/* MIT License */

clitools/hack/tool.go

@@ -0,0 +1,9 @@
+//go:build tools
+// +build tools
+
+package tools
+
+import (
+	_ "k8s.io/code-generator"
+	_ "sigs.k8s.io/controller-tools/cmd/controller-gen"
+)

clitools/hack/update-codegen.sh

@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+MODULE_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+echo "MODULE ROOT: ${MODULE_ROOT}"
+
+cd "${MODULE_ROOT}"
+mkdir -p \
+	"${MODULE_ROOT}/pkg/generated/clientset" \
+	"${MODULE_ROOT}/pkg/generated/listers" \
+	"${MODULE_ROOT}/pkg/generated/informers"
+
+controller-gen \
+  object:headerFile=hack/boilerplate.go.txt \
+  paths=./pkg/apis/...
+
+MODULE="$(go list -m -f '{{.Path}}')"
+echo "MODULE: ${MODULE}"
+
+CODEGEN_PKG="$(go list -f '{{.Dir}}' -m k8s.io/code-generator)"
+echo "CODEGEN PKG: ${CODEGEN_PKG}"
+
+source "${CODEGEN_PKG}/kube_codegen.sh"
+
+mkdir -p "${MODULE_ROOT}/pkg/generated"
+
+kube::codegen::gen_helpers \
+  --boilerplate "${MODULE_ROOT}/hack/boilerplate.go.txt" \
+  "${MODULE_ROOT}/pkg/apis"
+
+kube::codegen::gen_client \
+  --with-watch \
+  --output-dir "${MODULE_ROOT}/pkg/generated" \
+  --output-pkg "${MODULE}/pkg/generated" \
+  --boilerplate "${MODULE_ROOT}/hack/boilerplate.go.txt" \
+  "${MODULE_ROOT}/pkg/apis"

clitools/makefile

@@ -1,25 +1,45 @@
+include ../build.env
+-include ../build.env.work
+export
+
+BUILD_PLATFORM ?= linux/amd64
+
 # Should be the same as upstream version in production
 VERSION ?= dev
-UBOOT_VERSION=v2026.01
+UBOOT_VERSION ?= v2026.01
 
 # Target kube version
-KUBE_VERSION ?= v1.35.0
+KUBE_VERSION ?= v1.33.3
 
 GIT_REV := $(shell git rev-parse HEAD)
 
-PACKAGES_DIR := packages
-BIN_DIR      := bin
-OUT_DIR      := out
-
-UBOOT_TAR     := $(PACKAGES_DIR)/uboot-$(UBOOT_VERSION).tar.gz
+PACKAGES_DIR    := packages
+OUT_DIR         := out
+UBOOT_TOOLS_OUT := $(OUT_DIR)/uboot-tools
 
+UBOOT_TAR      := $(PACKAGES_DIR)/uboot-$(UBOOT_VERSION).tar.gz
 BUILDINFO_FILE := pkg/buildinfo/buildinfo_gen.go
-CRD_PATHS := ./pkg/apis/...
+ASSETS_PATH      := ./pkg/assets
+
+BUILDX_BUILDER      := container-builder
+LOCAL_REGISTRY      := registry
+LOCAL_REGISTRY_PORT := 5000
+
+CTL_BUILD_BASE_REPO  := localhost:5000/monok8s/ctl-build-base
+CTL_IMAGE_REPO       := localhost:5000/monok8s/node-control
+
+CTL_BUILD_BASE_IMAGE := $(CTL_BUILD_BASE_REPO):$(VERSION)
+CTL_IMAGE            := $(CTL_IMAGE_REPO):$(VERSION)
+
+DOWNLOAD_PACKAGES_STAMP := $(PACKAGES_DIR)/.download-packages.stamp
 
 $(PACKAGES_DIR):
 	mkdir -p $@
 
-# Never cache this
+$(OUT_DIR):
+	mkdir -p $@
+
+# Keep buildinfo host-side since it is just generated source and lets Docker see it in build context.
 .buildinfo:
 	@mkdir -p $(dir $(BUILDINFO_FILE))
 	@printf '%s\n' \
@@ -33,44 +53,131 @@ $(PACKAGES_DIR):
 		')' \
 	> $(BUILDINFO_FILE)
 
-$(UBOOT_TAR): | $(PACKAGES_DIR)
-	git clone --depth 1 --branch v2026.01 --filter=blob:none https://github.com/u-boot/u-boot.git $(OUT_DIR)/u-boot-$(UBOOT_VERSION)
-	tar -C "$(OUT_DIR)/u-boot-$(UBOOT_VERSION)" -zcf "$@" .
-	rm -rf $(OUT_DIR)/u-boot-$(UBOOT_VERSION)
-	test -f $@
+ensure-buildx:
+	@if ! docker buildx inspect $(BUILDX_BUILDER) >/dev/null 2>&1; then \
+		echo "Creating buildx builder $(BUILDX_BUILDER)..."; \
+		docker buildx create \
+			--name $(BUILDX_BUILDER) \
+			--driver docker-container \
+			--driver-opt network=host \
+			--bootstrap --use; \
+	else \
+		echo "Using existing buildx builder $(BUILDX_BUILDER)"; \
+		docker buildx use $(BUILDX_BUILDER); \
+	fi
 
-uboot-tools: $(UBOOT_TAR)
-	docker buildx build --platform linux/arm64 \
+ensure-registry:
+	@if ! docker container inspect $(LOCAL_REGISTRY) >/dev/null 2>&1; then \
+		echo "Creating local registry..."; \
+		docker run -d \
+			--restart=always \
+			-p $(LOCAL_REGISTRY_PORT):5000 \
+			--name $(LOCAL_REGISTRY) \
+			registry:2; \
+	else \
+		if [ "$$(docker inspect -f '{{.State.Running}}' $(LOCAL_REGISTRY))" != "true" ]; then \
+			echo "Starting existing local registry..."; \
+			docker start $(LOCAL_REGISTRY); \
+		fi; \
+	fi
+
+$(DOWNLOAD_PACKAGES_STAMP): docker/download-packages.Dockerfile makefile | $(PACKAGES_DIR)
+	docker build \
+		-f docker/download-packages.Dockerfile \
+		--build-arg UBOOT_VERSION=$(UBOOT_VERSION) \
+		--output type=local,dest=./$(PACKAGES_DIR) .
+	@touch $@
+
+uboot-tools: $(DOWNLOAD_PACKAGES_STAMP)
+	rm -rf "$(UBOOT_TOOLS_OUT)"
+	mkdir -p "$(UBOOT_TOOLS_OUT)"
+	docker buildx build \
+		--platform linux/amd64,linux/arm64 \
 		-f docker/uboot-tools.Dockerfile \
 		--build-arg UBOOT_VERSION=$(UBOOT_VERSION) \
 		--build-arg UBOOT_TAR=$(UBOOT_TAR) \
-		--output type=local,dest=./$(OUT_DIR) .
+		--output type=local,dest=./$(UBOOT_TOOLS_OUT),platform-split=true .
 
-build: .buildinfo
-	mkdir -p $(BIN_DIR) $(OUT_DIR)/crds
-	controller-gen crd paths=$(CRD_PATHS) output:crd:dir=$(OUT_DIR)/crds
-	GOOS=linux GOARCH=arm64 go build -o $(BIN_DIR)/ctl-linux-aarch64-$(VERSION) ./cmd/ctl
+ctl-build-base: ensure-buildx ensure-registry
+	docker buildx build \
+		--platform linux/amd64,linux/arm64 \
+		-f docker/ctl-build-base.Dockerfile \
+		-t $(CTL_BUILD_BASE_IMAGE) \
+		--output type=image,push=true,registry.insecure=true .
 
-build-agent: build uboot-tools
-	docker build \
+build-crds: ctl-build-base | $(OUT_DIR)
+	mkdir -p "$(OUT_DIR)/crds"
+	docker buildx build \
+		--platform $(BUILD_PLATFORM) \
+		-f docker/crdgen.Dockerfile \
+		--build-arg BASE_IMAGE=$(CTL_BUILD_BASE_IMAGE) \
+		--output type=local,dest=./$(OUT_DIR)/crds .
+	rm -rf "$(ASSETS_PATH)/crds"
+	mkdir -p "$(ASSETS_PATH)/crds"
+	cp -R "$(OUT_DIR)/crds/." "$(ASSETS_PATH)/crds/"
+
+build-agent: .buildinfo build-crds uboot-tools
+	docker buildx build \
+		--platform linux/amd64,linux/arm64 \
 		-f docker/ctl-agent.Dockerfile \
-		--platform=linux/arm64 \
+		--build-arg BASE_IMAGE=$(CTL_BUILD_BASE_IMAGE) \
 		--build-arg VERSION=$(VERSION) \
-		-t localhost/monok8s/control-agent:$(VERSION) .
+		-t $(CTL_IMAGE) \
+		--output type=image,push=true,registry.insecure=true .
 
-build-local: .buildinfo
-	mkdir -p $(BIN_DIR)
-	go build -o $(BIN_DIR)/ctl-$(VERSION) ./cmd/ctl
+build-local: .buildinfo build-crds uboot-tools
+	docker buildx build \
+		--platform linux/arm64 \
+		-f docker/ctl-agent.Dockerfile \
+		--build-arg BASE_IMAGE=$(CTL_BUILD_BASE_IMAGE) \
+		--build-arg VERSION=$(VERSION) \
+		--load \
+		-t localhost/monok8s/node-control:$(VERSION) .
+
+push-agent: .buildinfo build-crds uboot-tools
+	test -n "$(IMAGE_REPOSITORY)"
+	docker buildx build \
+		--platform linux/amd64,linux/arm64 \
+		-f docker/ctl-agent.Dockerfile \
+		--build-arg BASE_IMAGE=$(CTL_BUILD_BASE_IMAGE) \
+		--build-arg VERSION=$(VERSION) \
+		-t $(IMAGE_REPOSITORY)/node-control:$(VERSION) \
+		--push .
 
 run-agent:
-	go run -tags dev ./cmd/ctl agent --env-file ./out/cluster.env
+	docker run --rm \
+		-v "$$(pwd)/out:/work/out" \
+		$(CTL_IMAGE) \
+		agent --env-file /work/out/cluster.env
 
 clean:
-	-docker image rm localhost/monok8s/control-agent:$(VERSION)
-	rm -rf $(BIN_DIR) \
-		$(BUILDINFO_FILE) \
-		$(OUT_DIR)/crds
+	-docker image rm localhost/monok8s/node-control:$(VERSION) >/dev/null 2>&1 || true
+	rm -rf \
+		$(OUT_DIR)/crds \
+		$(BUILDINFO_FILE)
 
-all: build build-agent build-local
+distclean: clean
+	rm -rf $(OUT_DIR)
 
-.PHONY: clean all run .buildinfo build build-local build-agent uboot-tools
+dockerclean:
+	@echo "Removing tagged images..."
+	- docker rmi \
+		localhost/monok8s/ctl-build-base:$(VERSION) \
+		localhost/monok8s/node-control:$(VERSION) \
+		localhost/monok8s/crdgen:$(VERSION) \
+		2>/dev/null || true
+
+	@echo "Removing dangling build cache/images..."
+	- docker image prune -f
+	- docker builder prune -f
+
+pkgclean:
+	rm -rf $(PACKAGES_DIR)
+
+all: build-agent build-local
+
+.PHONY: \
+	all clean dockerclean \
+	.buildinfo ensure-buildx ensure-registry \
+	build-crds build-local build-agent build-agent-local push-agent \
+	uboot-tools run-agent run-agent-local

clitools/pkg/apis/monok8s/v1alpha1/groupversion_info.go

@@ -9,30 +9,47 @@ import (
 )
 
 var (
-	Group           = "monok8s.io"
-	Version         = "v1alpha1"
-	MonoKSConfigCRD = "monoksconfigs.monok8s.io"
-	OSUpgradeCRD    = "osupgrades.monok8s.io"
-	APIVersion      = "monok8s.io/v1alpha1"
-	Label           = "monok8s.io/label"
-	Annotation      = "monok8s.io/annotation"
-	ControlAgentKey = "monok8s.io/control-agent"
-	CatalogURL      = "https://example.com/monok8s.io/v1alpha1/catalog.yaml"
+	Group      = "monok8s.io"
+	Version    = "v1alpha1"
+	APIVersion = "monok8s.io/v1alpha1"
+
+	AltPartDeviceLink = "/dev/mksaltpart"
+	BootStateFile     = "/run/monok8s/boot-state.env"
+	CatalogURL        = "https://example.com/monok8s.io/v1alpha1/catalog.yaml"
+	NodeControlKey    = "monok8s.io/node-control"
+	NodeControlName   = "node-control"
+	ControllerName    = "node-controller"
+	NodeAgentName     = "node-agent"
+	EnvConfigDir      = "/opt/monok8s/config"
+	Label             = "monok8s.io/label"
+	MonoKSConfigCRD   = "monoksconfigs.monok8s.io"
+	OSUpgradeCRD      = "osupgrades.monok8s.io"
 )
 
 var (
-	GroupVersion  = schema.GroupVersion{Group: Group, Version: Version}
-	SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)
-	AddToScheme   = SchemeBuilder.AddToScheme
+	SchemeGroupVersion = schema.GroupVersion{Group: Group, Version: Version}
+	SchemeBuilder      = runtime.NewSchemeBuilder(addKnownTypes)
+	AddToScheme        = SchemeBuilder.AddToScheme
 )
 
 func addKnownTypes(scheme *runtime.Scheme) error {
-	scheme.AddKnownTypes(GroupVersion,
+	scheme.AddKnownTypes(SchemeGroupVersion,
 		&MonoKSConfig{},
 		&MonoKSConfigList{},
 		&OSUpgrade{},
 		&OSUpgradeList{},
+		&OSUpgradeProgress{},
+		&OSUpgradeProgressList{},
 	)
-	metav1.AddToGroupVersion(scheme, GroupVersion)
+	metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
 	return nil
 }
+
+func NodeAgentLabels() map[string]string {
+	return map[string]string{
+		"app.kubernetes.io/name":       NodeAgentName,
+		"app.kubernetes.io/component":  "agent",
+		"app.kubernetes.io/part-of":    "monok8s",
+		"app.kubernetes.io/managed-by": NodeControlName,
+	}
+}

clitools/pkg/apis/monok8s/v1alpha1/monoksconfig.go

@@ -25,7 +25,7 @@ type MonoKSConfigSpec struct {
 	ClusterDomain                 string            `json:"clusterDomain,omitempty" yaml:"clusterDomain,omitempty"`
 	ClusterRole                   string            `json:"clusterRole,omitempty" yaml:"clusterRole,omitempty"`
 	InitControlPlane              bool              `json:"initControlPlane,omitempty" yaml:"initControlPlane,omitempty"`
-	EnableControlAgent            bool              `json:"enableControlAgent,omitempty" yaml:"enableControlAgent,omitempty"`
+	EnableNodeControl            bool              `json:"enableNodeControl,omitempty" yaml:"enableNodeControl,omitempty"`
 	PodSubnet                     string            `json:"podSubnet,omitempty" yaml:"podSubnet,omitempty"`
 	ServiceSubnet                 string            `json:"serviceSubnet,omitempty" yaml:"serviceSubnet,omitempty"`
 	APIServerAdvertiseAddress     string            `json:"apiServerAdvertiseAddress,omitempty" yaml:"apiServerAdvertiseAddress,omitempty"`
@@ -40,7 +40,6 @@ type MonoKSConfigSpec struct {
 	KubeProxyNodePortAddresses    []string          `json:"kubeProxyNodePortAddresses,omitempty" yaml:"kubeProxyNodePortAddresses,omitempty"`
 	SubjectAltNames               []string          `json:"subjectAltNames,omitempty" yaml:"subjectAltNames,omitempty"`
 	NodeLabels                    map[string]string `json:"nodeLabels,omitempty" yaml:"nodeLabels,omitempty"`
-	NodeAnnotations               map[string]string `json:"nodeAnnotations,omitempty" yaml:"nodeAnnotations,omitempty"`
 	Network                       NetworkSpec       `json:"network,omitempty" yaml:"network,omitempty"`
 }
 

clitools/pkg/apis/monok8s/v1alpha1/osupgrade.go

@@ -2,17 +2,14 @@ package v1alpha1
 
 import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/runtime"
 )
 
 type OSUpgradePhase string
 
 const (
-	OSUpgradePhasePending    OSUpgradePhase = "Pending"
-	OSUpgradePhaseAccepted   OSUpgradePhase = "Accepted"
-	OSUpgradePhaseRollingOut OSUpgradePhase = "RollingOut"
-	OSUpgradePhaseCompleted  OSUpgradePhase = "Completed"
-	OSUpgradePhaseRejected   OSUpgradePhase = "Rejected"
+	OSUpgradePhasePending  OSUpgradePhase = "Pending"
+	OSUpgradePhaseAccepted OSUpgradePhase = "Accepted"
+	OSUpgradePhaseRejected OSUpgradePhase = "Rejected"
 )
 
 type OSUpgradeProgressPhase string
@@ -21,27 +18,35 @@ const (
 	OSUpgradeProgressPhasePending     OSUpgradeProgressPhase = "pending"
 	OSUpgradeProgressPhaseDownloading OSUpgradeProgressPhase = "downloading"
 	OSUpgradeProgressPhaseWriting     OSUpgradeProgressPhase = "writing"
-	OSUpgradeProgressPhaseRebooting   OSUpgradeProgressPhase = "rebooting"
 	OSUpgradeProgressPhaseVerifying   OSUpgradeProgressPhase = "verifying"
 	OSUpgradeProgressPhaseCompleted   OSUpgradeProgressPhase = "completed"
 	OSUpgradeProgressPhaseFailed      OSUpgradeProgressPhase = "failed"
 	OSUpgradeProgressPhaseRejected    OSUpgradeProgressPhase = "rejected"
+
+	// Rebooting is the point-of-no-return phase.
+	//
+	// Once a node reaches Rebooting, the agent may have already changed the boot
+	// environment and requested a reboot. The controller must not supersede,
+	// retry, retarget, or otherwise mutate this progress object until the node
+	// comes back and the agent reports Completed or Failed.
+	OSUpgradeProgressPhaseRebooting OSUpgradeProgressPhase = "rebooting"
 )
 
+// +genclient
 // +kubebuilder:object:root=true
 // +kubebuilder:subresource:status
 // +kubebuilder:resource:scope=Namespaced,shortName=osu
 // +kubebuilder:printcolumn:name="Desired",type=string,JSONPath=`.spec.desiredVersion`
 // +kubebuilder:printcolumn:name="Resolved",type=string,JSONPath=`.status.resolvedVersion`
 // +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
-// +kubebuilder:printcolumn:name="Targets",type=integer,JSONPath=`.status.summary.targetedNodes`
-// +kubebuilder:printcolumn:name="OK",type=integer,JSONPath=`.status.summary.succeededNodes`
-// +kubebuilder:printcolumn:name="Fail",type=integer,JSONPath=`.status.summary.failedNodes`
 type OSUpgrade struct {
 	metav1.TypeMeta   `json:",inline" yaml:",inline"`
 	metav1.ObjectMeta `json:"metadata,omitempty" yaml:"metadata,omitempty"`
 
-	Spec   OSUpgradeSpec    `json:"spec,omitempty" yaml:"spec,omitempty"`
+	// Specification of the desired behavior of the OSUpgrade.
+	Spec OSUpgradeSpec `json:"spec,omitempty" yaml:"spec,omitempty"`
+
+	// Most recently observed status of the OSUpgrade.
 	Status *OSUpgradeStatus `json:"status,omitempty" yaml:"status,omitempty"`
 }
 
@@ -53,40 +58,37 @@ type OSUpgradeList struct {
 }
 
 type OSUpgradeSpec struct {
-	// User request, can be "stable" or an explicit version like "v1.35.3".
+	// +kubebuilder:validation:MinLength=1
 	DesiredVersion string `json:"desiredVersion,omitempty" yaml:"desiredVersion,omitempty"`
 
-	Catalog *VersionCatalogSource `json:"catalog,omitempty"`
+	// +kubebuilder:validation:Enum=fast;balanced;safe
+	// +kubebuilder:default=balanced
+	// Profiles (TODO)
+	//    safe     - api-server can be responsive most of the time
+	//    balanced - api-server can sometimes be unresponsive
+	//    fast     - disable throttling. Good for worker node.
+	FlashProfile string `json:"flashProfile,omitempty" yaml:"flashProfile,omitempty"`
 
+	Catalog      *VersionCatalogSource `json:"catalog,omitempty" yaml:"catalog,omitempty"`
 	NodeSelector *metav1.LabelSelector `json:"nodeSelector,omitempty" yaml:"nodeSelector,omitempty"`
 }
 
 type VersionCatalogSource struct {
-	URL       string `json:"url,omitempty"`
-	Inline    string `json:"inline,omitempty"`
-	ConfigMap string `json:"configMapRef,omitempty"`
+	URL       string `json:"url,omitempty" yaml:"url,omitempty"`
+	Inline    string `json:"inline,omitempty" yaml:"inline,omitempty"`
+	ConfigMap string `json:"configMapRef,omitempty" yaml:"configMapRef,omitempty"`
 }
 
 type OSUpgradeStatus struct {
 	Phase              OSUpgradePhase     `json:"phase,omitempty" yaml:"phase,omitempty"`
 	ResolvedVersion    string             `json:"resolvedVersion,omitempty" yaml:"resolvedVersion,omitempty"`
 	ObservedGeneration int64              `json:"observedGeneration,omitempty" yaml:"observedGeneration,omitempty"`
-	Summary            OSUpgradeSummary   `json:"summary,omitempty" yaml:"summary,omitempty"`
 	Conditions         []metav1.Condition `json:"conditions,omitempty" yaml:"conditions,omitempty"`
-
-	// Optional, useful when rejected.
-	Reason  string `json:"reason,omitempty" yaml:"reason,omitempty"`
-	Message string `json:"message,omitempty" yaml:"message,omitempty"`
-}
-
-type OSUpgradeSummary struct {
-	TargetedNodes  int32 `json:"targetedNodes,omitempty" yaml:"targetedNodes,omitempty"`
-	PendingNodes   int32 `json:"pendingNodes,omitempty" yaml:"pendingNodes,omitempty"`
-	RunningNodes   int32 `json:"runningNodes,omitempty" yaml:"runningNodes,omitempty"`
-	SucceededNodes int32 `json:"succeededNodes,omitempty" yaml:"succeededNodes,omitempty"`
-	FailedNodes    int32 `json:"failedNodes,omitempty" yaml:"failedNodes,omitempty"`
+	Reason             string             `json:"reason,omitempty" yaml:"reason,omitempty"`
+	Message            string             `json:"message,omitempty" yaml:"message,omitempty"`
 }
 
+// +genclient
 // +kubebuilder:object:root=true
 // +kubebuilder:subresource:status
 // +kubebuilder:resource:scope=Namespaced,shortName=osup
@@ -94,12 +96,15 @@ type OSUpgradeSummary struct {
 // +kubebuilder:printcolumn:name="Source",type=string,JSONPath=`.spec.sourceRef.name`
 // +kubebuilder:printcolumn:name="Current",type=string,JSONPath=`.status.currentVersion`
 // +kubebuilder:printcolumn:name="Target",type=string,JSONPath=`.status.targetVersion`
-// +kubebuilder:printcolumn:name="Status",type=string,JSONPath=`.status.phase`
+// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
 type OSUpgradeProgress struct {
 	metav1.TypeMeta   `json:",inline" yaml:",inline"`
 	metav1.ObjectMeta `json:"metadata,omitempty" yaml:"metadata,omitempty"`
 
-	Spec   OSUpgradeProgressSpec    `json:"spec,omitempty" yaml:"spec,omitempty"`
+	// Specification of the desired behavior of the OSUpgradeProgress.
+	Spec OSUpgradeProgressSpec `json:"spec,omitempty" yaml:"spec,omitempty"`
+
+	// Most recently observed status of the OSUpgradeProgress.
 	Status *OSUpgradeProgressStatus `json:"status,omitempty" yaml:"status,omitempty"`
 }
 
@@ -112,60 +117,55 @@ type OSUpgradeProgressList struct {
 
 type OSUpgradeProgressSpec struct {
 	SourceRef OSUpgradeSourceRef `json:"sourceRef,omitempty" yaml:"sourceRef,omitempty"`
-	NodeName  string             `json:"nodeName,omitempty" yaml:"nodeName,omitempty"`
+
+	// RetryNonce triggers a retry when its value changes.
+	// Users can update this field (for example, set it to the current time)
+	// to request a retry of a failed OS upgrade.
+	RetryNonce string `json:"retryNonce,omitempty" yaml:"retryNonce,omitempty"`
+
+	NodeName string `json:"nodeName,omitempty" yaml:"nodeName,omitempty"`
 }
 
 type OSUpgradeSourceRef struct {
-	Name string `json:"name,omitempty" yaml:"name,omitempty"`
+	Name      string `json:"name,omitempty" yaml:"name,omitempty"`
+	Namespace string `json:"namespace,omitempty" yaml:"namespace,omitempty"`
 }
 
 type OSUpgradeProgressStatus struct {
-	CurrentVersion string                 `json:"currentVersion,omitempty" yaml:"currentVersion,omitempty"`
-	TargetVersion  string                 `json:"targetVersion,omitempty" yaml:"targetVersion,omitempty"`
-	Phase          OSUpgradeProgressPhase `json:"phase,omitempty" yaml:"phase,omitempty"`
+	CurrentVersion    string                 `json:"currentVersion,omitempty" yaml:"currentVersion,omitempty"`
+	TargetVersion     string                 `json:"targetVersion,omitempty" yaml:"targetVersion,omitempty"`
+	Phase             OSUpgradeProgressPhase `json:"phase,omitempty" yaml:"phase,omitempty"`
+	StartedAt         *metav1.Time           `json:"startedAt,omitempty" yaml:"startedAt,omitempty"`
+	CompletedAt       *metav1.Time           `json:"completedAt,omitempty" yaml:"completedAt,omitempty"`
+	LastUpdatedAt     *metav1.Time           `json:"lastUpdatedAt,omitempty" yaml:"lastUpdatedAt,omitempty"`
+	RetryCount        int32                  `json:"retryCount,omitempty" yaml:"retryCount,omitempty"`
+	InactivePartition string                 `json:"inactivePartition,omitempty" yaml:"inactivePartition,omitempty"`
+	FailureReason     string                 `json:"failureReason,omitempty" yaml:"failureReason,omitempty"`
+	Message           string                 `json:"message,omitempty" yaml:"message,omitempty"`
+	PlannedPath       []string               `json:"plannedPath,omitempty" yaml:"plannedPath,omitempty"`
+	CurrentStep       int32                  `json:"currentStep,omitempty" yaml:"currentStep,omitempty"`
+	CurrentFrom       string                 `json:"currentFrom,omitempty" yaml:"currentFrom,omitempty"`
+	CurrentTo         string                 `json:"currentTo,omitempty" yaml:"currentTo,omitempty"`
 
-	StartedAt         *metav1.Time `json:"startedAt,omitempty" yaml:"startedAt,omitempty"`
-	CompletedAt       *metav1.Time `json:"completedAt,omitempty" yaml:"completedAt,omitempty"`
-	LastUpdatedAt     *metav1.Time `json:"lastUpdatedAt,omitempty" yaml:"lastUpdatedAt,omitempty"`
-	RetryCount        int32        `json:"retryCount,omitempty" yaml:"retryCount,omitempty"`
-	InactivePartition string       `json:"inactivePartition,omitempty" yaml:"inactivePartition,omitempty"`
-	FailureReason     string       `json:"failureReason,omitempty" yaml:"failureReason,omitempty"`
-	Message           string       `json:"message,omitempty" yaml:"message,omitempty"`
-
-	PlannedPath []string `json:"plannedPath,omitempty"`
-	CurrentStep int32    `json:"currentStep,omitempty"`
-	CurrentFrom string   `json:"currentFrom,omitempty"`
-	CurrentTo   string   `json:"currentTo,omitempty"`
+	// ObservedRetryNonce records the last retryNonce value the agent accepted.
+	// When spec.retryNonce is changed by the user and differs from this value,
+	// the agent may retry a failed upgrade.
+	// +optional
+	ObservedRetryNonce string `json:"observedRetryNonce,omitempty"`
 }
 
-func (in *OSUpgrade) DeepCopyObject() runtime.Object {
-	if in == nil {
-		return nil
+func (osu OSUpgrade) StatusPhase() string {
+	phase := ""
+	if osu.Status != nil {
+		phase = string(osu.Status.Phase)
 	}
-	out := *in
-	return &out
+	return phase
 }
 
-func (in *OSUpgradeList) DeepCopyObject() runtime.Object {
-	if in == nil {
-		return nil
+func (osup OSUpgradeProgress) StatusPhase() string {
+	phase := ""
+	if osup.Status != nil {
+		phase = string(osup.Status.Phase)
 	}
-	out := *in
-	return &out
-}
-
-func (in *OSUpgradeProgress) DeepCopyObject() runtime.Object {
-	if in == nil {
-		return nil
-	}
-	out := *in
-	return &out
-}
-
-func (in *OSUpgradeProgressList) DeepCopyObject() runtime.Object {
-	if in == nil {
-		return nil
-	}
-	out := *in
-	return &out
+	return phase
 }

clitools/pkg/apis/monok8s/v1alpha1/zz_generated.deepcopy.go

@@ -0,0 +1,395 @@
+//go:build !ignore_autogenerated
+
+/* MIT License */
+
+// Code generated by controller-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	"k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+)
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *MonoKSConfig) DeepCopyInto(out *MonoKSConfig) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	in.Spec.DeepCopyInto(&out.Spec)
+	if in.Status != nil {
+		in, out := &in.Status, &out.Status
+		*out = new(MonoKSConfigStatus)
+		(*in).DeepCopyInto(*out)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MonoKSConfig.
+func (in *MonoKSConfig) DeepCopy() *MonoKSConfig {
+	if in == nil {
+		return nil
+	}
+	out := new(MonoKSConfig)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *MonoKSConfigList) DeepCopyInto(out *MonoKSConfigList) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ListMeta.DeepCopyInto(&out.ListMeta)
+	if in.Items != nil {
+		in, out := &in.Items, &out.Items
+		*out = make([]MonoKSConfig, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MonoKSConfigList.
+func (in *MonoKSConfigList) DeepCopy() *MonoKSConfigList {
+	if in == nil {
+		return nil
+	}
+	out := new(MonoKSConfigList)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *MonoKSConfigSpec) DeepCopyInto(out *MonoKSConfigSpec) {
+	*out = *in
+	if in.KubeProxyNodePortAddresses != nil {
+		in, out := &in.KubeProxyNodePortAddresses, &out.KubeProxyNodePortAddresses
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+	if in.SubjectAltNames != nil {
+		in, out := &in.SubjectAltNames, &out.SubjectAltNames
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+	if in.NodeLabels != nil {
+		in, out := &in.NodeLabels, &out.NodeLabels
+		*out = make(map[string]string, len(*in))
+		for key, val := range *in {
+			(*out)[key] = val
+		}
+	}
+	in.Network.DeepCopyInto(&out.Network)
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MonoKSConfigSpec.
+func (in *MonoKSConfigSpec) DeepCopy() *MonoKSConfigSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(MonoKSConfigSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *MonoKSConfigStatus) DeepCopyInto(out *MonoKSConfigStatus) {
+	*out = *in
+	if in.Conditions != nil {
+		in, out := &in.Conditions, &out.Conditions
+		*out = make([]v1.Condition, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+	if in.AppliedSteps != nil {
+		in, out := &in.AppliedSteps, &out.AppliedSteps
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MonoKSConfigStatus.
+func (in *MonoKSConfigStatus) DeepCopy() *MonoKSConfigStatus {
+	if in == nil {
+		return nil
+	}
+	out := new(MonoKSConfigStatus)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *NetworkSpec) DeepCopyInto(out *NetworkSpec) {
+	*out = *in
+	if in.DNSNameservers != nil {
+		in, out := &in.DNSNameservers, &out.DNSNameservers
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+	if in.DNSSearchDomains != nil {
+		in, out := &in.DNSSearchDomains, &out.DNSSearchDomains
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new NetworkSpec.
+func (in *NetworkSpec) DeepCopy() *NetworkSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(NetworkSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgrade) DeepCopyInto(out *OSUpgrade) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	in.Spec.DeepCopyInto(&out.Spec)
+	if in.Status != nil {
+		in, out := &in.Status, &out.Status
+		*out = new(OSUpgradeStatus)
+		(*in).DeepCopyInto(*out)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgrade.
+func (in *OSUpgrade) DeepCopy() *OSUpgrade {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgrade)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *OSUpgrade) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeList) DeepCopyInto(out *OSUpgradeList) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ListMeta.DeepCopyInto(&out.ListMeta)
+	if in.Items != nil {
+		in, out := &in.Items, &out.Items
+		*out = make([]OSUpgrade, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeList.
+func (in *OSUpgradeList) DeepCopy() *OSUpgradeList {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeList)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *OSUpgradeList) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeProgress) DeepCopyInto(out *OSUpgradeProgress) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	out.Spec = in.Spec
+	if in.Status != nil {
+		in, out := &in.Status, &out.Status
+		*out = new(OSUpgradeProgressStatus)
+		(*in).DeepCopyInto(*out)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeProgress.
+func (in *OSUpgradeProgress) DeepCopy() *OSUpgradeProgress {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeProgress)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *OSUpgradeProgress) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeProgressList) DeepCopyInto(out *OSUpgradeProgressList) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ListMeta.DeepCopyInto(&out.ListMeta)
+	if in.Items != nil {
+		in, out := &in.Items, &out.Items
+		*out = make([]OSUpgradeProgress, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeProgressList.
+func (in *OSUpgradeProgressList) DeepCopy() *OSUpgradeProgressList {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeProgressList)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *OSUpgradeProgressList) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeProgressSpec) DeepCopyInto(out *OSUpgradeProgressSpec) {
+	*out = *in
+	out.SourceRef = in.SourceRef
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeProgressSpec.
+func (in *OSUpgradeProgressSpec) DeepCopy() *OSUpgradeProgressSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeProgressSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeProgressStatus) DeepCopyInto(out *OSUpgradeProgressStatus) {
+	*out = *in
+	if in.StartedAt != nil {
+		in, out := &in.StartedAt, &out.StartedAt
+		*out = (*in).DeepCopy()
+	}
+	if in.CompletedAt != nil {
+		in, out := &in.CompletedAt, &out.CompletedAt
+		*out = (*in).DeepCopy()
+	}
+	if in.LastUpdatedAt != nil {
+		in, out := &in.LastUpdatedAt, &out.LastUpdatedAt
+		*out = (*in).DeepCopy()
+	}
+	if in.PlannedPath != nil {
+		in, out := &in.PlannedPath, &out.PlannedPath
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeProgressStatus.
+func (in *OSUpgradeProgressStatus) DeepCopy() *OSUpgradeProgressStatus {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeProgressStatus)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeSourceRef) DeepCopyInto(out *OSUpgradeSourceRef) {
+	*out = *in
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeSourceRef.
+func (in *OSUpgradeSourceRef) DeepCopy() *OSUpgradeSourceRef {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeSourceRef)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeSpec) DeepCopyInto(out *OSUpgradeSpec) {
+	*out = *in
+	if in.Catalog != nil {
+		in, out := &in.Catalog, &out.Catalog
+		*out = new(VersionCatalogSource)
+		**out = **in
+	}
+	if in.NodeSelector != nil {
+		in, out := &in.NodeSelector, &out.NodeSelector
+		*out = new(v1.LabelSelector)
+		(*in).DeepCopyInto(*out)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeSpec.
+func (in *OSUpgradeSpec) DeepCopy() *OSUpgradeSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeStatus) DeepCopyInto(out *OSUpgradeStatus) {
+	*out = *in
+	if in.Conditions != nil {
+		in, out := &in.Conditions, &out.Conditions
+		*out = make([]v1.Condition, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeStatus.
+func (in *OSUpgradeStatus) DeepCopy() *OSUpgradeStatus {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeStatus)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *VersionCatalogSource) DeepCopyInto(out *VersionCatalogSource) {
+	*out = *in
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new VersionCatalogSource.
+func (in *VersionCatalogSource) DeepCopy() *VersionCatalogSource {
+	if in == nil {
+		return nil
+	}
+	out := new(VersionCatalogSource)
+	in.DeepCopyInto(out)
+	return out
+}

clitools/pkg/assets/crds/monok8s.io_monoksconfigs.yaml

@@ -0,0 +1,178 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.20.1
+  name: monoksconfigs.monok8s.io
+spec:
+  group: monok8s.io
+  names:
+    kind: MonoKSConfig
+    listKind: MonoKSConfigList
+    plural: monoksconfigs
+    singular: monoksconfig
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              allowSchedulingOnControlPlane:
+                type: boolean
+              apiServerAdvertiseAddress:
+                type: string
+              apiServerEndpoint:
+                type: string
+              bootstrapToken:
+                type: string
+              clusterDomain:
+                type: string
+              clusterName:
+                type: string
+              clusterRole:
+                type: string
+              cniPlugin:
+                type: string
+              containerRuntimeEndpoint:
+                type: string
+              controlPlaneCertKey:
+                type: string
+              discoveryTokenCACertHash:
+                type: string
+              enableNodeControl:
+                type: boolean
+              initControlPlane:
+                type: boolean
+              kubeProxyNodePortAddresses:
+                items:
+                  type: string
+                type: array
+              kubernetesVersion:
+                type: string
+              network:
+                properties:
+                  dnsNameservers:
+                    items:
+                      type: string
+                    type: array
+                  dnsSearchDomains:
+                    items:
+                      type: string
+                    type: array
+                  hostname:
+                    type: string
+                  managementCIDR:
+                    type: string
+                  managementGateway:
+                    type: string
+                  managementIface:
+                    type: string
+                type: object
+              nodeLabels:
+                additionalProperties:
+                  type: string
+                type: object
+              nodeName:
+                type: string
+              podSubnet:
+                type: string
+              serviceSubnet:
+                type: string
+              skipImageCheck:
+                type: boolean
+              subjectAltNames:
+                items:
+                  type: string
+                type: array
+            type: object
+          status:
+            properties:
+              appliedSteps:
+                items:
+                  type: string
+                type: array
+              conditions:
+                items:
+                  description: Condition contains details for one aspect of the current
+                    state of this API Resource.
+                  properties:
+                    lastTransitionTime:
+                      description: |-
+                        lastTransitionTime is the last time the condition transitioned from one status to another.
+                        This should be when the underlying condition changed.  If that is not known, then using the time when the API field changed is acceptable.
+                      format: date-time
+                      type: string
+                    message:
+                      description: |-
+                        message is a human readable message indicating details about the transition.
+                        This may be an empty string.
+                      maxLength: 32768
+                      type: string
+                    observedGeneration:
+                      description: |-
+                        observedGeneration represents the .metadata.generation that the condition was set based upon.
+                        For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
+                        with respect to the current state of the instance.
+                      format: int64
+                      minimum: 0
+                      type: integer
+                    reason:
+                      description: |-
+                        reason contains a programmatic identifier indicating the reason for the condition's last transition.
+                        Producers of specific condition types may define expected values and meanings for this field,
+                        and whether the values are considered a guaranteed API.
+                        The value should be a CamelCase string.
+                        This field may not be empty.
+                      maxLength: 1024
+                      minLength: 1
+                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
+                      type: string
+                    status:
+                      description: status of the condition, one of True, False, Unknown.
+                      enum:
+                      - "True"
+                      - "False"
+                      - Unknown
+                      type: string
+                    type:
+                      description: type of condition in CamelCase or in foo.example.com/CamelCase.
+                      maxLength: 316
+                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
+                      type: string
+                  required:
+                  - lastTransitionTime
+                  - message
+                  - reason
+                  - status
+                  - type
+                  type: object
+                type: array
+              observedGeneration:
+                format: int64
+                type: integer
+              phase:
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true

clitools/pkg/assets/crds/monok8s.io_osupgradeprogresses.yaml

@@ -0,0 +1,124 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.20.1
+  name: osupgradeprogresses.monok8s.io
+spec:
+  group: monok8s.io
+  names:
+    kind: OSUpgradeProgress
+    listKind: OSUpgradeProgressList
+    plural: osupgradeprogresses
+    shortNames:
+    - osup
+    singular: osupgradeprogress
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.nodeName
+      name: Node
+      type: string
+    - jsonPath: .spec.sourceRef.name
+      name: Source
+      type: string
+    - jsonPath: .status.currentVersion
+      name: Current
+      type: string
+    - jsonPath: .status.targetVersion
+      name: Target
+      type: string
+    - jsonPath: .status.phase
+      name: Phase
+      type: string
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: Specification of the desired behavior of the OSUpgradeProgress.
+            properties:
+              nodeName:
+                type: string
+              retryNonce:
+                description: |-
+                  RetryNonce triggers a retry when its value changes.
+                  Users can update this field (for example, set it to the current time)
+                  to request a retry of a failed OS upgrade.
+                type: string
+              sourceRef:
+                properties:
+                  name:
+                    type: string
+                  namespace:
+                    type: string
+                type: object
+            type: object
+          status:
+            description: Most recently observed status of the OSUpgradeProgress.
+            properties:
+              completedAt:
+                format: date-time
+                type: string
+              currentFrom:
+                type: string
+              currentStep:
+                format: int32
+                type: integer
+              currentTo:
+                type: string
+              currentVersion:
+                type: string
+              failureReason:
+                type: string
+              inactivePartition:
+                type: string
+              lastUpdatedAt:
+                format: date-time
+                type: string
+              message:
+                type: string
+              observedRetryNonce:
+                description: |-
+                  ObservedRetryNonce records the last retryNonce value the agent accepted.
+                  When spec.retryNonce is changed by the user and differs from this value,
+                  the agent may retry a failed upgrade.
+                type: string
+              phase:
+                type: string
+              plannedPath:
+                items:
+                  type: string
+                type: array
+              retryCount:
+                format: int32
+                type: integer
+              startedAt:
+                format: date-time
+                type: string
+              targetVersion:
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}

clitools/pkg/assets/crds/monok8s.io_osupgrades.yaml

@@ -0,0 +1,202 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.20.1
+  name: osupgrades.monok8s.io
+spec:
+  group: monok8s.io
+  names:
+    kind: OSUpgrade
+    listKind: OSUpgradeList
+    plural: osupgrades
+    shortNames:
+    - osu
+    singular: osupgrade
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.desiredVersion
+      name: Desired
+      type: string
+    - jsonPath: .status.resolvedVersion
+      name: Resolved
+      type: string
+    - jsonPath: .status.phase
+      name: Phase
+      type: string
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: Specification of the desired behavior of the OSUpgrade.
+            properties:
+              catalog:
+                properties:
+                  configMapRef:
+                    type: string
+                  inline:
+                    type: string
+                  url:
+                    type: string
+                type: object
+              desiredVersion:
+                minLength: 1
+                type: string
+              flashProfile:
+                default: balanced
+                description: |-
+                  Profiles (TODO)
+                     safe     - api-server can be responsive most of the time
+                     balanced - api-server can sometimes be unresponsive
+                     fast     - disable throttling. Good for worker node.
+                enum:
+                - fast
+                - balanced
+                - safe
+                type: string
+              nodeSelector:
+                description: |-
+                  A label selector is a label query over a set of resources. The result of matchLabels and
+                  matchExpressions are ANDed. An empty label selector matches all objects. A null
+                  label selector matches no objects.
+                properties:
+                  matchExpressions:
+                    description: matchExpressions is a list of label selector requirements.
+                      The requirements are ANDed.
+                    items:
+                      description: |-
+                        A label selector requirement is a selector that contains values, a key, and an operator that
+                        relates the key and values.
+                      properties:
+                        key:
+                          description: key is the label key that the selector applies
+                            to.
+                          type: string
+                        operator:
+                          description: |-
+                            operator represents a key's relationship to a set of values.
+                            Valid operators are In, NotIn, Exists and DoesNotExist.
+                          type: string
+                        values:
+                          description: |-
+                            values is an array of string values. If the operator is In or NotIn,
+                            the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                            the values array must be empty. This array is replaced during a strategic
+                            merge patch.
+                          items:
+                            type: string
+                          type: array
+                          x-kubernetes-list-type: atomic
+                      required:
+                      - key
+                      - operator
+                      type: object
+                    type: array
+                    x-kubernetes-list-type: atomic
+                  matchLabels:
+                    additionalProperties:
+                      type: string
+                    description: |-
+                      matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                      map is equivalent to an element of matchExpressions, whose key field is "key", the
+                      operator is "In", and the values array contains only "value". The requirements are ANDed.
+                    type: object
+                type: object
+                x-kubernetes-map-type: atomic
+            type: object
+          status:
+            description: Most recently observed status of the OSUpgrade.
+            properties:
+              conditions:
+                items:
+                  description: Condition contains details for one aspect of the current
+                    state of this API Resource.
+                  properties:
+                    lastTransitionTime:
+                      description: |-
+                        lastTransitionTime is the last time the condition transitioned from one status to another.
+                        This should be when the underlying condition changed.  If that is not known, then using the time when the API field changed is acceptable.
+                      format: date-time
+                      type: string
+                    message:
+                      description: |-
+                        message is a human readable message indicating details about the transition.
+                        This may be an empty string.
+                      maxLength: 32768
+                      type: string
+                    observedGeneration:
+                      description: |-
+                        observedGeneration represents the .metadata.generation that the condition was set based upon.
+                        For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
+                        with respect to the current state of the instance.
+                      format: int64
+                      minimum: 0
+                      type: integer
+                    reason:
+                      description: |-
+                        reason contains a programmatic identifier indicating the reason for the condition's last transition.
+                        Producers of specific condition types may define expected values and meanings for this field,
+                        and whether the values are considered a guaranteed API.
+                        The value should be a CamelCase string.
+                        This field may not be empty.
+                      maxLength: 1024
+                      minLength: 1
+                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
+                      type: string
+                    status:
+                      description: status of the condition, one of True, False, Unknown.
+                      enum:
+                      - "True"
+                      - "False"
+                      - Unknown
+                      type: string
+                    type:
+                      description: type of condition in CamelCase or in foo.example.com/CamelCase.
+                      maxLength: 316
+                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
+                      type: string
+                  required:
+                  - lastTransitionTime
+                  - message
+                  - reason
+                  - status
+                  - type
+                  type: object
+                type: array
+              message:
+                type: string
+              observedGeneration:
+                format: int64
+                type: integer
+              phase:
+                type: string
+              reason:
+                type: string
+              resolvedVersion:
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}

clitools/pkg/assets/embed.go

@@ -0,0 +1,6 @@
+package assets
+
+import "embed"
+
+//go:embed crds/*.yaml
+var CRDs embed.FS

clitools/pkg/assets/render.go

@@ -0,0 +1,49 @@
+package assets
+
+import (
+	"fmt"
+	"io"
+	"path/filepath"
+	"sort"
+)
+
+func PrintCRDs(out io.Writer) error {
+	entries, err := CRDs.ReadDir("crds")
+	if err != nil {
+		return err
+	}
+
+	names := make([]string, 0, len(entries))
+	for _, entry := range entries {
+		if entry.IsDir() {
+			continue
+		}
+		if filepath.Ext(entry.Name()) != ".yaml" {
+			continue
+		}
+		names = append(names, entry.Name())
+	}
+
+	sort.Strings(names)
+
+	for _, name := range names {
+		b, err := CRDs.ReadFile("crds/" + name)
+		if err != nil {
+			return err
+		}
+
+		if _, err := fmt.Fprintln(out, "---"); err != nil {
+			return err
+		}
+		if _, err := out.Write(b); err != nil {
+			return err
+		}
+		if len(b) == 0 || b[len(b)-1] != '\n' {
+			if _, err := fmt.Fprintln(out); err != nil {
+				return err
+			}
+		}
+	}
+
+	return nil
+}

clitools/pkg/bootstrap/registry.go

@@ -23,28 +23,35 @@ func NewRegistry(ctx *node.NodeContext) *Registry {
 
 	return &Registry{
 		steps: map[string]node.Step{
-			"ApplyControlAgentDaemonSetResources":    node.ApplyControlAgentDaemonSetResources,
+			"ApplyNodeControlDaemonSetResources":    node.ApplyNodeControlDaemonSetResources,
 			"ApplyLocalNodeMetadataIfPossible":       node.ApplyLocalNodeMetadataIfPossible,
 			"CheckForVersionSkew":                    node.CheckForVersionSkew,
 			"ClassifyBootstrapAction":                node.ClassifyBootstrapAction,
+			"ConfigureABBoot":                        uboot.ConfigureABBoot,
 			"ConfigureDNS":                           node.ConfigureDNS(netCfg),
 			"ConfigureDefaultCNI":                    node.ConfigureDefaultCNI,
 			"ConfigureHostname":                      node.ConfigureHostname(netCfg),
 			"ConfigureMgmtInterface":                 node.ConfigureMgmtInterface(netCfg),
-			"ConfigureABBoot":                        uboot.ConfigureABBoot,
 			"ConfigureUBootCommands":                 uboot.ConfigureUBootCommands,
 			"DetectLocalClusterState":                node.DetectLocalClusterState,
+			"EngageControlGate":                      node.EngageControlGate,
 			"EnsureIPForward":                        node.EnsureIPForward,
+			"MountAltImageStore":                     node.MountAltImageStore,
 			"ReconcileControlPlane":                  node.ReconcileControlPlane,
 			"ReconcileWorker":                        node.ReconcileWorker,
+			"ReleaseControlGate":                     node.ReleaseControlGate,
 			"RunKubeadmInit":                         node.RunKubeadmInit,
 			"RunKubeadmJoin":                         node.RunKubeadmJoin,
 			"RunKubeadmUpgradeApply":                 node.RunKubeadmUpgradeApply,
 			"RunKubeadmUpgradeNode":                  node.RunKubeadmUpgradeNode,
 			"StartCRIO":                              node.StartCRIO,
+			"UnmountAltImageStore":                   node.UnmountAltImageStore,
 			"ValidateNodeIPAndAPIServerReachability": node.ValidateNodeIPAndAPIServerReachability,
 			"ValidateRequiredImagesPresent":          node.ValidateRequiredImagesPresent,
 			"WaitForExistingClusterIfNeeded":         node.WaitForExistingClusterIfNeeded,
+
+			// Diagnostics
+			"DiagTestDiskWrite": node.DiagTestDiskWrite,
 		},
 	}
 }

clitools/pkg/bootstrap/runner.go

@@ -62,6 +62,16 @@ func NewRunner(cfg *monov1alpha1.MonoKSConfig) *Runner {
 				Name:   "Configure default CNI",
 				Desc:   "Install or configure default container networking (CNI bridge, IPAM, etc.)",
 			},
+			{
+				RegKey: "MountAltImageStore",
+				Name:   "Mount alt image store for CRI-O. Needed for upgrade.",
+				Desc:   "Will be unmount after kubeadm upgrade apply",
+			},
+			{
+				RegKey: "EngageControlGate",
+				Name:   "Engage the control gate",
+				Desc:   "Prevents agent watching resources prematurely",
+			},
 			{
 				RegKey: "StartCRIO",
 				Name:   "Start CRI-O runtime",
@@ -102,6 +112,11 @@ func NewRunner(cfg *monov1alpha1.MonoKSConfig) *Runner {
 				Name:   "Wait for existing cluster",
 				Desc:   "Block until control plane is reachable when joining or reconciling an existing cluster",
 			},
+			{
+				RegKey: "CheckForVersionSkew",
+				Name:   "Check for version skew",
+				Desc:   "Validate wether version satisfy the requirements againts current cluster if any",
+			},
 			{
 				RegKey: "ReconcileControlPlane",
 				Name:   "Reconcile control plane",
@@ -112,11 +127,6 @@ func NewRunner(cfg *monov1alpha1.MonoKSConfig) *Runner {
 				Name:   "Reconcile worker node",
 				Desc:   "Reconcile the worker node",
 			},
-			{
-				RegKey: "CheckForVersionSkew",
-				Name:   "Check for version skew",
-				Desc:   "Validate wether version satisfy the requirements againts current cluster if any",
-			},
 			{
 				RegKey: "RunKubeadmUpgradeApply",
 				Name:   "Run kubeadm upgrade apply",
@@ -127,6 +137,11 @@ func NewRunner(cfg *monov1alpha1.MonoKSConfig) *Runner {
 				Name:   "Run kubeadm upgrade node",
 				Desc:   "Upgrade node components (kubelet, config) to match control plane",
 			},
+			{
+				RegKey: "UnmountAltImageStore",
+				Name:   "Unmount alt image store",
+				Desc:   "Rewrite CRIO storage.conf. Then restart CRIO. Then unmount.",
+			},
 			{
 				RegKey: "ApplyLocalNodeMetadataIfPossible",
 				Name:   "Apply node metadata",
@@ -143,10 +158,15 @@ func NewRunner(cfg *monov1alpha1.MonoKSConfig) *Runner {
 				Desc:   "Make A/B booting possible",
 			},
 			{
-				RegKey: "ApplyControlAgentDaemonSetResources",
+				RegKey: "ApplyNodeControlDaemonSetResources",
 				Name:   "Apply daemonset for control agent",
 				Desc:   "Control agent handles OSUpgrade resources",
 			},
+			{
+				RegKey: "ReleaseControlGate",
+				Name:   "Release the control gate",
+				Desc:   "Allow agent to start watching resources",
+			},
 		},
 	}
 }

clitools/pkg/catalog/types.go

@@ -8,6 +8,7 @@ type VersionCatalog struct {
 
 type CatalogImage struct {
 	Version  string `json:"version" yaml:"version"`
+	Patch    int    `json:"patch" yaml:"version"`
 	URL      string `json:"url" yaml:"url"`
 	Checksum string `json:"checksum,omitempty" yaml:"checksum,omitempty"`
 	Size     int64  `json:"size,omitempty" yaml:"size,omitempty"`

clitools/pkg/cmd/agent/agent.go

@@ -3,14 +3,13 @@ package agent
 import (
 	"context"
 	"fmt"
+	"os"
+	"path/filepath"
 	"time"
 
 	"github.com/spf13/cobra"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
-	"k8s.io/apimachinery/pkg/labels"
-	"k8s.io/apimachinery/pkg/runtime"
-	"k8s.io/apimachinery/pkg/runtime/schema"
+	"k8s.io/apimachinery/pkg/watch"
 	"k8s.io/cli-runtime/pkg/genericclioptions"
 	"k8s.io/klog/v2"
 
@@ -21,19 +20,18 @@ import (
 	"example.com/monok8s/pkg/templates"
 )
 
-const defaultPollInterval = 15 * time.Second
-
-var runtimeDefaultUnstructuredConverter = runtime.DefaultUnstructuredConverter
-
 func NewCmdAgent(flags *genericclioptions.ConfigFlags) *cobra.Command {
-	var namespace string
 	var envFile string
-	var pollInterval time.Duration
 
 	cmd := &cobra.Command{
 		Use:   "agent --env-file path",
-		Short: "Watch OSUpgrade resources and process matching upgrades for this node",
+		Short: "Watch OSUpgradeProgress resources for this node and process upgrades",
 		RunE: func(cmd *cobra.Command, _ []string) error {
+			ns, _, err := flags.ToRawKubeConfigLoader().Namespace()
+			if err != nil {
+				return err
+			}
+
 			if envFile == "" {
 				return fmt.Errorf("--env-file is required")
 			}
@@ -50,11 +48,15 @@ func NewCmdAgent(flags *genericclioptions.ConfigFlags) *cobra.Command {
 				return fmt.Errorf("node name is empty in rendered config")
 			}
 
+			ctx := cmd.Context()
+			if err := waitForControlGate(ctx, envFile, 2*time.Second); err != nil {
+				return fmt.Errorf("wait for control gate to release: %w", err)
+			}
+
 			klog.InfoS("starting agent",
 				"node", cfg.Spec.NodeName,
-				"namespace", namespace,
+				"namespace", ns,
 				"envFile", envFile,
-				"pollInterval", pollInterval,
 			)
 
 			clients, err := kube.NewClients(flags)
@@ -62,31 +64,35 @@ func NewCmdAgent(flags *genericclioptions.ConfigFlags) *cobra.Command {
 				return fmt.Errorf("create kube clients: %w", err)
 			}
 
-			ctx := cmd.Context()
-			return runPollLoop(ctx, clients, namespace, cfg.Spec.NodeName, pollInterval)
+			return runWatchLoop(ctx, clients, ns, cfg.Spec.NodeName)
 		},
 	}
 
-	cmd.Flags().StringVar(&namespace, "namespace", "kube-system", "namespace to watch")
 	cmd.Flags().StringVar(&envFile, "env-file", "", "path to env file containing MKS_* variables")
-	cmd.Flags().DurationVar(&pollInterval, "poll-interval", defaultPollInterval, "poll interval for OSUpgrade resources")
 
 	return cmd
 }
 
-func runPollLoop(ctx context.Context, clients *kube.Clients, namespace, nodeName string, interval time.Duration) error {
-	gvr := schema.GroupVersionResource{
-		Group:    monov1alpha1.Group,
-		Version:  monov1alpha1.Version,
-		Resource: "osupgrades",
+func waitForControlGate(ctx context.Context, envFile string, pollInterval time.Duration) error {
+	dir := filepath.Dir(envFile)
+	marker := filepath.Join(dir, ".control-gate")
+
+	if pollInterval <= 0 {
+		pollInterval = 2 * time.Second
 	}
 
-	ticker := time.NewTicker(interval)
+	ticker := time.NewTicker(pollInterval)
 	defer ticker.Stop()
 
 	for {
-		if err := pollOnce(ctx, clients, gvr, namespace, nodeName); err != nil {
-			klog.ErrorS(err, "poll failed", "namespace", namespace, "node", nodeName)
+		_, err := os.Stat(marker)
+		if err == nil {
+			klog.InfoS("Control gate is present; waiting before starting watch loop", "path", marker)
+		} else if os.IsNotExist(err) {
+			klog.InfoS("Control gate not present; starting watch loop", "path", marker)
+			return nil
+		} else {
+			return fmt.Errorf("stat upgrade marker %s: %w", marker, err)
 		}
 
 		select {
@@ -97,105 +103,150 @@ func runPollLoop(ctx context.Context, clients *kube.Clients, namespace, nodeName
 	}
 }
 
-func pollOnce(
+func runWatchLoop(ctx context.Context, clients *kube.Clients, namespace, nodeName string) error {
+	var resourceVersion string
+
+	for {
+		if ctx.Err() != nil {
+			return ctx.Err()
+		}
+
+		err := watchOnce(ctx, clients, namespace, nodeName, &resourceVersion)
+		if err != nil {
+			if ctx.Err() != nil {
+				return ctx.Err()
+			}
+			klog.ErrorS(err, "watch failed; retrying",
+				"namespace", namespace,
+				"node", nodeName,
+				"resourceVersion", resourceVersion,
+			)
+			select {
+			case <-ctx.Done():
+				return ctx.Err()
+			case <-time.After(2 * time.Second):
+			}
+			continue
+		}
+	}
+}
+
+func watchOnce(
 	ctx context.Context,
 	clients *kube.Clients,
-	gvr schema.GroupVersionResource,
 	namespace string,
 	nodeName string,
+	resourceVersion *string,
 ) error {
-	list, err := clients.Dynamic.Resource(gvr).Namespace(namespace).List(ctx, metav1.ListOptions{})
+
+	list, err := clients.MonoKS.
+		Monok8sV1alpha1().
+		OSUpgradeProgresses(namespace).
+		List(ctx, metav1.ListOptions{})
 	if err != nil {
-		return fmt.Errorf("list osupgrades: %w", err)
-	}
-
-	klog.InfoS("agent tick", "namespace", namespace, "items", len(list.Items), "node", nodeName)
-
-	nodeLabels := labels.Set{
-		"kubernetes.io/hostname":   nodeName,
-		"monok8s.io/node-name":     nodeName,
-		"monok8s.io/control-agent": "true",
+		return fmt.Errorf("list osupgradeprogresses: %w", err)
 	}
 
 	for i := range list.Items {
 		item := &list.Items[i]
-
-		osu, err := decodeOSUpgrade(item)
-		if err != nil {
-			klog.ErrorS(err, "failed to decode osupgrade",
-				"name", item.GetName(),
-				"resourceVersion", item.GetResourceVersion(),
-			)
+		if !targetsNode(item, nodeName) {
 			continue
 		}
 
-		if !matchesNode(osu, nodeName, nodeLabels) {
-			klog.V(2).InfoS("skipping osupgrade; not targeted to this node",
-				"name", osu.Name,
-				"node", nodeName,
-			)
-			continue
-		}
-
-		klog.InfoS("matched osupgrade",
-			"name", osu.Name,
+		klog.InfoS("found existing osupgradeprogress",
+			"name", item.Name,
 			"node", nodeName,
-			"desiredVersion", osu.Spec.DesiredVersion,
-			"phase", statusPhase(osu.Status),
-			"resourceVersion", osu.ResourceVersion,
+			"phase", item.StatusPhase(),
+			"resourceVersion", item.ResourceVersion,
 		)
 
-		if err := osupgradeController.HandleOSUpgrade(ctx, clients, namespace, nodeName, osu); err != nil {
-			klog.ErrorS(err, "failed to handle osupgrade",
-				"name", osu.Name,
+		if err := osupgradeController.HandleOSUpgradeProgress(ctx, clients, namespace, nodeName, item); err != nil {
+			klog.ErrorS(err, "failed to handle existing osupgradeprogress",
+				"name", item.Name,
 				"node", nodeName,
 			)
-			continue
 		}
 	}
 
-	return nil
-}
+	*resourceVersion = list.ResourceVersion
 
-func decodeOSUpgrade(item *unstructured.Unstructured) (*monov1alpha1.OSUpgrade, error) {
-	var osu monov1alpha1.OSUpgrade
-	if err := runtimeDefaultUnstructuredConverter.FromUnstructured(item.Object, &osu); err != nil {
-		return nil, fmt.Errorf("convert unstructured to OSUpgrade: %w", err)
-	}
-	return &osu, nil
-}
-
-func matchesNode(osu *monov1alpha1.OSUpgrade, nodeName string, nodeLabels labels.Set) bool {
-	if osu == nil {
-		return false
-	}
-
-	sel := osu.Spec.NodeSelector
-	if sel == nil {
-		// No selector means "match all nodes".
-		return true
-	}
-
-	selector, err := metav1.LabelSelectorAsSelector(sel)
+	w, err := clients.MonoKS.
+		Monok8sV1alpha1().
+		OSUpgradeProgresses(namespace).
+		Watch(ctx, metav1.ListOptions{
+			ResourceVersion: *resourceVersion,
+		})
 	if err != nil {
-		klog.ErrorS(err, "invalid node selector on osupgrade", "name", osu.Name)
+		return fmt.Errorf("watch osupgradeprogresses: %w", err)
+	}
+	defer w.Stop()
+
+	klog.InfoS("watching osupgradeprogresses",
+		"namespace", namespace,
+		"node", nodeName,
+		"resourceVersion", *resourceVersion,
+	)
+
+	for {
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+
+		case evt, ok := <-w.ResultChan():
+			if !ok {
+				return fmt.Errorf("watch channel closed")
+			}
+
+			switch evt.Type {
+			case watch.Bookmark:
+				obj, ok := evt.Object.(*monov1alpha1.OSUpgradeProgress)
+				if ok && obj != nil && obj.ResourceVersion != "" {
+					*resourceVersion = obj.ResourceVersion
+				}
+				continue
+
+			case watch.Error:
+				return fmt.Errorf("watch returned error event")
+			}
+
+			osup, ok := evt.Object.(*monov1alpha1.OSUpgradeProgress)
+			if !ok {
+				klog.V(1).InfoS("skipping unexpected watch object type",
+					"type", fmt.Sprintf("%T", evt.Object),
+				)
+				continue
+			}
+
+			if osup.ResourceVersion != "" {
+				*resourceVersion = osup.ResourceVersion
+			}
+
+			if !targetsNode(osup, nodeName) {
+				continue
+			}
+
+			klog.V(4).InfoS("received osupgradeprogress event",
+				"name", osup.Name,
+				"node", nodeName,
+				"phase", osup.StatusPhase(),
+				"eventType", evt.Type,
+				"resourceVersion", osup.ResourceVersion,
+			)
+
+			if err := osupgradeController.HandleOSUpgradeProgress(ctx, clients, namespace, nodeName, osup); err != nil {
+				klog.ErrorS(err, "failed to handle osupgradeprogress",
+					"name", osup.Name,
+					"node", nodeName,
+					"eventType", evt.Type,
+				)
+			}
+		}
+	}
+}
+
+func targetsNode(osup *monov1alpha1.OSUpgradeProgress, nodeName string) bool {
+	if osup == nil {
 		return false
 	}
-
-	if selector.Empty() {
-		return true
-	}
-
-	if selector.Matches(nodeLabels) {
-		return true
-	}
-
-	return false
-}
-
-func statusPhase(st *monov1alpha1.OSUpgradeStatus) string {
-	if st == nil {
-		return ""
-	}
-	return string(st.Phase)
+	return osup.Spec.NodeName == nodeName
 }

clitools/pkg/cmd/controller/controller.go

@@ -0,0 +1,217 @@
+package controller
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"net"
+	"net/http"
+	"os"
+	"time"
+
+	"github.com/spf13/cobra"
+	"k8s.io/cli-runtime/pkg/genericclioptions"
+	"k8s.io/klog/v2"
+
+	mkscontroller "example.com/monok8s/pkg/controller"
+	osupgradectrl "example.com/monok8s/pkg/controller/osupgrade"
+	"example.com/monok8s/pkg/kube"
+)
+
+type ServerConfig struct {
+	Namespace         string `json:"namespace,omitempty"`
+	TLSCertFile       string `json:"tlsCertFile,omitempty"`
+	TLSPrivateKeyFile string `json:"tlsPrivateKeyFile,omitempty"`
+}
+
+func NewCmdController(flags *genericclioptions.ConfigFlags) *cobra.Command {
+	var conf ServerConfig
+
+	cmd := &cobra.Command{
+		Use:   "controller",
+		Short: "Start a controller that handles OSUpgrade resources",
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			ns, _, err := flags.ToRawKubeConfigLoader().Namespace()
+			if err != nil {
+				return err
+			}
+			conf.Namespace = ns
+
+			ctx := cmd.Context()
+
+			klog.InfoS("starting controller", "namespace", conf.Namespace)
+
+			clients, err := kube.NewClients(flags)
+			if err != nil {
+				return err
+			}
+
+			ctx, cancel := context.WithCancel(ctx)
+			defer cancel()
+
+			httpErrCh := make(chan error, 1)
+			watchErrCh := make(chan error, 1)
+
+			go func() {
+				klog.InfoS("starting OSUpgrade watch loop", "namespace", conf.Namespace)
+				watchErrCh <- osupgradectrl.Watch(ctx, clients, conf.Namespace)
+			}()
+
+			go func() {
+				httpErrCh <- listenAndServe(ctx, clients, conf)
+			}()
+
+			select {
+			case <-ctx.Done():
+				klog.InfoS("controller context canceled")
+				return ctx.Err()
+
+			case err := <-watchErrCh:
+				if err != nil && !errors.Is(err, context.Canceled) {
+					cancel()
+					return err
+				}
+				cancel()
+				return nil
+
+			case err := <-httpErrCh:
+				if err != nil && !errors.Is(err, context.Canceled) {
+					cancel()
+					return err
+				}
+				cancel()
+				return nil
+			}
+		},
+	}
+
+	cmd.Flags().StringVar(&conf.TLSCertFile, "tls-cert-file", conf.TLSCertFile,
+		"File containing x509 Certificate used for serving HTTPS (with intermediate certs, if any, concatenated after server cert).")
+	cmd.Flags().StringVar(&conf.TLSPrivateKeyFile, "tls-private-key-file", conf.TLSPrivateKeyFile,
+		"File containing x509 private key matching --tls-cert-file.")
+
+	return cmd
+}
+
+func listenAndServe(ctx context.Context, clients *kube.Clients, conf ServerConfig) error {
+	nodeName := os.Getenv("NODE_NAME")
+
+	controllerServer := mkscontroller.NewServer(ctx, clients, conf.Namespace, nodeName)
+
+	healthMux := http.NewServeMux()
+	healthMux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("ok\n"))
+	})
+	healthMux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("ok\n"))
+	})
+
+	healthAddr := net.JoinHostPort("", "8080")
+	controllerAddr := net.JoinHostPort("", "8443")
+
+	healthHTTPServer := &http.Server{
+		Addr:           healthAddr,
+		Handler:        healthMux,
+		IdleTimeout:    90 * time.Second,
+		ReadTimeout:    10 * time.Second,
+		WriteTimeout:   10 * time.Second,
+		MaxHeaderBytes: 1 << 20,
+	}
+
+	controllerHTTPServer := &http.Server{
+		Addr:           controllerAddr,
+		Handler:        controllerServer,
+		IdleTimeout:    90 * time.Second,
+		ReadTimeout:    4 * time.Minute,
+		WriteTimeout:   4 * time.Minute,
+		MaxHeaderBytes: 1 << 20,
+	}
+
+	serverErrCh := make(chan error, 2)
+
+	go func() {
+		klog.InfoS("starting health HTTP server", "addr", healthAddr)
+
+		err := healthHTTPServer.ListenAndServe()
+		if err != nil && !errors.Is(err, http.ErrServerClosed) {
+			serverErrCh <- fmt.Errorf("health HTTP server: %w", err)
+			return
+		}
+
+		serverErrCh <- nil
+	}()
+
+	go func() {
+		if conf.TLSCertFile != "" {
+			klog.InfoS("starting controller HTTPS server",
+				"addr", controllerAddr,
+				"certFile", conf.TLSCertFile,
+				"keyFile", conf.TLSPrivateKeyFile,
+			)
+
+			err := controllerHTTPServer.ListenAndServeTLS(conf.TLSCertFile, conf.TLSPrivateKeyFile)
+			if err != nil && !errors.Is(err, http.ErrServerClosed) {
+				serverErrCh <- fmt.Errorf("controller HTTPS server: %w", err)
+				return
+			}
+
+			serverErrCh <- nil
+			return
+		}
+
+		klog.InfoS("starting controller HTTP server", "addr", controllerAddr)
+
+		err := controllerHTTPServer.ListenAndServe()
+		if err != nil && !errors.Is(err, http.ErrServerClosed) {
+			serverErrCh <- fmt.Errorf("controller HTTP server: %w", err)
+			return
+		}
+
+		serverErrCh <- nil
+	}()
+
+	select {
+	case <-ctx.Done():
+		klog.InfoS("shutting down HTTP servers",
+			"healthAddr", healthAddr,
+			"controllerAddr", controllerAddr,
+		)
+
+		shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+		defer cancel()
+
+		var errs []error
+
+		if err := healthHTTPServer.Shutdown(shutdownCtx); err != nil {
+			errs = append(errs, fmt.Errorf("shutdown health HTTP server: %w", err))
+		}
+
+		if err := controllerHTTPServer.Shutdown(shutdownCtx); err != nil {
+			errs = append(errs, fmt.Errorf("shutdown controller HTTP server: %w", err))
+		}
+
+		for i := 0; i < 2; i++ {
+			if err := <-serverErrCh; err != nil {
+				errs = append(errs, err)
+			}
+		}
+
+		if len(errs) > 0 {
+			return errors.Join(errs...)
+		}
+
+		return context.Canceled
+
+	case err := <-serverErrCh:
+		if err != nil {
+			klog.ErrorS(err, "HTTP server failed")
+			return err
+		}
+
+		// One server exited cleanly unexpectedly. Treat that as failure because
+		// the process should keep both servers alive until ctx is canceled.
+		return fmt.Errorf("HTTP server exited unexpectedly")
+	}
+}

clitools/pkg/cmd/create/create.go

@@ -1,13 +1,18 @@
 package create
 
 import (
+	"bytes"
 	"fmt"
 	"github.com/spf13/cobra"
+	"k8s.io/cli-runtime/pkg/genericclioptions"
+	"os"
+	"strings"
 
+	assets "example.com/monok8s/pkg/assets"
 	render "example.com/monok8s/pkg/render"
 )
 
-func NewCmdCreate() *cobra.Command {
+func NewCmdCreate(flags *genericclioptions.ConfigFlags) *cobra.Command {
 	cmd := &cobra.Command{Use: "create", Short: "Create starter resources"}
 	cmd.AddCommand(
 		&cobra.Command{
@@ -26,7 +31,12 @@ func NewCmdCreate() *cobra.Command {
 			Use:   "osupgrade",
 			Short: "Print an OSUpgrade template",
 			RunE: func(cmd *cobra.Command, _ []string) error {
-				out, err := render.RenderOSUpgrade()
+				ns, _, err := flags.ToRawKubeConfigLoader().Namespace()
+				if err != nil {
+					return err
+				}
+
+				out, err := render.RenderOSUpgrade(ns)
 				if err != nil {
 					return err
 				}
@@ -34,6 +44,143 @@ func NewCmdCreate() *cobra.Command {
 				return err
 			},
 		},
+		&cobra.Command{
+			Use:   "crds",
+			Short: "Print the bundled CRDs",
+			RunE: func(cmd *cobra.Command, _ []string) error {
+				return assets.PrintCRDs(cmd.OutOrStdout())
+			},
+		},
 	)
+
+	var authorizedKeysPath string
+
+	sshdcmd := cobra.Command{
+		Use:   "sshd",
+		Short: "Print sshd deployments template",
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			ns, _, err := flags.ToRawKubeConfigLoader().Namespace()
+			if err != nil {
+				return err
+			}
+
+			authorizedKeys, err := readAuthorizedKeysFile(authorizedKeysPath)
+			if err != nil {
+				return err
+			}
+
+			out, err := render.RenderSSHDDeployments(ns, authorizedKeys)
+			if err != nil {
+				return err
+			}
+
+			_, err = fmt.Fprint(cmd.OutOrStdout(), out)
+			return err
+		},
+	}
+
+	sshdcmd.Flags().StringVar(&authorizedKeysPath, "authkeys", "", "path to authorized_keys file")
+
+	cmd.AddCommand(&sshdcmd)
+
+	cconf := render.ControllerConf{}
+	controllercmd := cobra.Command{
+		Use:   "controller",
+		Short: "Print controller deployments template",
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if len(cconf.ImagePullSecrets) > 0 && strings.TrimSpace(cconf.Image) == "" {
+				return fmt.Errorf("--image-pull-secret requires --image")
+			}
+
+			ns, _, err := flags.ToRawKubeConfigLoader().Namespace()
+			if err != nil {
+				return err
+			}
+
+			cconf.Namespace = ns
+
+			out, err := render.RenderControllerDeployments(cconf)
+			if err != nil {
+				return err
+			}
+
+			_, err = fmt.Fprint(cmd.OutOrStdout(), out)
+			return err
+		},
+	}
+
+	controllercmd.Flags().StringVar(
+		&cconf.Image,
+		"image",
+		"",
+		"Controller image, including optional registry and tag",
+	)
+	controllercmd.Flags().StringSliceVar(
+		&cconf.ImagePullSecrets,
+		"image-pull-secret",
+		nil,
+		"Image pull secret name for the agent image; may be specified multiple times or as a comma-separated list",
+	)
+
+	cmd.AddCommand(&controllercmd)
+
+	aconf := render.AgentConf{}
+	agentcmd := cobra.Command{
+		Use:   "agent",
+		Short: "Print agent daemonsets template",
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if len(aconf.ImagePullSecrets) > 0 && strings.TrimSpace(aconf.Image) == "" {
+				return fmt.Errorf("--image-pull-secret requires --image")
+			}
+
+			ns, _, err := flags.ToRawKubeConfigLoader().Namespace()
+			if err != nil {
+				return err
+			}
+
+			aconf.Namespace = ns
+
+			out, err := render.RenderAgentDaemonSets(aconf)
+			if err != nil {
+				return err
+			}
+
+			_, err = fmt.Fprint(cmd.OutOrStdout(), out)
+			return err
+		},
+	}
+
+	agentcmd.Flags().StringVar(
+		&aconf.Image,
+		"image",
+		"",
+		"Agent image, including optional registry and tag",
+	)
+	agentcmd.Flags().StringSliceVar(
+		&aconf.ImagePullSecrets,
+		"image-pull-secret",
+		nil,
+		"Image pull secret name for the agent image; may be specified multiple times or as a comma-separated list",
+	)
+
+	cmd.AddCommand(&agentcmd)
+
 	return cmd
 }
+
+func readAuthorizedKeysFile(path string) (string, error) {
+	if path == "" {
+		return "", fmt.Errorf("--authkeys is required")
+	}
+
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return "", fmt.Errorf("read authorized_keys file %q: %w", path, err)
+	}
+
+	if len(bytes.TrimSpace(b)) == 0 {
+		return "", fmt.Errorf("authorized_keys file %q is empty", path)
+	}
+
+	return string(b), nil
+}

clitools/pkg/cmd/root/root.go

@@ -6,6 +6,7 @@ import (
 
 	agentcmd "example.com/monok8s/pkg/cmd/agent"
 	checkconfigcmd "example.com/monok8s/pkg/cmd/checkconfig"
+	controllercmd "example.com/monok8s/pkg/cmd/controller"
 	createcmd "example.com/monok8s/pkg/cmd/create"
 	initcmd "example.com/monok8s/pkg/cmd/initcmd"
 	internalcmd "example.com/monok8s/pkg/cmd/internal"
@@ -18,8 +19,10 @@ import (
 func init() {
 	klog.InitFlags(nil)
 
+	_ = flag.Set("logtostderr", "true")
+
 	if os.Getenv("DEBUG") != "" {
-		_ = flag.Set("v", "4") // debug level
+		_ = flag.Set("v", "4")
 	} else {
 		_ = flag.Set("v", "0")
 	}
@@ -38,14 +41,20 @@ func NewRootCmd() *cobra.Command {
 		},
 	}
 
+	// Expose klog stdlib flags through Cobra/pflag.
+	cmd.PersistentFlags().AddGoFlagSet(flag.CommandLine)
+
 	flags.AddFlags(cmd.PersistentFlags())
+
 	cmd.AddCommand(
 		versioncmd.NewCmdVersion(),
 		initcmd.NewCmdInit(flags),
 		checkconfigcmd.NewCmdCheckConfig(),
-		createcmd.NewCmdCreate(),
+		createcmd.NewCmdCreate(flags),
 		agentcmd.NewCmdAgent(flags),
+		controllercmd.NewCmdController(flags),
 		internalcmd.NewCmdInternal(),
 	)
+
 	return cmd
 }

clitools/pkg/cmd/version/version.go

@@ -1,22 +1,76 @@
-package apply
+package version
 
 import (
+	"encoding/json"
 	"fmt"
 
 	"github.com/spf13/cobra"
 
-	buildInfo "example.com/monok8s/pkg/buildinfo"
+	buildinfo "example.com/monok8s/pkg/buildinfo"
 )
 
+type versionInfo struct {
+	Version     string `json:"version"`
+	GitRevision string `json:"gitRevision"`
+	Timestamp   string `json:"timestamp"`
+	KubeVersion string `json:"kubernetesVersion"`
+}
+
 func NewCmdVersion() *cobra.Command {
+	var (
+		shortOutput      bool
+		jsonOutput       bool
+		kubernetesOutput bool
+	)
+
 	cmd := &cobra.Command{
 		Use:   "version",
-		Short: "Print the version information",
+		Short: "Print version information",
+		Args:  cobra.NoArgs,
 		RunE: func(cmd *cobra.Command, _ []string) error {
+			info := versionInfo{
+				Version:     buildinfo.Version,
+				GitRevision: buildinfo.GitRevision,
+				Timestamp:   buildinfo.Timestamp,
+				KubeVersion: buildinfo.KubeVersion,
+			}
 
-			_, err := fmt.Fprintln(cmd.OutOrStdout(), fmt.Sprintf("%s %s(%s)", buildInfo.Version, buildInfo.GitRevision, buildInfo.Timestamp))
-			return err
+			out := cmd.OutOrStdout()
+
+			switch {
+			case jsonOutput:
+				enc := json.NewEncoder(out)
+				enc.SetIndent("", "  ")
+				return enc.Encode(info)
+
+			case kubernetesOutput:
+				_, err := fmt.Fprintln(out, info.KubeVersion)
+				return err
+
+			case shortOutput:
+				_, err := fmt.Fprintln(out, info.Version)
+				return err
+
+			default:
+				_, err := fmt.Fprintf(
+					out,
+					"Version:    %s\nGit commit: %s\nBuilt at:   %s\nKubernetes: %s\n",
+					info.Version,
+					info.GitRevision,
+					info.Timestamp,
+					info.KubeVersion,
+				)
+				return err
+			}
 		},
 	}
+
+	flags := cmd.Flags()
+	flags.BoolVar(&shortOutput, "short", false, "Show only the application version")
+	flags.BoolVar(&jsonOutput, "json", false, "Show version information as JSON")
+	flags.BoolVarP(&kubernetesOutput, "kubernetes", "k", false, "Show only the Kubernetes version this binary was built for")
+
+	cmd.MarkFlagsMutuallyExclusive("short", "json", "kubernetes")
+
 	return cmd
 }

clitools/pkg/controller/osimage/helpers.go

@@ -1,5 +1,20 @@
 package osimage
 
+import (
+	"fmt"
+	"os"
+	"strings"
+	"sync"
+
+	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+)
+
+var (
+	bootStateOnce sync.Once
+	bootState     map[string]string
+	bootStateErr  error
+)
+
 func PercentOf(done, total int64) int64 {
 	if total <= 0 {
 		return 0
@@ -13,3 +28,34 @@ func PercentOf(done, total int64) int64 {
 	}
 	return p
 }
+
+func ReadBootState() (map[string]string, error) {
+	bootStateOnce.Do(func() {
+		data, err := os.ReadFile(monov1alpha1.BootStateFile)
+		if err != nil {
+			bootStateErr = err
+			return
+		}
+
+		out := make(map[string]string)
+
+		for _, line := range strings.Split(string(data), "\n") {
+			line = strings.TrimSpace(line)
+			if line == "" || strings.HasPrefix(line, "#") {
+				continue
+			}
+
+			k, v, ok := strings.Cut(line, "=")
+			if !ok {
+				bootStateErr = fmt.Errorf("invalid line: %q", line)
+				return
+			}
+
+			out[strings.TrimSpace(k)] = strings.TrimSpace(v)
+		}
+
+		bootState = out
+	})
+
+	return bootState, bootStateErr
+}

clitools/pkg/controller/osimage/progress.go

@@ -2,6 +2,7 @@ package osimage
 
 import (
 	"k8s.io/klog/v2"
+	"sync"
 	"time"
 )
 
@@ -92,8 +93,10 @@ func (l *ProgressLogger) Log(p Progress) {
 }
 
 type TimeBasedUpdater struct {
+	mu       sync.Mutex
 	interval time.Duration
 	lastRun  time.Time
+	inFlight bool
 }
 
 func NewTimeBasedUpdater(seconds int) *TimeBasedUpdater {
@@ -106,16 +109,28 @@ func NewTimeBasedUpdater(seconds int) *TimeBasedUpdater {
 }
 
 func (u *TimeBasedUpdater) Run(fn func() error) error {
+	u.mu.Lock()
 	now := time.Now()
 
-	if !u.lastRun.IsZero() && now.Sub(u.lastRun) < u.interval {
+	if u.inFlight {
+		u.mu.Unlock()
 		return nil
 	}
 
-	if err := fn(); err != nil {
-		return err
+	if !u.lastRun.IsZero() && now.Sub(u.lastRun) < u.interval {
+		u.mu.Unlock()
+		return nil
 	}
 
 	u.lastRun = now
-	return nil
+	u.inFlight = true
+	u.mu.Unlock()
+
+	defer func() {
+		u.mu.Lock()
+		u.inFlight = false
+		u.mu.Unlock()
+	}()
+
+	return fn()
 }

clitools/pkg/controller/osimage/write.go

@@ -5,11 +5,30 @@ import (
 	"fmt"
 	"io"
 	"os"
+	"time"
+)
+
+const (
+	defaultWriteBufferSize = 1 * 1024 * 1024
+
+	defaultMinWriteBPS     = int64(2 * 1024 * 1024)
+	defaultInitialWriteBPS = int64(4 * 1024 * 1024)
+	defaultMaxWriteBPS     = int64(8 * 1024 * 1024)
+	defaultBurstBytes      = int64(512 * 1024)
+
+	defaultSampleInterval = 250 * time.Millisecond
+
+	defaultSyncEveryBytes = 0
+
+	defaultBusyHighPct = 80.0
+	defaultBusyLowPct  = 40.0
+
+	defaultSlowAwait = 20 * time.Millisecond
+	defaultFastAwait = 5 * time.Millisecond
 )
 
 func WriteStreamToTarget(ctx context.Context,
-	src io.Reader,
-	targetPath string,
+	src io.Reader, targetPath string,
 	expectedSize int64, bufferSize int,
 	progress ProgressFunc,
 ) (int64, error) {
@@ -17,7 +36,7 @@ func WriteStreamToTarget(ctx context.Context,
 		return 0, fmt.Errorf("target path is required")
 	}
 	if bufferSize <= 0 {
-		bufferSize = 4 * 1024 * 1024
+		bufferSize = defaultWriteBufferSize
 	}
 
 	f, err := os.OpenFile(targetPath, os.O_WRONLY, 0)
@@ -26,7 +45,22 @@ func WriteStreamToTarget(ctx context.Context,
 	}
 	defer f.Close()
 
-	written, err := copyWithProgressBuffer(ctx, f, src, expectedSize, "flash", progress, make([]byte, bufferSize))
+	ctrl, err := newAdaptiveWriteController(targetPath)
+	if err != nil {
+		ctrl = newNoopAdaptiveWriteController()
+	}
+
+	written, err := copyWithProgressBuffer(
+		ctx,
+		f,
+		src,
+		expectedSize,
+		"flash",
+		progress,
+		make([]byte, bufferSize),
+		ctrl,
+		defaultSyncEveryBytes,
+	)
 	if err != nil {
 		return written, err
 	}
@@ -42,8 +76,19 @@ func WriteStreamToTarget(ctx context.Context,
 	return written, nil
 }
 
-func copyWithProgressBuffer(ctx context.Context, dst io.Writer, src io.Reader, total int64, stage string, progress ProgressFunc, buf []byte) (int64, error) {
+func copyWithProgressBuffer(
+	ctx context.Context,
+	dst *os.File,
+	src io.Reader,
+	total int64,
+	stage string,
+	progress ProgressFunc,
+	buf []byte,
+	ctrl *adaptiveWriteController,
+	syncEvery int64,
+) (int64, error) {
 	var written int64
+	var sinceSync int64
 
 	for {
 		select {
@@ -54,9 +99,21 @@ func copyWithProgressBuffer(ctx context.Context, dst io.Writer, src io.Reader, t
 
 		nr, er := src.Read(buf)
 		if nr > 0 {
+			if ctrl != nil {
+				if err := ctrl.Wait(ctx, nr); err != nil {
+					return written, err
+				}
+			}
+
 			nw, ew := dst.Write(buf[:nr])
 			if nw > 0 {
 				written += int64(nw)
+				sinceSync += int64(nw)
+
+				if ctrl != nil {
+					ctrl.ObserveWrite(nw)
+				}
+
 				if progress != nil {
 					progress(Progress{
 						Stage:         stage,
@@ -64,7 +121,19 @@ func copyWithProgressBuffer(ctx context.Context, dst io.Writer, src io.Reader, t
 						BytesTotal:    total,
 					})
 				}
+
+				if syncEvery > 0 && sinceSync >= syncEvery {
+					if err := dst.Sync(); err != nil {
+						return written, fmt.Errorf("periodic sync target: %w", err)
+					}
+					sinceSync = 0
+
+					if ctrl != nil {
+						ctrl.ObserveSync()
+					}
+				}
 			}
+
 			if ew != nil {
 				return written, ew
 			}
@@ -72,6 +141,7 @@ func copyWithProgressBuffer(ctx context.Context, dst io.Writer, src io.Reader, t
 				return written, io.ErrShortWrite
 			}
 		}
+
 		if er != nil {
 			if er == io.EOF {
 				return written, nil

clitools/pkg/controller/osimage/write_diag.go

@@ -0,0 +1,106 @@
+package osimage
+
+import (
+	"context"
+	"fmt"
+	"io"
+	"time"
+
+	"k8s.io/klog/v2"
+)
+
+type repeatPatternReader struct {
+	pattern []byte
+	remain  int64
+	off     int
+}
+
+func newRepeatPatternReader(total int64, pattern []byte) *repeatPatternReader {
+	if len(pattern) == 0 {
+		pattern = []byte("monok8s-test-pattern-0123456789abcdef")
+	}
+	return &repeatPatternReader{
+		pattern: pattern,
+		remain:  total,
+	}
+}
+
+func (r *repeatPatternReader) Read(p []byte) (int, error) {
+	if r.remain <= 0 {
+		return 0, io.EOF
+	}
+
+	if int64(len(p)) > r.remain {
+		p = p[:r.remain]
+	}
+
+	n := 0
+	for n < len(p) {
+		copied := copy(p[n:], r.pattern[r.off:])
+		n += copied
+		r.off += copied
+		if r.off == len(r.pattern) {
+			r.off = 0
+		}
+	}
+
+	r.remain -= int64(n)
+	return n, nil
+}
+
+func TestStreamToTarget(ctx context.Context, targetPath string) error {
+	const (
+		totalSize  = int64(512 * 1024 * 1024) // 512 MiB
+		bufferSize = 128 * 1024               // test the conservative setting
+	)
+
+	src := newRepeatPatternReader(totalSize, nil)
+
+	start := time.Now()
+	lastLog := start
+
+	progress := func(p Progress) {
+		now := time.Now()
+		if now.Sub(lastLog) < 1*time.Second && p.BytesComplete != p.BytesTotal {
+			return
+		}
+		lastLog = now
+
+		var mbps float64
+		elapsed := now.Sub(start).Seconds()
+		if elapsed > 0 {
+			mbps = float64(p.BytesComplete) / 1024.0 / 1024.0 / elapsed
+		}
+
+		klog.InfoS("test write progress",
+			"stage", p.Stage,
+			"bytesComplete", p.BytesComplete,
+			"bytesTotal", p.BytesTotal,
+			"mbpsAvg", fmt.Sprintf("%.2f", mbps),
+		)
+	}
+
+	written, err := WriteStreamToTarget(
+		ctx,
+		src,
+		targetPath,
+		totalSize,
+		bufferSize,
+		progress,
+	)
+	if err != nil {
+		return fmt.Errorf("write stream to target: %w", err)
+	}
+
+	elapsed := time.Since(start)
+	mbps := float64(written) / 1024.0 / 1024.0 / elapsed.Seconds()
+
+	klog.InfoS("test write complete",
+		"targetPath", targetPath,
+		"written", written,
+		"elapsed", elapsed.String(),
+		"mbpsAvg", fmt.Sprintf("%.2f", mbps),
+	)
+
+	return nil
+}

clitools/pkg/controller/osimage/write_throttle_linux.go

@@ -0,0 +1,400 @@
+//go:build linux
+
+package osimage
+
+import (
+	"bufio"
+	"context"
+	"fmt"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"golang.org/x/sys/unix"
+)
+
+type adaptiveWriteController struct {
+	mu sync.Mutex
+
+	limiter *rateLimiter
+	monitor *diskBusyMonitor
+
+	sampleInterval time.Duration
+	nextSampleAt   time.Time
+
+	minBPS int64
+	maxBPS int64
+
+	busyHighPct float64
+	busyLowPct  float64
+}
+
+func newAdaptiveWriteController(targetPath string) (*adaptiveWriteController, error) {
+	mon, err := newDiskBusyMonitor(targetPath)
+	if err != nil {
+		return nil, err
+	}
+
+	now := time.Now()
+	return &adaptiveWriteController{
+		limiter: newRateLimiter(defaultInitialWriteBPS, defaultBurstBytes),
+		monitor: mon,
+
+		sampleInterval: defaultSampleInterval,
+		nextSampleAt:   now.Add(defaultSampleInterval),
+
+		minBPS: defaultMinWriteBPS,
+		maxBPS: defaultMaxWriteBPS,
+
+		busyHighPct: defaultBusyHighPct,
+		busyLowPct:  defaultBusyLowPct,
+	}, nil
+}
+
+func newNoopAdaptiveWriteController() *adaptiveWriteController {
+	return &adaptiveWriteController{
+		limiter:        newRateLimiter(0, 0),
+		sampleInterval: defaultSampleInterval,
+	}
+}
+
+func (c *adaptiveWriteController) Wait(ctx context.Context, n int) error {
+	if c == nil || c.limiter == nil {
+		return nil
+	}
+	return c.limiter.Wait(ctx, n)
+}
+
+func (c *adaptiveWriteController) ObserveWrite(n int) {
+	c.observe(false)
+}
+
+func (c *adaptiveWriteController) ObserveSync() {
+	c.observe(true)
+}
+
+func (c *adaptiveWriteController) observe(afterSync bool) {
+	if c == nil {
+		return
+	}
+
+	c.mu.Lock()
+	defer c.mu.Unlock()
+
+	now := time.Now()
+	if c.monitor == nil || now.Before(c.nextSampleAt) {
+		return
+	}
+	c.nextSampleAt = now.Add(c.sampleInterval)
+
+	s, err := c.monitor.Sample(now)
+	if err != nil {
+		return
+	}
+
+	cur := c.limiter.Rate()
+	if cur <= 0 {
+		cur = c.minBPS
+	}
+
+	switch {
+	case s.UtilPct >= c.busyHighPct || s.Await >= defaultSlowAwait || afterSync:
+		// Back off aggressively when the disk is obviously suffering.
+		next := cur / 2
+		if next < c.minBPS {
+			next = c.minBPS
+		}
+		c.limiter.SetRate(next)
+
+	case s.UtilPct <= c.busyLowPct && s.Await <= defaultFastAwait:
+		// Recover slowly.
+		next := cur + (cur / 5) // +20%
+		if next > c.maxBPS {
+			next = c.maxBPS
+		}
+		c.limiter.SetRate(next)
+	}
+}
+
+type rateLimiter struct {
+	mu sync.Mutex
+
+	rateBPS int64
+	burst   int64
+	tokens  float64
+	last    time.Time
+}
+
+func newRateLimiter(rateBPS, burst int64) *rateLimiter {
+	now := time.Now()
+	if burst < 0 {
+		burst = 0
+	}
+	return &rateLimiter{
+		rateBPS: rateBPS,
+		burst:   burst,
+		tokens:  float64(burst),
+		last:    now,
+	}
+}
+
+func (r *rateLimiter) Rate() int64 {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	return r.rateBPS
+}
+
+func (r *rateLimiter) SetRate(rateBPS int64) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+
+	r.refillLocked(time.Now())
+	r.rateBPS = rateBPS
+
+	if rateBPS <= 0 {
+		r.tokens = 0
+		r.burst = 0
+		return
+	}
+
+	// Keep burst small and fixed. Do not let burst scale with rate.
+	r.burst = defaultBurstBytes
+	if r.tokens > float64(r.burst) {
+		r.tokens = float64(r.burst)
+	}
+}
+
+func (r *rateLimiter) Wait(ctx context.Context, n int) error {
+	if n <= 0 {
+		return nil
+	}
+
+	remaining := n
+	for remaining > 0 {
+		r.mu.Lock()
+
+		if r.rateBPS <= 0 {
+			r.mu.Unlock()
+			return nil
+		}
+
+		now := time.Now()
+		r.refillLocked(now)
+
+		allowed := remaining
+		if int64(allowed) > r.burst && r.burst > 0 {
+			allowed = int(r.burst)
+		}
+
+		if allowed <= 0 {
+			allowed = remaining
+		}
+
+		if r.tokens >= float64(allowed) {
+			r.tokens -= float64(allowed)
+			r.mu.Unlock()
+			remaining -= allowed
+			continue
+		}
+
+		missing := float64(allowed) - r.tokens
+		waitDur := time.Duration(missing / float64(r.rateBPS) * float64(time.Second))
+		if waitDur < 5*time.Millisecond {
+			waitDur = 5 * time.Millisecond
+		}
+		r.mu.Unlock()
+
+		timer := time.NewTimer(waitDur)
+		select {
+		case <-ctx.Done():
+			timer.Stop()
+			return ctx.Err()
+		case <-timer.C:
+		}
+	}
+
+	return nil
+}
+
+func (r *rateLimiter) refillLocked(now time.Time) {
+	if r.rateBPS <= 0 {
+		r.last = now
+		return
+	}
+
+	elapsed := now.Sub(r.last)
+	if elapsed <= 0 {
+		return
+	}
+
+	r.tokens += elapsed.Seconds() * float64(r.rateBPS)
+	if r.tokens > float64(r.burst) {
+		r.tokens = float64(r.burst)
+	}
+	r.last = now
+}
+
+type diskBusySample struct {
+	UtilPct float64
+	Await   time.Duration
+}
+
+type diskBusyMonitor struct {
+	major int
+	minor int
+
+	lastAt     time.Time
+	lastIOMs   uint64
+	lastWrites uint64
+}
+
+func newDiskBusyMonitor(targetPath string) (*diskBusyMonitor, error) {
+	major, minor, err := resolveWholeDiskMajorMinor(targetPath)
+	if err != nil {
+		return nil, err
+	}
+
+	ioMs, writes, err := readDiskStats(major, minor)
+	if err != nil {
+		return nil, err
+	}
+
+	return &diskBusyMonitor{
+		major:      major,
+		minor:      minor,
+		lastAt:     time.Now(),
+		lastIOMs:   ioMs,
+		lastWrites: writes,
+	}, nil
+}
+
+func (m *diskBusyMonitor) Sample(now time.Time) (diskBusySample, error) {
+	ioMs, writes, err := readDiskStats(m.major, m.minor)
+	if err != nil {
+		return diskBusySample{}, err
+	}
+
+	elapsedMs := now.Sub(m.lastAt).Milliseconds()
+	if elapsedMs <= 0 {
+		return diskBusySample{}, nil
+	}
+
+	deltaIOMs := int64(ioMs - m.lastIOMs)
+	deltaWrites := int64(writes - m.lastWrites)
+
+	m.lastAt = now
+	m.lastIOMs = ioMs
+	m.lastWrites = writes
+
+	util := float64(deltaIOMs) * 100 / float64(elapsedMs)
+	if util < 0 {
+		util = 0
+	}
+	if util > 100 {
+		util = 100
+	}
+
+	var await time.Duration
+	if deltaWrites > 0 {
+		await = time.Duration(deltaIOMs/int64(deltaWrites)) * time.Millisecond
+	}
+
+	return diskBusySample{
+		UtilPct: util,
+		Await:   await,
+	}, nil
+}
+
+func resolveWholeDiskMajorMinor(targetPath string) (int, int, error) {
+	var st unix.Stat_t
+	if err := unix.Stat(targetPath, &st); err != nil {
+		return 0, 0, fmt.Errorf("stat target %q: %w", targetPath, err)
+	}
+	if st.Mode&unix.S_IFMT != unix.S_IFBLK {
+		return 0, 0, fmt.Errorf("target %q is not a block device", targetPath)
+	}
+
+	major := int(unix.Major(uint64(st.Rdev)))
+	minor := int(unix.Minor(uint64(st.Rdev)))
+
+	sysfsPath := fmt.Sprintf("/sys/dev/block/%d:%d", major, minor)
+	resolved, err := filepath.EvalSymlinks(sysfsPath)
+	if err != nil {
+		return major, minor, nil
+	}
+
+	// Partition path usually looks like .../block/sda/sda3
+	// Parent whole disk is .../block/sda
+	parent := filepath.Dir(resolved)
+	devName := filepath.Base(parent)
+
+	ueventPath := filepath.Join(parent, "dev")
+	data, err := os.ReadFile(ueventPath)
+	if err != nil {
+		return major, minor, nil
+	}
+
+	parts := strings.Split(strings.TrimSpace(string(data)), ":")
+	if len(parts) != 2 {
+		return major, minor, nil
+	}
+
+	parentMajor, err1 := strconv.Atoi(parts[0])
+	parentMinor, err2 := strconv.Atoi(parts[1])
+	if err1 != nil || err2 != nil || devName == "" {
+		return major, minor, nil
+	}
+
+	return parentMajor, parentMinor, nil
+}
+
+func readDiskStats(major, minor int) (ioMs uint64, writesCompleted uint64, err error) {
+	f, err := os.Open("/proc/diskstats")
+	if err != nil {
+		return 0, 0, fmt.Errorf("open /proc/diskstats: %w", err)
+	}
+	defer f.Close()
+
+	sc := bufio.NewScanner(f)
+	for sc.Scan() {
+		line := strings.Fields(sc.Text())
+		if len(line) < 14 {
+			continue
+		}
+
+		maj, err := strconv.Atoi(line[0])
+		if err != nil {
+			continue
+		}
+		min, err := strconv.Atoi(line[1])
+		if err != nil {
+			continue
+		}
+		if maj != major || min != minor {
+			continue
+		}
+
+		// writes completed successfully: field 5, index 4
+		writesCompleted, err = strconv.ParseUint(line[4], 10, 64)
+		if err != nil {
+			return 0, 0, fmt.Errorf("parse writes completed for %d:%d: %w", major, minor, err)
+		}
+
+		// time spent doing I/Os (ms): field 13, index 12
+		ioMs, err = strconv.ParseUint(line[12], 10, 64)
+		if err != nil {
+			return 0, 0, fmt.Errorf("parse io_ms for %d:%d: %w", major, minor, err)
+		}
+
+		return ioMs, writesCompleted, nil
+	}
+
+	if err := sc.Err(); err != nil {
+		return 0, 0, fmt.Errorf("scan /proc/diskstats: %w", err)
+	}
+
+	return 0, 0, fmt.Errorf("device %d:%d not found in /proc/diskstats", major, minor)
+}

clitools/pkg/controller/osimage/write_throttle_other.go

@@ -0,0 +1,19 @@
+//go:build !linux
+
+package osimage
+
+import "context"
+
+type adaptiveWriteController struct{}
+
+func newAdaptiveWriteController(string) (*adaptiveWriteController, error) {
+	return &adaptiveWriteController{}, nil
+}
+
+func newNoopAdaptiveWriteController() *adaptiveWriteController {
+	return &adaptiveWriteController{}
+}
+
+func (c *adaptiveWriteController) Wait(ctx context.Context, n int) error { return nil }
+func (c *adaptiveWriteController) ObserveWrite(n int)                    {}
+func (c *adaptiveWriteController) ObserveSync()                          {}

clitools/pkg/controller/osupgrade/bootenv.go

@@ -1,37 +0,0 @@
-package osupgrade
-
-import (
-	"context"
-	"fmt"
-	"os/exec"
-)
-
-type NextBootConfig struct {
-	Key   string
-	Value string
-}
-
-func SetNextBootEnv(ctx context.Context, cfg NextBootConfig) error {
-	if cfg.Key == "" {
-		return fmt.Errorf("boot env key is empty")
-	}
-	if cfg.Value == "" {
-		return fmt.Errorf("boot env value is empty")
-	}
-
-	cmd := exec.CommandContext(
-		ctx,
-		"/proc/self/exe",
-		"internal",
-		"fw-setenv",
-		"--key", cfg.Key,
-		"--value", cfg.Value,
-	)
-
-	out, err := cmd.CombinedOutput()
-	if err != nil {
-		return fmt.Errorf("invoke internal fw-setenv: %w: %s", err, string(out))
-	}
-
-	return nil
-}

clitools/pkg/controller/osupgrade/handler.go

@@ -3,6 +3,8 @@ package osupgrade
 import (
 	"context"
 	"fmt"
+	"os"
+	"sync/atomic"
 
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/klog/v2"
@@ -12,19 +14,89 @@ import (
 	"example.com/monok8s/pkg/catalog"
 	"example.com/monok8s/pkg/controller/osimage"
 	"example.com/monok8s/pkg/kube"
+	"example.com/monok8s/pkg/node/uboot"
 )
 
-func HandleOSUpgrade(ctx context.Context, clients *kube.Clients,
-	namespace string, nodeName string,
-	osu *monov1alpha1.OSUpgrade,
-) error {
-	osup, err := ensureProgressHeartbeat(ctx, clients, namespace, nodeName, osu)
-	if err != nil {
-		return err
+type UpgradeRunner struct {
+	running   atomic.Bool
+	rebooting atomic.Bool
+}
+
+var r UpgradeRunner
+
+func (r *UpgradeRunner) Run(fn func() error) error {
+	if r.rebooting.Load() {
+		return nil
 	}
 
-	klog.InfoS("handling osupgrade",
-		"name", osu.Name,
+	if !r.running.CompareAndSwap(false, true) {
+		return nil
+	}
+	defer r.running.Store(false)
+
+	if r.rebooting.Load() {
+		return nil
+	}
+
+	return fn()
+}
+
+func HandleOSUpgradeProgress(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace string,
+	nodeName string,
+	osup *monov1alpha1.OSUpgradeProgress,
+) error {
+	return r.Run(func() error {
+		return handleOSUpgradeProgressLocked(ctx, clients, namespace, nodeName, osup)
+	})
+}
+
+func handleOSUpgradeProgressLocked(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace string,
+	nodeName string,
+	osup *monov1alpha1.OSUpgradeProgress,
+) error {
+	if osup == nil {
+		return fmt.Errorf("osupgradeprogress is nil")
+	}
+
+	if osup.Spec.NodeName != nodeName {
+		klog.V(4).InfoS("skipping osupgradeprogress due to nodeName mismatch",
+			"name", osup.Name,
+			"node", nodeName,
+			"target", osup.Spec.NodeName,
+		)
+		return nil
+	}
+
+	if !shouldProcessProgress(osup) {
+		klog.V(2).InfoS("skipping osupgradeprogress due to phase",
+			"name", osup.Name,
+			"node", nodeName,
+			"phase", osup.StatusPhase(),
+		)
+		return nil
+	}
+
+	parentName := osup.Spec.SourceRef.Name
+	if parentName == "" {
+		return failProgress(ctx, clients, osup, "resolve parent osupgrade", fmt.Errorf("missing spec.osUpgradeName"))
+	}
+
+	osu, err := clients.MonoKS.Monok8sV1alpha1().
+		OSUpgrades(namespace).
+		Get(ctx, parentName, metav1.GetOptions{})
+	if err != nil {
+		return failProgress(ctx, clients, osup, "resolve parent osupgrade", err)
+	}
+
+	klog.InfoS("handling osupgradeprogress",
+		"name", osup.Name,
+		"osupgrade", osu.Name,
 		"node", nodeName,
 		"desiredVersion", osu.Spec.DesiredVersion,
 	)
@@ -55,14 +127,19 @@ func HandleOSUpgrade(ctx context.Context, clients *kube.Clients,
 
 	first := plan.Path[0]
 
-	osup.Status.TargetVersion = plan.ResolvedTarget
-	osup.Status.Phase = monov1alpha1.OSUpgradeProgressPhaseDownloading
-	osup.Status.Message = fmt.Sprintf("downloading image: %s", first.URL)
-
-	now := metav1.Now()
-	osup.Status.LastUpdatedAt = &now
-	osup, err = updateProgressStatus(ctx, clients, osup_gvr, osup)
-
+	updated, err := updateProgressRobust(ctx, clients, osup.Namespace, osup.Name, func(cur *monov1alpha1.OSUpgradeProgress) {
+		now := metav1.Now()
+		cur.Status.CurrentVersion = buildinfo.KubeVersion
+		cur.Status.TargetVersion = plan.ResolvedTarget
+		cur.Status.PlannedPath = plannedPath(plan)
+		cur.Status.Phase = monov1alpha1.OSUpgradeProgressPhaseDownloading
+		cur.Status.ObservedRetryNonce = cur.Spec.RetryNonce
+		cur.Status.Message = fmt.Sprintf("downloading image: %s", first.URL)
+		cur.Status.LastUpdatedAt = &now
+	})
+	if updated != nil {
+		osup = updated
+	}
 	if err != nil {
 		return fmt.Errorf("update progress status: %w", err)
 	}
@@ -73,14 +150,13 @@ func HandleOSUpgrade(ctx context.Context, clients *kube.Clients,
 		"resolvedTarget", plan.ResolvedTarget,
 		"steps", len(plan.Path),
 		"currentVersion", buildinfo.KubeVersion,
-		"fSHA256irstVersion", first.Version,
+		"firstVersion", first.Version,
 		"firstURL", first.URL,
 		"size", first.Size,
 	)
 
 	imageSHA, err := first.SHA256()
 	if err != nil {
-		now = metav1.Now()
 		return failProgress(ctx, clients, osup, "apply image", err)
 	}
 
@@ -89,31 +165,41 @@ func HandleOSUpgrade(ctx context.Context, clients *kube.Clients,
 
 	imageOptions := osimage.ApplyOptions{
 		URL:               first.URL,
-		TargetPath:        "./out/flash.img",
+		TargetPath:        monov1alpha1.AltPartDeviceLink,
 		ExpectedRawSHA256: imageSHA,
 		ExpectedRawSize:   first.Size,
 		BufferSize:        6 * 1024 * 1024,
 		Progress: func(p osimage.Progress) {
 			pLogger.Log(p)
+
 			if err := statusUpdater.Run(func() error {
 
-				now := metav1.Now()
-				switch p.Stage {
-				case "flash":
-					osup.Status.Phase = monov1alpha1.OSUpgradeProgressPhaseWriting
-				case "verify":
-					osup.Status.Phase = monov1alpha1.OSUpgradeProgressPhaseVerifying
-				}
-				osup.Status.LastUpdatedAt = &now
-				osup.Status.Message = fmt.Sprintf("%s: %d%%", p.Stage, osimage.PercentOf(p.BytesComplete, p.BytesTotal))
+				klog.Infof("%s: %d%%", p.Stage, osimage.PercentOf(p.BytesComplete, p.BytesTotal))
 
-				updated, err := updateProgressStatus(ctx, clients, osup_gvr, osup)
+				updated, err := updateProgressRobust(ctx, clients, osup.Namespace, osup.Name, func(cur *monov1alpha1.OSUpgradeProgress) {
+					now := metav1.Now()
+
+					switch p.Stage {
+					case "flash":
+						cur.Status.Phase = monov1alpha1.OSUpgradeProgressPhaseWriting
+					case "verify":
+						cur.Status.Phase = monov1alpha1.OSUpgradeProgressPhaseVerifying
+					}
+
+					cur.Status.TargetVersion = plan.ResolvedTarget
+					cur.Status.LastUpdatedAt = &now
+					cur.Status.Message = fmt.Sprintf(
+						"%s: %d%%",
+						p.Stage,
+						osimage.PercentOf(p.BytesComplete, p.BytesTotal),
+					)
+				})
+				if updated != nil {
+					osup = updated
+				}
 				if err != nil {
-					klog.ErrorS(err, "update progress status")
-					return err
+					return fmt.Errorf("update progress status: %w", err)
 				}
-
-				osup = updated
 				return nil
 			}); err != nil {
 				klog.ErrorS(err, "throttled progress update failed")
@@ -123,27 +209,67 @@ func HandleOSUpgrade(ctx context.Context, clients *kube.Clients,
 
 	result, err := osimage.ApplyImageStreamed(ctx, imageOptions)
 	if err != nil {
-		now = metav1.Now()
 		return failProgress(ctx, clients, osup, "apply image", err)
 	}
 
 	klog.Info(result)
-	if err := SetNextBootEnv(ctx, NextBootConfig{
-		Key:   "boot_part",
-		Value: "B",
-	}); err != nil {
+
+	cfgPath := os.Getenv("FW_ENV_CONFIG_FILE")
+	if err := uboot.ConfigureNextBoot(ctx, cfgPath); err != nil {
 		return failProgress(ctx, clients, osup, "set boot env", err)
 	}
 
-	now = metav1.Now()
-	osup.Status.LastUpdatedAt = &now
-	osup.Status.Message = "image applied, verified, and next boot environment updated"
-	osup.Status.Phase = monov1alpha1.OSUpgradeProgressPhaseRebooting
-
-	osup, err = updateProgressStatus(ctx, clients, osup_gvr, osup)
+	updated, err = updateProgressRobust(ctx, clients, osup.Namespace, osup.Name, func(cur *monov1alpha1.OSUpgradeProgress) {
+		now := metav1.Now()
+		cur.Status.TargetVersion = plan.ResolvedTarget
+		cur.Status.Message = "image applied, verified, and next boot environment updated"
+		cur.Status.Phase = monov1alpha1.OSUpgradeProgressPhaseRebooting
+		cur.Status.LastUpdatedAt = &now
+	})
+	if updated != nil {
+		osup = updated
+	}
 	if err != nil {
 		return fmt.Errorf("update progress status: %w", err)
 	}
 
-	return nil
+	// TODO: Drain the node here
+	// Get all running pods outta here!
+	// kubectl.Run()
+	// Wait for the node to be drained
+	// kubectl.Wait()
+
+	r.rebooting.Store(true)
+	if err := triggerReboot(); err != nil {
+		r.rebooting.Store(false)
+		return fmt.Errorf("trigger reboot: %w", err)
+	}
+	select {}
+}
+
+func shouldProcessProgress(osup *monov1alpha1.OSUpgradeProgress) bool {
+	if osup == nil {
+		return false
+	}
+
+	if osup.Status == nil {
+		return false
+	}
+
+	switch osup.Status.Phase {
+	case "",
+		monov1alpha1.OSUpgradeProgressPhasePending,
+		monov1alpha1.OSUpgradeProgressPhaseRebooting:
+		return true
+	case monov1alpha1.OSUpgradeProgressPhaseFailed:
+		return osup.Spec.RetryNonce != osup.Status.ObservedRetryNonce
+	default:
+		return false
+	}
+}
+
+func triggerReboot() error {
+	_ = os.WriteFile("/proc/sysrq-trigger", []byte("s\n"), 0)
+	_ = os.WriteFile("/proc/sysrq-trigger", []byte("u\n"), 0)
+	return os.WriteFile("/proc/sysrq-trigger", []byte("b\n"), 0)
 }

clitools/pkg/controller/osupgrade/planner.go

@@ -224,9 +224,9 @@ func calculatePath(current, target string, available []string) ([]string, error)
 		add(latestCurMinor)
 	}
 
-	// Step 2: walk each intermediate minor using the lowest available patch in that minor.
+	// Step 2: walk each intermediate minor using the latest available patch in that minor.
 	for minor := cur.Minor + 1; minor < tgt.Minor; minor++ {
-		bridge, ok := lowestPatchInMinor(versions, cur.Major, minor)
+		bridge, ok := latestAnyPatchInMinor(versions, cur.Major, minor)
 		if !ok {
 			return nil, fmt.Errorf("no available bridge version for v%d.%d.x", cur.Major, minor)
 		}
@@ -239,6 +239,23 @@ func calculatePath(current, target string, available []string) ([]string, error)
 	return versionsToStrings(path), nil
 }
 
+func latestAnyPatchInMinor(versions []Version, major, minor int) (Version, bool) {
+	var found Version
+	ok := false
+
+	for _, v := range versions {
+		if v.Major != major || v.Minor != minor {
+			continue
+		}
+		if !ok || found.Compare(v) < 0 {
+			found = v
+			ok = true
+		}
+	}
+
+	return found, ok
+}
+
 func parseAndSortVersions(raw []string) ([]Version, error) {
 	out := make([]Version, 0, len(raw))
 	seen := map[string]struct{}{}
@@ -318,6 +335,14 @@ func lowestPatchInMinor(versions []Version, major, minor int) (Version, bool) {
 	return Version{}, false
 }
 
+func plannedPath(plan *Plan) []string {
+	ppath := []string{}
+	for _, img := range plan.Path {
+		ppath = append(ppath, img.Version)
+	}
+	return ppath
+}
+
 func versionsToStrings(vs []Version) []string {
 	out := make([]string, 0, len(vs))
 	for _, v := range vs {

clitools/pkg/controller/osupgrade/planner_test.go

@@ -0,0 +1,149 @@
+package osupgrade
+
+import (
+	"reflect"
+	"testing"
+)
+
+func TestCalculatePath(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name      string
+		current   string
+		target    string
+		available []string
+		want      []string
+		wantErr   bool
+	}{
+		{
+			name:      "same version returns nil path",
+			current:   "v1.34.6",
+			target:    "v1.34.6",
+			available: []string{"v1.34.6"},
+			want:      nil,
+			wantErr:   false,
+		},
+		{
+			name:      "same minor jumps directly to target",
+			current:   "v1.34.1",
+			target:    "v1.34.6",
+			available: []string{"v1.34.1", "v1.34.3", "v1.34.6"},
+			want:      []string{"v1.34.6"},
+			wantErr:   false,
+		},
+		{
+			name:      "next minor direct jump when no current minor patch available",
+			current:   "v1.34.6",
+			target:    "v1.35.3",
+			available: []string{"v1.34.6", "v1.35.1", "v1.35.3"},
+			want:      []string{"v1.35.3"},
+			wantErr:   false,
+		},
+		{
+			name:      "finish current minor then target",
+			current:   "v1.34.1",
+			target:    "v1.35.3",
+			available: []string{"v1.34.1", "v1.34.6", "v1.35.1", "v1.35.3"},
+			want:      []string{"v1.34.6", "v1.35.3"},
+			wantErr:   false,
+		},
+		{
+			name:      "multi minor path uses latest bridge patch",
+			current:   "v1.33.10",
+			target:    "v1.35.3",
+			available: []string{"v1.34.1", "v1.34.6", "v1.35.1", "v1.35.3"},
+			want:      []string{"v1.34.6", "v1.35.3"},
+			wantErr:   false,
+		},
+		{
+			name:      "multi minor path finishes current minor and latest bridge patch",
+			current:   "v1.33.1",
+			target:    "v1.35.3",
+			available: []string{"v1.33.5", "v1.33.9", "v1.34.1", "v1.34.6", "v1.35.3"},
+			want:      []string{"v1.33.9", "v1.34.6", "v1.35.3"},
+			wantErr:   false,
+		},
+		{
+			name:      "duplicates in available are ignored",
+			current:   "v1.33.10",
+			target:    "v1.35.3",
+			available: []string{"v1.34.6", "v1.34.6", "v1.35.3", "v1.35.3"},
+			want:      []string{"v1.34.6", "v1.35.3"},
+			wantErr:   false,
+		},
+		{
+			name:      "target missing returns error",
+			current:   "v1.34.6",
+			target:    "v1.35.3",
+			available: []string{"v1.34.6", "v1.35.1"},
+			wantErr:   true,
+		},
+		{
+			name:      "missing bridge minor returns error",
+			current:   "v1.33.10",
+			target:    "v1.35.3",
+			available: []string{"v1.35.3"},
+			wantErr:   true,
+		},
+		{
+			name:      "downgrade not supported",
+			current:   "v1.35.3",
+			target:    "v1.34.6",
+			available: []string{"v1.34.6", "v1.35.3"},
+			wantErr:   true,
+		},
+		{
+			name:      "cross major not supported",
+			current:   "v1.35.3",
+			target:    "v2.0.0",
+			available: []string{"v1.35.3", "v2.0.0"},
+			wantErr:   true,
+		},
+		{
+			name:      "invalid current version returns error",
+			current:   "garbage",
+			target:    "v1.35.3",
+			available: []string{"v1.35.3"},
+			wantErr:   true,
+		},
+		{
+			name:      "invalid target version returns error",
+			current:   "v1.34.6",
+			target:    "wat",
+			available: []string{"v1.34.6", "v1.35.3"},
+			wantErr:   true,
+		},
+		{
+			name:      "invalid available version returns error",
+			current:   "v1.34.6",
+			target:    "v1.35.3",
+			available: []string{"v1.34.6", "broken", "v1.35.3"},
+			wantErr:   true,
+		},
+	}
+
+	for _, tt := range tests {
+		tt := tt
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			got, err := calculatePath(tt.current, tt.target, tt.available)
+			if tt.wantErr {
+				if err == nil {
+					t.Fatalf("expected error, got nil; path=%v", got)
+				}
+				return
+			}
+
+			if err != nil {
+				t.Fatalf("unexpected error: %v", err)
+			}
+
+			if !reflect.DeepEqual(got, tt.want) {
+				t.Fatalf("calculatePath(%q, %q, %v)\n got:  %v\n want: %v",
+					tt.current, tt.target, tt.available, got, tt.want)
+			}
+		})
+	}
+}

clitools/pkg/controller/osupgrade/progress.go

@@ -3,44 +3,31 @@ package osupgrade
 import (
 	"context"
 	"fmt"
+	"strings"
 
 	apierrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
-	"k8s.io/apimachinery/pkg/runtime"
-	"k8s.io/apimachinery/pkg/runtime/schema"
+	"k8s.io/client-go/util/retry"
 	"k8s.io/klog/v2"
 
 	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
-	"example.com/monok8s/pkg/buildinfo"
 	"example.com/monok8s/pkg/kube"
 )
 
-var (
-	unstructuredConverter = runtime.DefaultUnstructuredConverter
-
-	osup_gvr = schema.GroupVersionResource{
-		Group:    monov1alpha1.Group,
-		Version:  monov1alpha1.Version,
-		Resource: "osupgradeprogresses",
-	}
-)
-
-func ensureProgressHeartbeat(ctx context.Context, clients *kube.Clients,
-	namespace string, nodeName string,
+func EnsureOSUpgradeProgressForNode(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace string,
+	nodeName string,
 	osu *monov1alpha1.OSUpgrade,
-) (*monov1alpha1.OSUpgradeProgress, error) {
+) error {
+	if osu == nil {
+		return fmt.Errorf("osupgrade is nil")
+	}
 
 	name := fmt.Sprintf("%s-%s", osu.Name, nodeName)
 	now := metav1.Now()
 
-	currentVersion := buildinfo.KubeVersion
-	targetVersion := ""
-
-	if osu.Status != nil {
-		targetVersion = osu.Status.ResolvedVersion
-	}
-
 	progress := &monov1alpha1.OSUpgradeProgress{
 		TypeMeta: metav1.TypeMeta{
 			APIVersion: monov1alpha1.APIVersion,
@@ -57,139 +44,155 @@ func ensureProgressHeartbeat(ctx context.Context, clients *kube.Clients,
 			},
 		},
 		Status: &monov1alpha1.OSUpgradeProgressStatus{
-			CurrentVersion: currentVersion,
-			TargetVersion:  targetVersion,
-			Phase:          monov1alpha1.OSUpgradeProgressPhasePending,
-			LastUpdatedAt:  &now,
-			Message:        "acknowledged",
+			Phase:         monov1alpha1.OSUpgradeProgressPhasePending,
+			LastUpdatedAt: &now,
 		},
 	}
 
-	created, err := createProgress(ctx, clients, osup_gvr, progress)
+	created, err := createProgress(ctx, clients, progress)
 	if err == nil {
 		klog.InfoS("created osupgradeprogress", "name", created.Name, "namespace", created.Namespace)
-		return created, nil
+		return nil
 	}
 	if !apierrors.IsAlreadyExists(err) {
-		return nil, fmt.Errorf("create OSUpgradeProgress %s/%s: %w", namespace, name, err)
+		return fmt.Errorf("create OSUpgradeProgress %s/%s: %w", namespace, name, err)
 	}
 
-	existing, err := getProgress(ctx, clients, osup_gvr, namespace, name)
+	existing, err := getProgress(ctx, clients, namespace, name)
 	if err != nil {
-		return nil, fmt.Errorf("get existing OSUpgradeProgress %s/%s: %w", namespace, name, err)
+		return fmt.Errorf("get existing OSUpgradeProgress %s/%s: %w", namespace, name, err)
 	}
 
-	// Spec should remain aligned with the source and node.
-	existing.Spec.NodeName = nodeName
-	existing.Spec.SourceRef.Name = osu.Name
-
-	if existing, err = updateProgressSpec(ctx, clients, osup_gvr, existing); err != nil {
-		return nil, fmt.Errorf("update OSUpgradeProgress spec %s/%s: %w", namespace, name, err)
+	if existing.Spec.NodeName != nodeName || existing.Spec.SourceRef.Name != osu.Name {
+		return fmt.Errorf(
+			"conflicting OSUpgradeProgress %s/%s already exists: got spec.nodeName=%q spec.sourceRef.name=%q, want nodeName=%q sourceRef.name=%q",
+			namespace,
+			name,
+			existing.Spec.NodeName,
+			existing.Spec.SourceRef.Name,
+			nodeName,
+			osu.Name,
+		)
 	}
 
-	if existing.Status == nil {
-		existing.Status = &monov1alpha1.OSUpgradeProgressStatus{}
+	return nil
+}
+
+func updateProgressRobust(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace string,
+	name string,
+	mutate func(*monov1alpha1.OSUpgradeProgress),
+) (*monov1alpha1.OSUpgradeProgress, error) {
+	var out *monov1alpha1.OSUpgradeProgress
+
+	err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
+		current, err := getProgress(ctx, clients, namespace, name)
+		if err != nil {
+			return err
+		}
+
+		if current.Status == nil {
+			current.Status = &monov1alpha1.OSUpgradeProgressStatus{}
+		}
+
+		mutate(current)
+
+		updated, err := updateProgressStatus(ctx, clients, current)
+		if err != nil {
+			if isUnknownUpdateResult(err) {
+				latest, getErr := getProgress(ctx, clients, namespace, name)
+				if getErr == nil {
+					out = latest
+				}
+			}
+			return err
+		}
+
+		out = updated
+		return nil
+	})
+
+	if err != nil && out != nil {
+		// Unknown-result case: caller gets latest known server state plus error.
+		return out, err
 	}
 
-	existing.Status.CurrentVersion = currentVersion
-	existing.Status.TargetVersion = targetVersion
-	existing.Status.LastUpdatedAt = &now
+	return out, err
+}
 
-	// Only set phase/message if they are still empty, so later real state machine
-	// updates are not clobbered by the heartbeat.
-	if existing.Status.Phase == "" {
-		existing.Status.Phase = monov1alpha1.OSUpgradeProgressPhasePending
+func isUnknownUpdateResult(err error) bool {
+	if err == nil {
+		return false
 	}
 
-	if existing, err = updateProgressStatus(ctx, clients, osup_gvr, existing); err != nil {
-		return nil, fmt.Errorf("update OSUpgradeProgress status %s/%s: %w", namespace, name, err)
+	if apierrors.IsTimeout(err) ||
+		apierrors.IsServerTimeout(err) ||
+		apierrors.IsTooManyRequests(err) {
+		return true
 	}
 
-	klog.InfoS("updated osupgradeprogress", "name", existing.Name, "namespace", existing.Namespace)
-	return existing, nil
+	msg := strings.ToLower(err.Error())
+	return strings.Contains(msg, "request timed out") ||
+		strings.Contains(msg, "context deadline exceeded") ||
+		strings.Contains(msg, "etcdserver: request timed out") ||
+		strings.Contains(msg, "connection reset by peer") ||
+		strings.Contains(msg, "http2: client connection lost")
 }
 
 func createProgress(
 	ctx context.Context,
 	clients *kube.Clients,
-	gvr schema.GroupVersionResource,
 	progress *monov1alpha1.OSUpgradeProgress,
 ) (*monov1alpha1.OSUpgradeProgress, error) {
-	obj, err := toUnstructured(progress)
+	toCreate := progress.DeepCopy()
+	toCreate.Status = nil
+
+	created, err := clients.MonoKS.
+		Monok8sV1alpha1().
+		OSUpgradeProgresses(toCreate.Namespace).
+		Create(ctx, toCreate, metav1.CreateOptions{})
 	if err != nil {
 		return nil, err
 	}
 
-	created, err := clients.Dynamic.
-		Resource(gvr).
-		Namespace(progress.Namespace).
-		Create(ctx, obj, metav1.CreateOptions{})
-	if err != nil {
-		return nil, err
+	if progress.Status != nil {
+		toUpdate := created.DeepCopy()
+		toUpdate.Status = progress.Status
+		return updateProgressStatus(ctx, clients, toUpdate)
 	}
 
-	return fromUnstructuredProgress(created)
-}
-
-func getProgress(
-	ctx context.Context,
-	clients *kube.Clients,
-	gvr schema.GroupVersionResource,
-	namespace, name string,
-) (*monov1alpha1.OSUpgradeProgress, error) {
-	got, err := clients.Dynamic.
-		Resource(gvr).
-		Namespace(namespace).
-		Get(ctx, name, metav1.GetOptions{})
-	if err != nil {
-		return nil, err
-	}
-
-	return fromUnstructuredProgress(got)
-}
-
-func updateProgressSpec(
-	ctx context.Context,
-	clients *kube.Clients,
-	gvr schema.GroupVersionResource,
-	progress *monov1alpha1.OSUpgradeProgress,
-) (*monov1alpha1.OSUpgradeProgress, error) {
-	obj, err := toUnstructured(progress)
-	if err != nil {
-		return nil, err
-	}
-
-	updated, err := clients.Dynamic.
-		Resource(gvr).
-		Namespace(progress.Namespace).
-		Update(ctx, obj, metav1.UpdateOptions{})
-	if err != nil {
-		return nil, err
-	}
-
-	return fromUnstructuredProgress(updated)
+	return created, nil
 }
 
 func updateProgressStatus(
 	ctx context.Context,
 	clients *kube.Clients,
-	gvr schema.GroupVersionResource,
 	progress *monov1alpha1.OSUpgradeProgress,
 ) (*monov1alpha1.OSUpgradeProgress, error) {
-	obj, err := toUnstructured(progress)
+	updated, err := clients.MonoKS.
+		Monok8sV1alpha1().
+		OSUpgradeProgresses(progress.Namespace).
+		UpdateStatus(ctx, progress, metav1.UpdateOptions{})
 	if err != nil {
-		return nil, err
+		return nil, fmt.Errorf(
+			"update status for OSUpgradeProgress %s/%s: %w",
+			progress.Namespace, progress.Name, err,
+		)
 	}
+	return updated, nil
+}
 
-	updated, err := clients.Dynamic.
-		Resource(gvr).
-		Namespace(progress.Namespace).
-		UpdateStatus(ctx, obj, metav1.UpdateOptions{})
-	if err != nil {
-		return nil, err
-	}
-
-	return fromUnstructuredProgress(updated)
+func getProgress(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace, name string,
+) (*monov1alpha1.OSUpgradeProgress, error) {
+	return clients.MonoKS.
+		Monok8sV1alpha1().
+		OSUpgradeProgresses(namespace).
+		Get(ctx, name, metav1.GetOptions{})
 }
 
 func failProgress(
@@ -199,17 +202,19 @@ func failProgress(
 	action string,
 	cause error,
 ) error {
-	now := metav1.Now()
+	_, err := updateProgressRobust(ctx, clients, osup.Namespace, osup.Name, func(cur *monov1alpha1.OSUpgradeProgress) {
+		now := metav1.Now()
 
-	if osup.Status == nil {
-		osup.Status = &monov1alpha1.OSUpgradeProgressStatus{}
-	}
+		if cur.Status == nil {
+			cur.Status = &monov1alpha1.OSUpgradeProgressStatus{}
+		}
 
-	osup.Status.LastUpdatedAt = &now
-	osup.Status.Message = fmt.Sprintf("%s: %v", action, cause)
-	osup.Status.Phase = monov1alpha1.OSUpgradeProgressPhaseFailed
-
-	if _, err := updateProgressStatus(ctx, clients, osup_gvr, osup); err != nil {
+		cur.Status.ObservedRetryNonce = cur.Spec.RetryNonce
+		cur.Status.LastUpdatedAt = &now
+		cur.Status.Message = fmt.Sprintf("%s: %v", action, cause)
+		cur.Status.Phase = monov1alpha1.OSUpgradeProgressPhaseFailed
+	})
+	if err != nil {
 		klog.ErrorS(err, "failed to update osupgradeprogress status after error",
 			"action", action,
 			"name", osup.Name,
@@ -226,37 +231,24 @@ func markProgressCompleted(
 	osup *monov1alpha1.OSUpgradeProgress,
 	message string,
 ) error {
-	now := metav1.Now()
+	_, err := updateProgressRobust(ctx, clients, osup.Namespace, osup.Name, func(cur *monov1alpha1.OSUpgradeProgress) {
+		now := metav1.Now()
 
-	if osup.Status == nil {
-		osup.Status = &monov1alpha1.OSUpgradeProgressStatus{}
-	}
+		if cur.Status == nil {
+			cur.Status = &monov1alpha1.OSUpgradeProgressStatus{}
+		}
 
-	osup.Status.Phase = monov1alpha1.OSUpgradeProgressPhaseCompleted
-	osup.Status.Message = message
-	osup.Status.LastUpdatedAt = &now
-	osup.Status.CompletedAt = &now
-
-	_, err := updateProgressStatus(ctx, clients, osup_gvr, osup)
+		cur.Status.ObservedRetryNonce = cur.Spec.RetryNonce
+		cur.Status.Phase = monov1alpha1.OSUpgradeProgressPhaseCompleted
+		cur.Status.Message = message
+		cur.Status.CurrentVersion = osup.Status.CurrentVersion
+		cur.Status.TargetVersion = osup.Status.TargetVersion
+		cur.Status.LastUpdatedAt = &now
+		cur.Status.CompletedAt = &now
+	})
 	if err != nil {
 		return fmt.Errorf("mark progress completed: %w", err)
 	}
 
 	return nil
 }
-
-func toUnstructured(progress *monov1alpha1.OSUpgradeProgress) (*unstructured.Unstructured, error) {
-	m, err := unstructuredConverter.ToUnstructured(progress)
-	if err != nil {
-		return nil, fmt.Errorf("convert OSUpgradeProgress to unstructured: %w", err)
-	}
-	return &unstructured.Unstructured{Object: m}, nil
-}
-
-func fromUnstructuredProgress(obj *unstructured.Unstructured) (*monov1alpha1.OSUpgradeProgress, error) {
-	var progress monov1alpha1.OSUpgradeProgress
-	if err := unstructuredConverter.FromUnstructured(obj.Object, &progress); err != nil {
-		return nil, fmt.Errorf("convert unstructured to OSUpgradeProgress: %w", err)
-	}
-	return &progress, nil
-}

clitools/pkg/controller/osupgrade/watch.go

@@ -0,0 +1,345 @@
+package osupgrade
+
+import (
+	"context"
+	"fmt"
+	"time"
+
+	corev1 "k8s.io/api/core/v1"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/labels"
+	"k8s.io/apimachinery/pkg/watch"
+	"k8s.io/klog/v2"
+
+	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	"example.com/monok8s/pkg/kube"
+)
+
+func Watch(ctx context.Context, clients *kube.Clients, namespace string) error {
+	var resourceVersion string
+
+	for {
+		if ctx.Err() != nil {
+			return ctx.Err()
+		}
+
+		err := watchOnce(ctx, clients, namespace, &resourceVersion)
+		if err != nil {
+			if ctx.Err() != nil {
+				return ctx.Err()
+			}
+
+			// Expired RV is normal enough; clear it and relist.
+			if apierrors.IsResourceExpired(err) {
+				klog.InfoS("OSUpgrade watch resourceVersion expired; resetting",
+					"namespace", namespace,
+					"resourceVersion", resourceVersion,
+				)
+				resourceVersion = ""
+			} else {
+				klog.ErrorS(err, "OSUpgrade watch failed; retrying",
+					"namespace", namespace,
+					"resourceVersion", resourceVersion,
+				)
+			}
+
+			select {
+			case <-ctx.Done():
+				return ctx.Err()
+			case <-time.After(2 * time.Second):
+			}
+			continue
+		}
+	}
+}
+
+func watchOnce(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace string,
+	resourceVersion *string,
+) error {
+	// Cold start: list existing objects once, handle them, then watch from list RV.
+	if *resourceVersion == "" {
+		list, err := clients.MonoKS.
+			Monok8sV1alpha1().
+			OSUpgrades(namespace).
+			List(ctx, metav1.ListOptions{})
+		if err != nil {
+			return fmt.Errorf("list OSUpgrades: %w", err)
+		}
+
+		for i := range list.Items {
+			osu := &list.Items[i]
+			handled, err := handleOSUpgrade(ctx, clients, namespace, osu)
+			if err != nil {
+				klog.ErrorS(err, "reconcile existing OSUpgrade failed",
+					"name", osu.Name,
+					"resourceVersion", osu.ResourceVersion,
+				)
+				continue
+			}
+			if !handled {
+				klog.V(2).InfoS("skipping existing OSUpgrade",
+					"name", osu.Name,
+					"phase", osu.StatusPhase(),
+				)
+			}
+		}
+
+		*resourceVersion = list.ResourceVersion
+		klog.InfoS("initial OSUpgrade sync complete",
+			"namespace", namespace,
+			"resourceVersion", *resourceVersion,
+			"count", len(list.Items),
+		)
+	}
+
+	w, err := clients.MonoKS.
+		Monok8sV1alpha1().
+		OSUpgrades(namespace).
+		Watch(ctx, metav1.ListOptions{
+			ResourceVersion:     *resourceVersion,
+			AllowWatchBookmarks: true,
+		})
+	if err != nil {
+		return fmt.Errorf("watch OSUpgrades: %w", err)
+	}
+	defer w.Stop()
+
+	klog.InfoS("watching OSUpgrades",
+		"namespace", namespace,
+		"resourceVersion", *resourceVersion,
+	)
+
+	for {
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+
+		case evt, ok := <-w.ResultChan():
+			if !ok {
+				return fmt.Errorf("watch channel closed")
+			}
+
+			switch evt.Type {
+			case watch.Bookmark:
+				if rv := extractResourceVersion(evt.Object); rv != "" {
+					*resourceVersion = rv
+				}
+				continue
+
+			case watch.Error:
+				// Let outer loop retry / relist.
+				return fmt.Errorf("watch returned error event")
+
+			case watch.Deleted:
+				// Top-level delete does not require action here.
+				continue
+
+			case watch.Added, watch.Modified:
+				// handled below
+
+			default:
+				klog.V(1).InfoS("skipping unexpected watch event type",
+					"eventType", evt.Type,
+				)
+				continue
+			}
+
+			osu, ok := evt.Object.(*monov1alpha1.OSUpgrade)
+			if !ok {
+				klog.V(1).InfoS("skipping unexpected watch object type",
+					"type", fmt.Sprintf("%T", evt.Object),
+				)
+				continue
+			}
+
+			if osu.ResourceVersion != "" {
+				*resourceVersion = osu.ResourceVersion
+			}
+
+			handled, err := handleOSUpgrade(ctx, clients, namespace, osu)
+			if err != nil {
+				klog.ErrorS(err, "reconcile OSUpgrade failed",
+					"name", osu.Name,
+					"eventType", evt.Type,
+					"resourceVersion", osu.ResourceVersion,
+				)
+				continue
+			}
+			if !handled {
+				klog.V(2).InfoS("skipping OSUpgrade",
+					"name", osu.Name,
+					"eventType", evt.Type,
+					"phase", osu.StatusPhase(),
+				)
+			}
+		}
+	}
+}
+
+func handleOSUpgrade(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace string,
+	osu *monov1alpha1.OSUpgrade,
+) (bool, error) {
+	if !shouldHandle(osu) {
+		return false, nil
+	}
+
+	if osu.Status == nil || osu.Status.ObservedGeneration != osu.Generation {
+		return true, reconcileSpec(ctx, clients, namespace, osu)
+	}
+
+	if osu.Status.Phase == monov1alpha1.OSUpgradePhaseAccepted {
+		return true, reconcileFanout(ctx, clients, namespace, osu)
+	}
+
+	return false, nil
+}
+
+func reconcileSpec(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace string,
+	osu *monov1alpha1.OSUpgrade,
+) error {
+	osu = osu.DeepCopy()
+
+	osu.Status = &monov1alpha1.OSUpgradeStatus{
+		Phase:              monov1alpha1.OSUpgradePhaseAccepted,
+		ResolvedVersion:    osu.Spec.DesiredVersion,
+		ObservedGeneration: osu.Generation,
+	}
+
+	_, err := clients.MonoKS.
+		Monok8sV1alpha1().
+		OSUpgrades(namespace).
+		UpdateStatus(ctx, osu, metav1.UpdateOptions{})
+
+	return err
+}
+
+func reconcileFanout(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace string,
+	osu *monov1alpha1.OSUpgrade,
+) error {
+	nodeNames, err := listTargetNodeNames(ctx, clients, osu)
+	if err != nil {
+		return fmt.Errorf("list target nodes for %s: %w", osu.Name, err)
+	}
+
+	if len(nodeNames) == 0 {
+		klog.InfoS("no targets", "osupgrade", osu.Name)
+		return nil
+	}
+
+	klog.InfoS("ensuring OSUpgradeProgress for target nodes",
+		"osupgrade", osu.Name,
+		"targets", len(nodeNames),
+	)
+
+	for _, nodeName := range nodeNames {
+		if err := EnsureOSUpgradeProgressForNode(
+			ctx,
+			clients,
+			namespace,
+			nodeName,
+			osu,
+		); err != nil {
+			klog.ErrorS(err, "ensure OSUpgradeProgress for node failed",
+				"osupgrade", osu.Name,
+				"node", nodeName,
+			)
+		}
+	}
+
+	return nil
+}
+
+func listTargetNodeNames(
+	ctx context.Context,
+	clients *kube.Clients,
+	osu *monov1alpha1.OSUpgrade,
+) ([]string, error) {
+	selector := labels.SelectorFromSet(labels.Set{
+		monov1alpha1.NodeControlKey: "true",
+	})
+
+	if osu.Spec.NodeSelector != nil {
+		userSelector, err := metav1.LabelSelectorAsSelector(osu.Spec.NodeSelector)
+		if err != nil {
+			return nil, fmt.Errorf("invalid nodeSelector: %w", err)
+		}
+
+		reqs, selectable := userSelector.Requirements()
+		if !selectable {
+			selector = labels.Nothing()
+		} else {
+			selector = selector.Add(reqs...)
+		}
+	}
+
+	list, err := clients.Kubernetes.CoreV1().
+		Nodes().
+		List(ctx, metav1.ListOptions{
+			LabelSelector: selector.String(),
+		})
+	if err != nil {
+		return nil, fmt.Errorf("list nodes: %w", err)
+	}
+
+	out := make([]string, 0, len(list.Items))
+	for i := range list.Items {
+		node := &list.Items[i]
+		if shouldUseNode(node) {
+			out = append(out, node.Name)
+		}
+	}
+
+	return out, nil
+}
+
+func shouldUseNode(node *corev1.Node) bool {
+	return node != nil && node.Name != ""
+}
+
+func shouldHandle(osu *monov1alpha1.OSUpgrade) bool {
+	if osu == nil || osu.DeletionTimestamp != nil {
+		return false
+	}
+
+	if osu.Spec.DesiredVersion == "" {
+		return false
+	}
+
+	// NEW: initial processing stage
+	if osu.Status == nil {
+		return true
+	}
+
+	// Reconcile if spec changed
+	if osu.Status.ObservedGeneration != osu.Generation {
+		return true
+	}
+
+	// Fanout stage
+	return osu.Status.Phase == monov1alpha1.OSUpgradePhaseAccepted
+}
+
+func extractResourceVersion(obj interface{}) string {
+	type hasRV interface {
+		GetResourceVersion() string
+	}
+
+	if o, ok := obj.(hasRV); ok {
+		return o.GetResourceVersion()
+	}
+
+	return ""
+}

clitools/pkg/controller/server.go

@@ -0,0 +1,92 @@
+package controller
+
+import (
+	"context"
+	"net/http"
+	"time"
+
+	"example.com/monok8s/pkg/kube"
+
+	"github.com/emicklei/go-restful/v3"
+	"k8s.io/apiserver/pkg/server/httplog"
+)
+
+var statusesNoTracePred = httplog.StatusIsNot(
+	http.StatusOK,
+	http.StatusFound,
+	http.StatusMovedPermanently,
+	http.StatusTemporaryRedirect,
+	http.StatusBadRequest,
+	http.StatusNotFound,
+	http.StatusSwitchingProtocols,
+)
+
+type Server struct {
+	restfulCont *restful.Container
+
+	ctx       context.Context
+	clients   *kube.Clients
+	namespace string
+	nodeName  string
+	startedAt time.Time
+}
+
+type StatusResponse struct {
+	OK        bool   `json:"ok"`
+	Service   string `json:"service"`
+	Namespace string `json:"namespace,omitempty"`
+	NodeName  string `json:"nodeName,omitempty"`
+	UptimeSec int64  `json:"uptimeSec"`
+}
+
+func NewServer(ctx context.Context, clients *kube.Clients, namespace, nodeName string) *Server {
+	s := &Server{
+		ctx:       ctx,
+		clients:   clients,
+		namespace: namespace,
+		nodeName:  nodeName,
+		startedAt: time.Now(),
+	}
+	s.Initialize()
+	return s
+}
+
+func (s *Server) ServeHTTP(w http.ResponseWriter, req *http.Request) {
+	if s == nil {
+		http.Error(w, "server is nil", http.StatusInternalServerError)
+		return
+	}
+	if s.restfulCont == nil {
+		http.Error(w, "server not initialized", http.StatusInternalServerError)
+		return
+	}
+
+	handler := httplog.WithLogging(s.restfulCont, statusesNoTracePred)
+	handler.ServeHTTP(w, req)
+}
+
+func (s *Server) Initialize() {
+	s.restfulCont = restful.NewContainer()
+
+	ws := new(restful.WebService)
+	ws.Path("/")
+	ws.Consumes(restful.MIME_JSON)
+	ws.Produces(restful.MIME_JSON)
+
+	ws.Route(ws.GET("/status").To(s.queryStatus).
+		Doc("Return basic controller status"))
+
+	s.restfulCont.Add(ws)
+}
+
+func (s *Server) queryStatus(request *restful.Request, response *restful.Response) {
+	resp := StatusResponse{
+		OK:        true,
+		Service:   "monok8s-controller",
+		Namespace: s.namespace,
+		NodeName:  s.nodeName,
+		UptimeSec: int64(time.Since(s.startedAt).Seconds()),
+	}
+
+	_ = response.WriteHeaderAndEntity(http.StatusOK, resp)
+}

clitools/pkg/crds/crds.go

@@ -1,76 +0,0 @@
-package crds
-
-import (
-	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
-	apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-)
-
-func Definitions() []*apiextensionsv1.CustomResourceDefinition {
-	return []*apiextensionsv1.CustomResourceDefinition{
-		monoKSConfigCRD(),
-		osUpgradeCRD(),
-	}
-}
-
-func monoKSConfigCRD() *apiextensionsv1.CustomResourceDefinition {
-	return &apiextensionsv1.CustomResourceDefinition{
-		ObjectMeta: metav1.ObjectMeta{
-			Name: monov1alpha1.MonoKSConfigCRD,
-		},
-		Spec: apiextensionsv1.CustomResourceDefinitionSpec{
-			Group: monov1alpha1.Group,
-			Scope: apiextensionsv1.NamespaceScoped,
-			Names: apiextensionsv1.CustomResourceDefinitionNames{
-				Plural:     "monoksconfigs",
-				Singular:   "monoksconfig",
-				Kind:       "MonoKSConfig",
-				ShortNames: []string{"mkscfg"},
-			},
-			Versions: []apiextensionsv1.CustomResourceDefinitionVersion{{
-				Name:    "v1alpha1",
-				Served:  true,
-				Storage: true,
-				Schema: &apiextensionsv1.CustomResourceValidation{OpenAPIV3Schema: &apiextensionsv1.JSONSchemaProps{
-					Type: "object",
-					Properties: map[string]apiextensionsv1.JSONSchemaProps{
-						"spec":   {Type: "object", XPreserveUnknownFields: boolPtr(true)},
-						"status": {Type: "object", XPreserveUnknownFields: boolPtr(true)},
-					},
-				}},
-			}},
-		},
-	}
-}
-
-func osUpgradeCRD() *apiextensionsv1.CustomResourceDefinition {
-	return &apiextensionsv1.CustomResourceDefinition{
-		ObjectMeta: metav1.ObjectMeta{
-			Name: monov1alpha1.OSUpgradeCRD,
-		},
-		Spec: apiextensionsv1.CustomResourceDefinitionSpec{
-			Group: monov1alpha1.Group,
-			Scope: apiextensionsv1.NamespaceScoped,
-			Names: apiextensionsv1.CustomResourceDefinitionNames{
-				Plural:     "osupgrades",
-				Singular:   "osupgrade",
-				Kind:       "OSUpgrade",
-				ShortNames: []string{"osup"},
-			},
-			Versions: []apiextensionsv1.CustomResourceDefinitionVersion{{
-				Name:    "v1alpha1",
-				Served:  true,
-				Storage: true,
-				Schema: &apiextensionsv1.CustomResourceValidation{OpenAPIV3Schema: &apiextensionsv1.JSONSchemaProps{
-					Type: "object",
-					Properties: map[string]apiextensionsv1.JSONSchemaProps{
-						"spec":   {Type: "object", XPreserveUnknownFields: boolPtr(true)},
-						"status": {Type: "object", XPreserveUnknownFields: boolPtr(true)},
-					},
-				}},
-			}},
-		},
-	}
-}
-
-func boolPtr(v bool) *bool { return &v }

clitools/pkg/generated/clientset/versioned/clientset.go

@@ -0,0 +1,106 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package versioned
+
+import (
+	fmt "fmt"
+	http "net/http"
+
+	monok8sv1alpha1 "example.com/monok8s/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1"
+	discovery "k8s.io/client-go/discovery"
+	rest "k8s.io/client-go/rest"
+	flowcontrol "k8s.io/client-go/util/flowcontrol"
+)
+
+type Interface interface {
+	Discovery() discovery.DiscoveryInterface
+	Monok8sV1alpha1() monok8sv1alpha1.Monok8sV1alpha1Interface
+}
+
+// Clientset contains the clients for groups.
+type Clientset struct {
+	*discovery.DiscoveryClient
+	monok8sV1alpha1 *monok8sv1alpha1.Monok8sV1alpha1Client
+}
+
+// Monok8sV1alpha1 retrieves the Monok8sV1alpha1Client
+func (c *Clientset) Monok8sV1alpha1() monok8sv1alpha1.Monok8sV1alpha1Interface {
+	return c.monok8sV1alpha1
+}
+
+// Discovery retrieves the DiscoveryClient
+func (c *Clientset) Discovery() discovery.DiscoveryInterface {
+	if c == nil {
+		return nil
+	}
+	return c.DiscoveryClient
+}
+
+// NewForConfig creates a new Clientset for the given config.
+// If config's RateLimiter is not set and QPS and Burst are acceptable,
+// NewForConfig will generate a rate-limiter in configShallowCopy.
+// NewForConfig is equivalent to NewForConfigAndClient(c, httpClient),
+// where httpClient was generated with rest.HTTPClientFor(c).
+func NewForConfig(c *rest.Config) (*Clientset, error) {
+	configShallowCopy := *c
+
+	if configShallowCopy.UserAgent == "" {
+		configShallowCopy.UserAgent = rest.DefaultKubernetesUserAgent()
+	}
+
+	// share the transport between all clients
+	httpClient, err := rest.HTTPClientFor(&configShallowCopy)
+	if err != nil {
+		return nil, err
+	}
+
+	return NewForConfigAndClient(&configShallowCopy, httpClient)
+}
+
+// NewForConfigAndClient creates a new Clientset for the given config and http client.
+// Note the http client provided takes precedence over the configured transport values.
+// If config's RateLimiter is not set and QPS and Burst are acceptable,
+// NewForConfigAndClient will generate a rate-limiter in configShallowCopy.
+func NewForConfigAndClient(c *rest.Config, httpClient *http.Client) (*Clientset, error) {
+	configShallowCopy := *c
+	if configShallowCopy.RateLimiter == nil && configShallowCopy.QPS > 0 {
+		if configShallowCopy.Burst <= 0 {
+			return nil, fmt.Errorf("burst is required to be greater than 0 when RateLimiter is not set and QPS is set to greater than 0")
+		}
+		configShallowCopy.RateLimiter = flowcontrol.NewTokenBucketRateLimiter(configShallowCopy.QPS, configShallowCopy.Burst)
+	}
+
+	var cs Clientset
+	var err error
+	cs.monok8sV1alpha1, err = monok8sv1alpha1.NewForConfigAndClient(&configShallowCopy, httpClient)
+	if err != nil {
+		return nil, err
+	}
+
+	cs.DiscoveryClient, err = discovery.NewDiscoveryClientForConfigAndClient(&configShallowCopy, httpClient)
+	if err != nil {
+		return nil, err
+	}
+	return &cs, nil
+}
+
+// NewForConfigOrDie creates a new Clientset for the given config and
+// panics if there is an error in the config.
+func NewForConfigOrDie(c *rest.Config) *Clientset {
+	cs, err := NewForConfig(c)
+	if err != nil {
+		panic(err)
+	}
+	return cs
+}
+
+// New creates a new Clientset for the given RESTClient.
+func New(c rest.Interface) *Clientset {
+	var cs Clientset
+	cs.monok8sV1alpha1 = monok8sv1alpha1.New(c)
+
+	cs.DiscoveryClient = discovery.NewDiscoveryClient(c)
+	return &cs
+}

clitools/pkg/generated/clientset/versioned/fake/clientset_generated.go

@@ -0,0 +1,91 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package fake
+
+import (
+	clientset "example.com/monok8s/pkg/generated/clientset/versioned"
+	monok8sv1alpha1 "example.com/monok8s/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1"
+	fakemonok8sv1alpha1 "example.com/monok8s/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/fake"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/watch"
+	"k8s.io/client-go/discovery"
+	fakediscovery "k8s.io/client-go/discovery/fake"
+	"k8s.io/client-go/testing"
+)
+
+// NewSimpleClientset returns a clientset that will respond with the provided objects.
+// It's backed by a very simple object tracker that processes creates, updates and deletions as-is,
+// without applying any field management, validations and/or defaults. It shouldn't be considered a replacement
+// for a real clientset and is mostly useful in simple unit tests.
+//
+// Deprecated: NewClientset replaces this with support for field management, which significantly improves
+// server side apply testing. NewClientset is only available when apply configurations are generated (e.g.
+// via --with-applyconfig).
+func NewSimpleClientset(objects ...runtime.Object) *Clientset {
+	o := testing.NewObjectTracker(scheme, codecs.UniversalDecoder())
+	for _, obj := range objects {
+		if err := o.Add(obj); err != nil {
+			panic(err)
+		}
+	}
+
+	cs := &Clientset{tracker: o}
+	cs.discovery = &fakediscovery.FakeDiscovery{Fake: &cs.Fake}
+	cs.AddReactor("*", "*", testing.ObjectReaction(o))
+	cs.AddWatchReactor("*", func(action testing.Action) (handled bool, ret watch.Interface, err error) {
+		var opts metav1.ListOptions
+		if watchAction, ok := action.(testing.WatchActionImpl); ok {
+			opts = watchAction.ListOptions
+		}
+		gvr := action.GetResource()
+		ns := action.GetNamespace()
+		watch, err := o.Watch(gvr, ns, opts)
+		if err != nil {
+			return false, nil, err
+		}
+		return true, watch, nil
+	})
+
+	return cs
+}
+
+// Clientset implements clientset.Interface. Meant to be embedded into a
+// struct to get a default implementation. This makes faking out just the method
+// you want to test easier.
+type Clientset struct {
+	testing.Fake
+	discovery *fakediscovery.FakeDiscovery
+	tracker   testing.ObjectTracker
+}
+
+func (c *Clientset) Discovery() discovery.DiscoveryInterface {
+	return c.discovery
+}
+
+func (c *Clientset) Tracker() testing.ObjectTracker {
+	return c.tracker
+}
+
+// IsWatchListSemanticsSupported informs the reflector that this client
+// doesn't support WatchList semantics.
+//
+// This is a synthetic method whose sole purpose is to satisfy the optional
+// interface check performed by the reflector.
+// Returning true signals that WatchList can NOT be used.
+// No additional logic is implemented here.
+func (c *Clientset) IsWatchListSemanticsUnSupported() bool {
+	return true
+}
+
+var (
+	_ clientset.Interface = &Clientset{}
+	_ testing.FakeClient  = &Clientset{}
+)
+
+// Monok8sV1alpha1 retrieves the Monok8sV1alpha1Client
+func (c *Clientset) Monok8sV1alpha1() monok8sv1alpha1.Monok8sV1alpha1Interface {
+	return &fakemonok8sv1alpha1.FakeMonok8sV1alpha1{Fake: &c.Fake}
+}

clitools/pkg/generated/clientset/versioned/fake/doc.go

@@ -0,0 +1,6 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+// This package has the automatically generated fake clientset.
+package fake

clitools/pkg/generated/clientset/versioned/fake/register.go

@@ -0,0 +1,42 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package fake
+
+import (
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+	schema "k8s.io/apimachinery/pkg/runtime/schema"
+	serializer "k8s.io/apimachinery/pkg/runtime/serializer"
+	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
+)
+
+var scheme = runtime.NewScheme()
+var codecs = serializer.NewCodecFactory(scheme)
+
+var localSchemeBuilder = runtime.SchemeBuilder{
+	monok8sv1alpha1.AddToScheme,
+}
+
+// AddToScheme adds all types of this clientset into the given scheme. This allows composition
+// of clientsets, like in:
+//
+//	import (
+//	  "k8s.io/client-go/kubernetes"
+//	  clientsetscheme "k8s.io/client-go/kubernetes/scheme"
+//	  aggregatorclientsetscheme "k8s.io/kube-aggregator/pkg/client/clientset_generated/clientset/scheme"
+//	)
+//
+//	kclientset, _ := kubernetes.NewForConfig(c)
+//	_ = aggregatorclientsetscheme.AddToScheme(clientsetscheme.Scheme)
+//
+// After this, RawExtensions in Kubernetes types will serialize kube-aggregator types
+// correctly.
+var AddToScheme = localSchemeBuilder.AddToScheme
+
+func init() {
+	v1.AddToGroupVersion(scheme, schema.GroupVersion{Version: "v1"})
+	utilruntime.Must(AddToScheme(scheme))
+}

clitools/pkg/generated/clientset/versioned/scheme/doc.go

@@ -0,0 +1,6 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+// This package contains the scheme of the automatically generated clientset.
+package scheme

clitools/pkg/generated/clientset/versioned/scheme/register.go

@@ -0,0 +1,42 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package scheme
+
+import (
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+	schema "k8s.io/apimachinery/pkg/runtime/schema"
+	serializer "k8s.io/apimachinery/pkg/runtime/serializer"
+	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
+)
+
+var Scheme = runtime.NewScheme()
+var Codecs = serializer.NewCodecFactory(Scheme)
+var ParameterCodec = runtime.NewParameterCodec(Scheme)
+var localSchemeBuilder = runtime.SchemeBuilder{
+	monok8sv1alpha1.AddToScheme,
+}
+
+// AddToScheme adds all types of this clientset into the given scheme. This allows composition
+// of clientsets, like in:
+//
+//	import (
+//	  "k8s.io/client-go/kubernetes"
+//	  clientsetscheme "k8s.io/client-go/kubernetes/scheme"
+//	  aggregatorclientsetscheme "k8s.io/kube-aggregator/pkg/client/clientset_generated/clientset/scheme"
+//	)
+//
+//	kclientset, _ := kubernetes.NewForConfig(c)
+//	_ = aggregatorclientsetscheme.AddToScheme(clientsetscheme.Scheme)
+//
+// After this, RawExtensions in Kubernetes types will serialize kube-aggregator types
+// correctly.
+var AddToScheme = localSchemeBuilder.AddToScheme
+
+func init() {
+	v1.AddToGroupVersion(Scheme, schema.GroupVersion{Version: "v1"})
+	utilruntime.Must(AddToScheme(Scheme))
+}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/doc.go

@@ -0,0 +1,6 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+// This package has the automatically generated typed clients.
+package v1alpha1

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/fake/doc.go

@@ -0,0 +1,6 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+// Package fake has the automatically generated clients.
+package fake

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/fake/fake_monok8s_client.go

@@ -0,0 +1,30 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package fake
+
+import (
+	v1alpha1 "example.com/monok8s/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1"
+	rest "k8s.io/client-go/rest"
+	testing "k8s.io/client-go/testing"
+)
+
+type FakeMonok8sV1alpha1 struct {
+	*testing.Fake
+}
+
+func (c *FakeMonok8sV1alpha1) OSUpgrades(namespace string) v1alpha1.OSUpgradeInterface {
+	return newFakeOSUpgrades(c, namespace)
+}
+
+func (c *FakeMonok8sV1alpha1) OSUpgradeProgresses(namespace string) v1alpha1.OSUpgradeProgressInterface {
+	return newFakeOSUpgradeProgresses(c, namespace)
+}
+
+// RESTClient returns a RESTClient that is used to communicate
+// with API server by this client implementation.
+func (c *FakeMonok8sV1alpha1) RESTClient() rest.Interface {
+	var ret *rest.RESTClient
+	return ret
+}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/fake/fake_osupgrade.go

@@ -0,0 +1,36 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package fake
+
+import (
+	v1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	monok8sv1alpha1 "example.com/monok8s/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1"
+	gentype "k8s.io/client-go/gentype"
+)
+
+// fakeOSUpgrades implements OSUpgradeInterface
+type fakeOSUpgrades struct {
+	*gentype.FakeClientWithList[*v1alpha1.OSUpgrade, *v1alpha1.OSUpgradeList]
+	Fake *FakeMonok8sV1alpha1
+}
+
+func newFakeOSUpgrades(fake *FakeMonok8sV1alpha1, namespace string) monok8sv1alpha1.OSUpgradeInterface {
+	return &fakeOSUpgrades{
+		gentype.NewFakeClientWithList[*v1alpha1.OSUpgrade, *v1alpha1.OSUpgradeList](
+			fake.Fake,
+			namespace,
+			v1alpha1.SchemeGroupVersion.WithResource("osupgrades"),
+			v1alpha1.SchemeGroupVersion.WithKind("OSUpgrade"),
+			func() *v1alpha1.OSUpgrade { return &v1alpha1.OSUpgrade{} },
+			func() *v1alpha1.OSUpgradeList { return &v1alpha1.OSUpgradeList{} },
+			func(dst, src *v1alpha1.OSUpgradeList) { dst.ListMeta = src.ListMeta },
+			func(list *v1alpha1.OSUpgradeList) []*v1alpha1.OSUpgrade { return gentype.ToPointerSlice(list.Items) },
+			func(list *v1alpha1.OSUpgradeList, items []*v1alpha1.OSUpgrade) {
+				list.Items = gentype.FromPointerSlice(items)
+			},
+		),
+		fake,
+	}
+}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/fake/fake_osupgradeprogress.go

@@ -0,0 +1,38 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package fake
+
+import (
+	v1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	monok8sv1alpha1 "example.com/monok8s/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1"
+	gentype "k8s.io/client-go/gentype"
+)
+
+// fakeOSUpgradeProgresses implements OSUpgradeProgressInterface
+type fakeOSUpgradeProgresses struct {
+	*gentype.FakeClientWithList[*v1alpha1.OSUpgradeProgress, *v1alpha1.OSUpgradeProgressList]
+	Fake *FakeMonok8sV1alpha1
+}
+
+func newFakeOSUpgradeProgresses(fake *FakeMonok8sV1alpha1, namespace string) monok8sv1alpha1.OSUpgradeProgressInterface {
+	return &fakeOSUpgradeProgresses{
+		gentype.NewFakeClientWithList[*v1alpha1.OSUpgradeProgress, *v1alpha1.OSUpgradeProgressList](
+			fake.Fake,
+			namespace,
+			v1alpha1.SchemeGroupVersion.WithResource("osupgradeprogresses"),
+			v1alpha1.SchemeGroupVersion.WithKind("OSUpgradeProgress"),
+			func() *v1alpha1.OSUpgradeProgress { return &v1alpha1.OSUpgradeProgress{} },
+			func() *v1alpha1.OSUpgradeProgressList { return &v1alpha1.OSUpgradeProgressList{} },
+			func(dst, src *v1alpha1.OSUpgradeProgressList) { dst.ListMeta = src.ListMeta },
+			func(list *v1alpha1.OSUpgradeProgressList) []*v1alpha1.OSUpgradeProgress {
+				return gentype.ToPointerSlice(list.Items)
+			},
+			func(list *v1alpha1.OSUpgradeProgressList, items []*v1alpha1.OSUpgradeProgress) {
+				list.Items = gentype.FromPointerSlice(items)
+			},
+		),
+		fake,
+	}
+}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/generated_expansion.go

@@ -0,0 +1,9 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package v1alpha1
+
+type OSUpgradeExpansion interface{}
+
+type OSUpgradeProgressExpansion interface{}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/monok8s_client.go

@@ -0,0 +1,92 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	http "net/http"
+
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	scheme "example.com/monok8s/pkg/generated/clientset/versioned/scheme"
+	rest "k8s.io/client-go/rest"
+)
+
+type Monok8sV1alpha1Interface interface {
+	RESTClient() rest.Interface
+	OSUpgradesGetter
+	OSUpgradeProgressesGetter
+}
+
+// Monok8sV1alpha1Client is used to interact with features provided by the monok8s group.
+type Monok8sV1alpha1Client struct {
+	restClient rest.Interface
+}
+
+func (c *Monok8sV1alpha1Client) OSUpgrades(namespace string) OSUpgradeInterface {
+	return newOSUpgrades(c, namespace)
+}
+
+func (c *Monok8sV1alpha1Client) OSUpgradeProgresses(namespace string) OSUpgradeProgressInterface {
+	return newOSUpgradeProgresses(c, namespace)
+}
+
+// NewForConfig creates a new Monok8sV1alpha1Client for the given config.
+// NewForConfig is equivalent to NewForConfigAndClient(c, httpClient),
+// where httpClient was generated with rest.HTTPClientFor(c).
+func NewForConfig(c *rest.Config) (*Monok8sV1alpha1Client, error) {
+	config := *c
+	setConfigDefaults(&config)
+	httpClient, err := rest.HTTPClientFor(&config)
+	if err != nil {
+		return nil, err
+	}
+	return NewForConfigAndClient(&config, httpClient)
+}
+
+// NewForConfigAndClient creates a new Monok8sV1alpha1Client for the given config and http client.
+// Note the http client provided takes precedence over the configured transport values.
+func NewForConfigAndClient(c *rest.Config, h *http.Client) (*Monok8sV1alpha1Client, error) {
+	config := *c
+	setConfigDefaults(&config)
+	client, err := rest.RESTClientForConfigAndClient(&config, h)
+	if err != nil {
+		return nil, err
+	}
+	return &Monok8sV1alpha1Client{client}, nil
+}
+
+// NewForConfigOrDie creates a new Monok8sV1alpha1Client for the given config and
+// panics if there is an error in the config.
+func NewForConfigOrDie(c *rest.Config) *Monok8sV1alpha1Client {
+	client, err := NewForConfig(c)
+	if err != nil {
+		panic(err)
+	}
+	return client
+}
+
+// New creates a new Monok8sV1alpha1Client for the given RESTClient.
+func New(c rest.Interface) *Monok8sV1alpha1Client {
+	return &Monok8sV1alpha1Client{c}
+}
+
+func setConfigDefaults(config *rest.Config) {
+	gv := monok8sv1alpha1.SchemeGroupVersion
+	config.GroupVersion = &gv
+	config.APIPath = "/apis"
+	config.NegotiatedSerializer = rest.CodecFactoryForGeneratedClient(scheme.Scheme, scheme.Codecs).WithoutConversion()
+
+	if config.UserAgent == "" {
+		config.UserAgent = rest.DefaultKubernetesUserAgent()
+	}
+}
+
+// RESTClient returns a RESTClient that is used to communicate
+// with API server by this client implementation.
+func (c *Monok8sV1alpha1Client) RESTClient() rest.Interface {
+	if c == nil {
+		return nil
+	}
+	return c.restClient
+}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/osupgrade.go

@@ -0,0 +1,56 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	context "context"
+
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	scheme "example.com/monok8s/pkg/generated/clientset/versioned/scheme"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	types "k8s.io/apimachinery/pkg/types"
+	watch "k8s.io/apimachinery/pkg/watch"
+	gentype "k8s.io/client-go/gentype"
+)
+
+// OSUpgradesGetter has a method to return a OSUpgradeInterface.
+// A group's client should implement this interface.
+type OSUpgradesGetter interface {
+	OSUpgrades(namespace string) OSUpgradeInterface
+}
+
+// OSUpgradeInterface has methods to work with OSUpgrade resources.
+type OSUpgradeInterface interface {
+	Create(ctx context.Context, oSUpgrade *monok8sv1alpha1.OSUpgrade, opts v1.CreateOptions) (*monok8sv1alpha1.OSUpgrade, error)
+	Update(ctx context.Context, oSUpgrade *monok8sv1alpha1.OSUpgrade, opts v1.UpdateOptions) (*monok8sv1alpha1.OSUpgrade, error)
+	// Add a +genclient:noStatus comment above the type to avoid generating UpdateStatus().
+	UpdateStatus(ctx context.Context, oSUpgrade *monok8sv1alpha1.OSUpgrade, opts v1.UpdateOptions) (*monok8sv1alpha1.OSUpgrade, error)
+	Delete(ctx context.Context, name string, opts v1.DeleteOptions) error
+	DeleteCollection(ctx context.Context, opts v1.DeleteOptions, listOpts v1.ListOptions) error
+	Get(ctx context.Context, name string, opts v1.GetOptions) (*monok8sv1alpha1.OSUpgrade, error)
+	List(ctx context.Context, opts v1.ListOptions) (*monok8sv1alpha1.OSUpgradeList, error)
+	Watch(ctx context.Context, opts v1.ListOptions) (watch.Interface, error)
+	Patch(ctx context.Context, name string, pt types.PatchType, data []byte, opts v1.PatchOptions, subresources ...string) (result *monok8sv1alpha1.OSUpgrade, err error)
+	OSUpgradeExpansion
+}
+
+// oSUpgrades implements OSUpgradeInterface
+type oSUpgrades struct {
+	*gentype.ClientWithList[*monok8sv1alpha1.OSUpgrade, *monok8sv1alpha1.OSUpgradeList]
+}
+
+// newOSUpgrades returns a OSUpgrades
+func newOSUpgrades(c *Monok8sV1alpha1Client, namespace string) *oSUpgrades {
+	return &oSUpgrades{
+		gentype.NewClientWithList[*monok8sv1alpha1.OSUpgrade, *monok8sv1alpha1.OSUpgradeList](
+			"osupgrades",
+			c.RESTClient(),
+			scheme.ParameterCodec,
+			namespace,
+			func() *monok8sv1alpha1.OSUpgrade { return &monok8sv1alpha1.OSUpgrade{} },
+			func() *monok8sv1alpha1.OSUpgradeList { return &monok8sv1alpha1.OSUpgradeList{} },
+		),
+	}
+}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/osupgradeprogress.go

@@ -0,0 +1,56 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	context "context"
+
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	scheme "example.com/monok8s/pkg/generated/clientset/versioned/scheme"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	types "k8s.io/apimachinery/pkg/types"
+	watch "k8s.io/apimachinery/pkg/watch"
+	gentype "k8s.io/client-go/gentype"
+)
+
+// OSUpgradeProgressesGetter has a method to return a OSUpgradeProgressInterface.
+// A group's client should implement this interface.
+type OSUpgradeProgressesGetter interface {
+	OSUpgradeProgresses(namespace string) OSUpgradeProgressInterface
+}
+
+// OSUpgradeProgressInterface has methods to work with OSUpgradeProgress resources.
+type OSUpgradeProgressInterface interface {
+	Create(ctx context.Context, oSUpgradeProgress *monok8sv1alpha1.OSUpgradeProgress, opts v1.CreateOptions) (*monok8sv1alpha1.OSUpgradeProgress, error)
+	Update(ctx context.Context, oSUpgradeProgress *monok8sv1alpha1.OSUpgradeProgress, opts v1.UpdateOptions) (*monok8sv1alpha1.OSUpgradeProgress, error)
+	// Add a +genclient:noStatus comment above the type to avoid generating UpdateStatus().
+	UpdateStatus(ctx context.Context, oSUpgradeProgress *monok8sv1alpha1.OSUpgradeProgress, opts v1.UpdateOptions) (*monok8sv1alpha1.OSUpgradeProgress, error)
+	Delete(ctx context.Context, name string, opts v1.DeleteOptions) error
+	DeleteCollection(ctx context.Context, opts v1.DeleteOptions, listOpts v1.ListOptions) error
+	Get(ctx context.Context, name string, opts v1.GetOptions) (*monok8sv1alpha1.OSUpgradeProgress, error)
+	List(ctx context.Context, opts v1.ListOptions) (*monok8sv1alpha1.OSUpgradeProgressList, error)
+	Watch(ctx context.Context, opts v1.ListOptions) (watch.Interface, error)
+	Patch(ctx context.Context, name string, pt types.PatchType, data []byte, opts v1.PatchOptions, subresources ...string) (result *monok8sv1alpha1.OSUpgradeProgress, err error)
+	OSUpgradeProgressExpansion
+}
+
+// oSUpgradeProgresses implements OSUpgradeProgressInterface
+type oSUpgradeProgresses struct {
+	*gentype.ClientWithList[*monok8sv1alpha1.OSUpgradeProgress, *monok8sv1alpha1.OSUpgradeProgressList]
+}
+
+// newOSUpgradeProgresses returns a OSUpgradeProgresses
+func newOSUpgradeProgresses(c *Monok8sV1alpha1Client, namespace string) *oSUpgradeProgresses {
+	return &oSUpgradeProgresses{
+		gentype.NewClientWithList[*monok8sv1alpha1.OSUpgradeProgress, *monok8sv1alpha1.OSUpgradeProgressList](
+			"osupgradeprogresses",
+			c.RESTClient(),
+			scheme.ParameterCodec,
+			namespace,
+			func() *monok8sv1alpha1.OSUpgradeProgress { return &monok8sv1alpha1.OSUpgradeProgress{} },
+			func() *monok8sv1alpha1.OSUpgradeProgressList { return &monok8sv1alpha1.OSUpgradeProgressList{} },
+		),
+	}
+}

clitools/pkg/generated/informers/externalversions/factory.go

@@ -0,0 +1,249 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package externalversions
+
+import (
+	reflect "reflect"
+	sync "sync"
+	time "time"
+
+	versioned "example.com/monok8s/pkg/generated/clientset/versioned"
+	internalinterfaces "example.com/monok8s/pkg/generated/informers/externalversions/internalinterfaces"
+	monok8s "example.com/monok8s/pkg/generated/informers/externalversions/monok8s"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+	schema "k8s.io/apimachinery/pkg/runtime/schema"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// SharedInformerOption defines the functional option type for SharedInformerFactory.
+type SharedInformerOption func(*sharedInformerFactory) *sharedInformerFactory
+
+type sharedInformerFactory struct {
+	client           versioned.Interface
+	namespace        string
+	tweakListOptions internalinterfaces.TweakListOptionsFunc
+	lock             sync.Mutex
+	defaultResync    time.Duration
+	customResync     map[reflect.Type]time.Duration
+	transform        cache.TransformFunc
+
+	informers map[reflect.Type]cache.SharedIndexInformer
+	// startedInformers is used for tracking which informers have been started.
+	// This allows Start() to be called multiple times safely.
+	startedInformers map[reflect.Type]bool
+	// wg tracks how many goroutines were started.
+	wg sync.WaitGroup
+	// shuttingDown is true when Shutdown has been called. It may still be running
+	// because it needs to wait for goroutines.
+	shuttingDown bool
+}
+
+// WithCustomResyncConfig sets a custom resync period for the specified informer types.
+func WithCustomResyncConfig(resyncConfig map[v1.Object]time.Duration) SharedInformerOption {
+	return func(factory *sharedInformerFactory) *sharedInformerFactory {
+		for k, v := range resyncConfig {
+			factory.customResync[reflect.TypeOf(k)] = v
+		}
+		return factory
+	}
+}
+
+// WithTweakListOptions sets a custom filter on all listers of the configured SharedInformerFactory.
+func WithTweakListOptions(tweakListOptions internalinterfaces.TweakListOptionsFunc) SharedInformerOption {
+	return func(factory *sharedInformerFactory) *sharedInformerFactory {
+		factory.tweakListOptions = tweakListOptions
+		return factory
+	}
+}
+
+// WithNamespace limits the SharedInformerFactory to the specified namespace.
+func WithNamespace(namespace string) SharedInformerOption {
+	return func(factory *sharedInformerFactory) *sharedInformerFactory {
+		factory.namespace = namespace
+		return factory
+	}
+}
+
+// WithTransform sets a transform on all informers.
+func WithTransform(transform cache.TransformFunc) SharedInformerOption {
+	return func(factory *sharedInformerFactory) *sharedInformerFactory {
+		factory.transform = transform
+		return factory
+	}
+}
+
+// NewSharedInformerFactory constructs a new instance of sharedInformerFactory for all namespaces.
+func NewSharedInformerFactory(client versioned.Interface, defaultResync time.Duration) SharedInformerFactory {
+	return NewSharedInformerFactoryWithOptions(client, defaultResync)
+}
+
+// NewFilteredSharedInformerFactory constructs a new instance of sharedInformerFactory.
+// Listers obtained via this SharedInformerFactory will be subject to the same filters
+// as specified here.
+//
+// Deprecated: Please use NewSharedInformerFactoryWithOptions instead
+func NewFilteredSharedInformerFactory(client versioned.Interface, defaultResync time.Duration, namespace string, tweakListOptions internalinterfaces.TweakListOptionsFunc) SharedInformerFactory {
+	return NewSharedInformerFactoryWithOptions(client, defaultResync, WithNamespace(namespace), WithTweakListOptions(tweakListOptions))
+}
+
+// NewSharedInformerFactoryWithOptions constructs a new instance of a SharedInformerFactory with additional options.
+func NewSharedInformerFactoryWithOptions(client versioned.Interface, defaultResync time.Duration, options ...SharedInformerOption) SharedInformerFactory {
+	factory := &sharedInformerFactory{
+		client:           client,
+		namespace:        v1.NamespaceAll,
+		defaultResync:    defaultResync,
+		informers:        make(map[reflect.Type]cache.SharedIndexInformer),
+		startedInformers: make(map[reflect.Type]bool),
+		customResync:     make(map[reflect.Type]time.Duration),
+	}
+
+	// Apply all options
+	for _, opt := range options {
+		factory = opt(factory)
+	}
+
+	return factory
+}
+
+func (f *sharedInformerFactory) Start(stopCh <-chan struct{}) {
+	f.lock.Lock()
+	defer f.lock.Unlock()
+
+	if f.shuttingDown {
+		return
+	}
+
+	for informerType, informer := range f.informers {
+		if !f.startedInformers[informerType] {
+			f.wg.Add(1)
+			// We need a new variable in each loop iteration,
+			// otherwise the goroutine would use the loop variable
+			// and that keeps changing.
+			informer := informer
+			go func() {
+				defer f.wg.Done()
+				informer.Run(stopCh)
+			}()
+			f.startedInformers[informerType] = true
+		}
+	}
+}
+
+func (f *sharedInformerFactory) Shutdown() {
+	f.lock.Lock()
+	f.shuttingDown = true
+	f.lock.Unlock()
+
+	// Will return immediately if there is nothing to wait for.
+	f.wg.Wait()
+}
+
+func (f *sharedInformerFactory) WaitForCacheSync(stopCh <-chan struct{}) map[reflect.Type]bool {
+	informers := func() map[reflect.Type]cache.SharedIndexInformer {
+		f.lock.Lock()
+		defer f.lock.Unlock()
+
+		informers := map[reflect.Type]cache.SharedIndexInformer{}
+		for informerType, informer := range f.informers {
+			if f.startedInformers[informerType] {
+				informers[informerType] = informer
+			}
+		}
+		return informers
+	}()
+
+	res := map[reflect.Type]bool{}
+	for informType, informer := range informers {
+		res[informType] = cache.WaitForCacheSync(stopCh, informer.HasSynced)
+	}
+	return res
+}
+
+// InformerFor returns the SharedIndexInformer for obj using an internal
+// client.
+func (f *sharedInformerFactory) InformerFor(obj runtime.Object, newFunc internalinterfaces.NewInformerFunc) cache.SharedIndexInformer {
+	f.lock.Lock()
+	defer f.lock.Unlock()
+
+	informerType := reflect.TypeOf(obj)
+	informer, exists := f.informers[informerType]
+	if exists {
+		return informer
+	}
+
+	resyncPeriod, exists := f.customResync[informerType]
+	if !exists {
+		resyncPeriod = f.defaultResync
+	}
+
+	informer = newFunc(f.client, resyncPeriod)
+	informer.SetTransform(f.transform)
+	f.informers[informerType] = informer
+
+	return informer
+}
+
+// SharedInformerFactory provides shared informers for resources in all known
+// API group versions.
+//
+// It is typically used like this:
+//
+//	ctx, cancel := context.WithCancel(context.Background())
+//	defer cancel()
+//	factory := NewSharedInformerFactory(client, resyncPeriod)
+//	defer factory.WaitForStop()    // Returns immediately if nothing was started.
+//	genericInformer := factory.ForResource(resource)
+//	typedInformer := factory.SomeAPIGroup().V1().SomeType()
+//	factory.Start(ctx.Done())          // Start processing these informers.
+//	synced := factory.WaitForCacheSync(ctx.Done())
+//	for v, ok := range synced {
+//	    if !ok {
+//	        fmt.Fprintf(os.Stderr, "caches failed to sync: %v", v)
+//	        return
+//	    }
+//	}
+//
+//	// Creating informers can also be created after Start, but then
+//	// Start must be called again:
+//	anotherGenericInformer := factory.ForResource(resource)
+//	factory.Start(ctx.Done())
+type SharedInformerFactory interface {
+	internalinterfaces.SharedInformerFactory
+
+	// Start initializes all requested informers. They are handled in goroutines
+	// which run until the stop channel gets closed.
+	// Warning: Start does not block. When run in a go-routine, it will race with a later WaitForCacheSync.
+	Start(stopCh <-chan struct{})
+
+	// Shutdown marks a factory as shutting down. At that point no new
+	// informers can be started anymore and Start will return without
+	// doing anything.
+	//
+	// In addition, Shutdown blocks until all goroutines have terminated. For that
+	// to happen, the close channel(s) that they were started with must be closed,
+	// either before Shutdown gets called or while it is waiting.
+	//
+	// Shutdown may be called multiple times, even concurrently. All such calls will
+	// block until all goroutines have terminated.
+	Shutdown()
+
+	// WaitForCacheSync blocks until all started informers' caches were synced
+	// or the stop channel gets closed.
+	WaitForCacheSync(stopCh <-chan struct{}) map[reflect.Type]bool
+
+	// ForResource gives generic access to a shared informer of the matching type.
+	ForResource(resource schema.GroupVersionResource) (GenericInformer, error)
+
+	// InformerFor returns the SharedIndexInformer for obj using an internal
+	// client.
+	InformerFor(obj runtime.Object, newFunc internalinterfaces.NewInformerFunc) cache.SharedIndexInformer
+
+	Monok8s() monok8s.Interface
+}
+
+func (f *sharedInformerFactory) Monok8s() monok8s.Interface {
+	return monok8s.New(f, f.namespace, f.tweakListOptions)
+}

clitools/pkg/generated/informers/externalversions/generic.go

@@ -0,0 +1,50 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package externalversions
+
+import (
+	fmt "fmt"
+
+	v1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	schema "k8s.io/apimachinery/pkg/runtime/schema"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// GenericInformer is type of SharedIndexInformer which will locate and delegate to other
+// sharedInformers based on type
+type GenericInformer interface {
+	Informer() cache.SharedIndexInformer
+	Lister() cache.GenericLister
+}
+
+type genericInformer struct {
+	informer cache.SharedIndexInformer
+	resource schema.GroupResource
+}
+
+// Informer returns the SharedIndexInformer.
+func (f *genericInformer) Informer() cache.SharedIndexInformer {
+	return f.informer
+}
+
+// Lister returns the GenericLister.
+func (f *genericInformer) Lister() cache.GenericLister {
+	return cache.NewGenericLister(f.Informer().GetIndexer(), f.resource)
+}
+
+// ForResource gives generic access to a shared informer of the matching type
+// TODO extend this to unknown resources with a client pool
+func (f *sharedInformerFactory) ForResource(resource schema.GroupVersionResource) (GenericInformer, error) {
+	switch resource {
+	// Group=monok8s, Version=v1alpha1
+	case v1alpha1.SchemeGroupVersion.WithResource("osupgrades"):
+		return &genericInformer{resource: resource.GroupResource(), informer: f.Monok8s().V1alpha1().OSUpgrades().Informer()}, nil
+	case v1alpha1.SchemeGroupVersion.WithResource("osupgradeprogresses"):
+		return &genericInformer{resource: resource.GroupResource(), informer: f.Monok8s().V1alpha1().OSUpgradeProgresses().Informer()}, nil
+
+	}
+
+	return nil, fmt.Errorf("no informer found for %v", resource)
+}

clitools/pkg/generated/informers/externalversions/internalinterfaces/factory_interfaces.go

@@ -0,0 +1,26 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package internalinterfaces
+
+import (
+	time "time"
+
+	versioned "example.com/monok8s/pkg/generated/clientset/versioned"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// NewInformerFunc takes versioned.Interface and time.Duration to return a SharedIndexInformer.
+type NewInformerFunc func(versioned.Interface, time.Duration) cache.SharedIndexInformer
+
+// SharedInformerFactory a small interface to allow for adding an informer without an import cycle
+type SharedInformerFactory interface {
+	Start(stopCh <-chan struct{})
+	InformerFor(obj runtime.Object, newFunc NewInformerFunc) cache.SharedIndexInformer
+}
+
+// TweakListOptionsFunc is a function that transforms a v1.ListOptions.
+type TweakListOptionsFunc func(*v1.ListOptions)

clitools/pkg/generated/informers/externalversions/monok8s/interface.go

@@ -0,0 +1,32 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package monok8s
+
+import (
+	internalinterfaces "example.com/monok8s/pkg/generated/informers/externalversions/internalinterfaces"
+	v1alpha1 "example.com/monok8s/pkg/generated/informers/externalversions/monok8s/v1alpha1"
+)
+
+// Interface provides access to each of this group's versions.
+type Interface interface {
+	// V1alpha1 provides access to shared informers for resources in V1alpha1.
+	V1alpha1() v1alpha1.Interface
+}
+
+type group struct {
+	factory          internalinterfaces.SharedInformerFactory
+	namespace        string
+	tweakListOptions internalinterfaces.TweakListOptionsFunc
+}
+
+// New returns a new Interface.
+func New(f internalinterfaces.SharedInformerFactory, namespace string, tweakListOptions internalinterfaces.TweakListOptionsFunc) Interface {
+	return &group{factory: f, namespace: namespace, tweakListOptions: tweakListOptions}
+}
+
+// V1alpha1 returns a new v1alpha1.Interface.
+func (g *group) V1alpha1() v1alpha1.Interface {
+	return v1alpha1.New(g.factory, g.namespace, g.tweakListOptions)
+}

clitools/pkg/generated/informers/externalversions/monok8s/v1alpha1/interface.go

@@ -0,0 +1,38 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	internalinterfaces "example.com/monok8s/pkg/generated/informers/externalversions/internalinterfaces"
+)
+
+// Interface provides access to all the informers in this group version.
+type Interface interface {
+	// OSUpgrades returns a OSUpgradeInformer.
+	OSUpgrades() OSUpgradeInformer
+	// OSUpgradeProgresses returns a OSUpgradeProgressInformer.
+	OSUpgradeProgresses() OSUpgradeProgressInformer
+}
+
+type version struct {
+	factory          internalinterfaces.SharedInformerFactory
+	namespace        string
+	tweakListOptions internalinterfaces.TweakListOptionsFunc
+}
+
+// New returns a new Interface.
+func New(f internalinterfaces.SharedInformerFactory, namespace string, tweakListOptions internalinterfaces.TweakListOptionsFunc) Interface {
+	return &version{factory: f, namespace: namespace, tweakListOptions: tweakListOptions}
+}
+
+// OSUpgrades returns a OSUpgradeInformer.
+func (v *version) OSUpgrades() OSUpgradeInformer {
+	return &oSUpgradeInformer{factory: v.factory, namespace: v.namespace, tweakListOptions: v.tweakListOptions}
+}
+
+// OSUpgradeProgresses returns a OSUpgradeProgressInformer.
+func (v *version) OSUpgradeProgresses() OSUpgradeProgressInformer {
+	return &oSUpgradeProgressInformer{factory: v.factory, namespace: v.namespace, tweakListOptions: v.tweakListOptions}
+}

clitools/pkg/generated/informers/externalversions/monok8s/v1alpha1/osupgrade.go

@@ -0,0 +1,88 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	context "context"
+	time "time"
+
+	apismonok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	versioned "example.com/monok8s/pkg/generated/clientset/versioned"
+	internalinterfaces "example.com/monok8s/pkg/generated/informers/externalversions/internalinterfaces"
+	monok8sv1alpha1 "example.com/monok8s/pkg/generated/listers/monok8s/v1alpha1"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+	watch "k8s.io/apimachinery/pkg/watch"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// OSUpgradeInformer provides access to a shared informer and lister for
+// OSUpgrades.
+type OSUpgradeInformer interface {
+	Informer() cache.SharedIndexInformer
+	Lister() monok8sv1alpha1.OSUpgradeLister
+}
+
+type oSUpgradeInformer struct {
+	factory          internalinterfaces.SharedInformerFactory
+	tweakListOptions internalinterfaces.TweakListOptionsFunc
+	namespace        string
+}
+
+// NewOSUpgradeInformer constructs a new informer for OSUpgrade type.
+// Always prefer using an informer factory to get a shared informer instead of getting an independent
+// one. This reduces memory footprint and number of connections to the server.
+func NewOSUpgradeInformer(client versioned.Interface, namespace string, resyncPeriod time.Duration, indexers cache.Indexers) cache.SharedIndexInformer {
+	return NewFilteredOSUpgradeInformer(client, namespace, resyncPeriod, indexers, nil)
+}
+
+// NewFilteredOSUpgradeInformer constructs a new informer for OSUpgrade type.
+// Always prefer using an informer factory to get a shared informer instead of getting an independent
+// one. This reduces memory footprint and number of connections to the server.
+func NewFilteredOSUpgradeInformer(client versioned.Interface, namespace string, resyncPeriod time.Duration, indexers cache.Indexers, tweakListOptions internalinterfaces.TweakListOptionsFunc) cache.SharedIndexInformer {
+	return cache.NewSharedIndexInformer(
+		cache.ToListWatcherWithWatchListSemantics(&cache.ListWatch{
+			ListFunc: func(options v1.ListOptions) (runtime.Object, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgrades(namespace).List(context.Background(), options)
+			},
+			WatchFunc: func(options v1.ListOptions) (watch.Interface, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgrades(namespace).Watch(context.Background(), options)
+			},
+			ListWithContextFunc: func(ctx context.Context, options v1.ListOptions) (runtime.Object, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgrades(namespace).List(ctx, options)
+			},
+			WatchFuncWithContext: func(ctx context.Context, options v1.ListOptions) (watch.Interface, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgrades(namespace).Watch(ctx, options)
+			},
+		}, client),
+		&apismonok8sv1alpha1.OSUpgrade{},
+		resyncPeriod,
+		indexers,
+	)
+}
+
+func (f *oSUpgradeInformer) defaultInformer(client versioned.Interface, resyncPeriod time.Duration) cache.SharedIndexInformer {
+	return NewFilteredOSUpgradeInformer(client, f.namespace, resyncPeriod, cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc}, f.tweakListOptions)
+}
+
+func (f *oSUpgradeInformer) Informer() cache.SharedIndexInformer {
+	return f.factory.InformerFor(&apismonok8sv1alpha1.OSUpgrade{}, f.defaultInformer)
+}
+
+func (f *oSUpgradeInformer) Lister() monok8sv1alpha1.OSUpgradeLister {
+	return monok8sv1alpha1.NewOSUpgradeLister(f.Informer().GetIndexer())
+}

clitools/pkg/generated/informers/externalversions/monok8s/v1alpha1/osupgradeprogress.go

@@ -0,0 +1,88 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	context "context"
+	time "time"
+
+	apismonok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	versioned "example.com/monok8s/pkg/generated/clientset/versioned"
+	internalinterfaces "example.com/monok8s/pkg/generated/informers/externalversions/internalinterfaces"
+	monok8sv1alpha1 "example.com/monok8s/pkg/generated/listers/monok8s/v1alpha1"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+	watch "k8s.io/apimachinery/pkg/watch"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// OSUpgradeProgressInformer provides access to a shared informer and lister for
+// OSUpgradeProgresses.
+type OSUpgradeProgressInformer interface {
+	Informer() cache.SharedIndexInformer
+	Lister() monok8sv1alpha1.OSUpgradeProgressLister
+}
+
+type oSUpgradeProgressInformer struct {
+	factory          internalinterfaces.SharedInformerFactory
+	tweakListOptions internalinterfaces.TweakListOptionsFunc
+	namespace        string
+}
+
+// NewOSUpgradeProgressInformer constructs a new informer for OSUpgradeProgress type.
+// Always prefer using an informer factory to get a shared informer instead of getting an independent
+// one. This reduces memory footprint and number of connections to the server.
+func NewOSUpgradeProgressInformer(client versioned.Interface, namespace string, resyncPeriod time.Duration, indexers cache.Indexers) cache.SharedIndexInformer {
+	return NewFilteredOSUpgradeProgressInformer(client, namespace, resyncPeriod, indexers, nil)
+}
+
+// NewFilteredOSUpgradeProgressInformer constructs a new informer for OSUpgradeProgress type.
+// Always prefer using an informer factory to get a shared informer instead of getting an independent
+// one. This reduces memory footprint and number of connections to the server.
+func NewFilteredOSUpgradeProgressInformer(client versioned.Interface, namespace string, resyncPeriod time.Duration, indexers cache.Indexers, tweakListOptions internalinterfaces.TweakListOptionsFunc) cache.SharedIndexInformer {
+	return cache.NewSharedIndexInformer(
+		cache.ToListWatcherWithWatchListSemantics(&cache.ListWatch{
+			ListFunc: func(options v1.ListOptions) (runtime.Object, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgradeProgresses(namespace).List(context.Background(), options)
+			},
+			WatchFunc: func(options v1.ListOptions) (watch.Interface, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgradeProgresses(namespace).Watch(context.Background(), options)
+			},
+			ListWithContextFunc: func(ctx context.Context, options v1.ListOptions) (runtime.Object, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgradeProgresses(namespace).List(ctx, options)
+			},
+			WatchFuncWithContext: func(ctx context.Context, options v1.ListOptions) (watch.Interface, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgradeProgresses(namespace).Watch(ctx, options)
+			},
+		}, client),
+		&apismonok8sv1alpha1.OSUpgradeProgress{},
+		resyncPeriod,
+		indexers,
+	)
+}
+
+func (f *oSUpgradeProgressInformer) defaultInformer(client versioned.Interface, resyncPeriod time.Duration) cache.SharedIndexInformer {
+	return NewFilteredOSUpgradeProgressInformer(client, f.namespace, resyncPeriod, cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc}, f.tweakListOptions)
+}
+
+func (f *oSUpgradeProgressInformer) Informer() cache.SharedIndexInformer {
+	return f.factory.InformerFor(&apismonok8sv1alpha1.OSUpgradeProgress{}, f.defaultInformer)
+}
+
+func (f *oSUpgradeProgressInformer) Lister() monok8sv1alpha1.OSUpgradeProgressLister {
+	return monok8sv1alpha1.NewOSUpgradeProgressLister(f.Informer().GetIndexer())
+}

clitools/pkg/generated/listers/monok8s/v1alpha1/expansion_generated.go

@@ -0,0 +1,21 @@
+/* MIT License */
+
+// Code generated by lister-gen. DO NOT EDIT.
+
+package v1alpha1
+
+// OSUpgradeListerExpansion allows custom methods to be added to
+// OSUpgradeLister.
+type OSUpgradeListerExpansion interface{}
+
+// OSUpgradeNamespaceListerExpansion allows custom methods to be added to
+// OSUpgradeNamespaceLister.
+type OSUpgradeNamespaceListerExpansion interface{}
+
+// OSUpgradeProgressListerExpansion allows custom methods to be added to
+// OSUpgradeProgressLister.
+type OSUpgradeProgressListerExpansion interface{}
+
+// OSUpgradeProgressNamespaceListerExpansion allows custom methods to be added to
+// OSUpgradeProgressNamespaceLister.
+type OSUpgradeProgressNamespaceListerExpansion interface{}

clitools/pkg/generated/listers/monok8s/v1alpha1/osupgrade.go

@@ -0,0 +1,56 @@
+/* MIT License */
+
+// Code generated by lister-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	labels "k8s.io/apimachinery/pkg/labels"
+	listers "k8s.io/client-go/listers"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// OSUpgradeLister helps list OSUpgrades.
+// All objects returned here must be treated as read-only.
+type OSUpgradeLister interface {
+	// List lists all OSUpgrades in the indexer.
+	// Objects returned here must be treated as read-only.
+	List(selector labels.Selector) (ret []*monok8sv1alpha1.OSUpgrade, err error)
+	// OSUpgrades returns an object that can list and get OSUpgrades.
+	OSUpgrades(namespace string) OSUpgradeNamespaceLister
+	OSUpgradeListerExpansion
+}
+
+// oSUpgradeLister implements the OSUpgradeLister interface.
+type oSUpgradeLister struct {
+	listers.ResourceIndexer[*monok8sv1alpha1.OSUpgrade]
+}
+
+// NewOSUpgradeLister returns a new OSUpgradeLister.
+func NewOSUpgradeLister(indexer cache.Indexer) OSUpgradeLister {
+	return &oSUpgradeLister{listers.New[*monok8sv1alpha1.OSUpgrade](indexer, monok8sv1alpha1.Resource("osupgrade"))}
+}
+
+// OSUpgrades returns an object that can list and get OSUpgrades.
+func (s *oSUpgradeLister) OSUpgrades(namespace string) OSUpgradeNamespaceLister {
+	return oSUpgradeNamespaceLister{listers.NewNamespaced[*monok8sv1alpha1.OSUpgrade](s.ResourceIndexer, namespace)}
+}
+
+// OSUpgradeNamespaceLister helps list and get OSUpgrades.
+// All objects returned here must be treated as read-only.
+type OSUpgradeNamespaceLister interface {
+	// List lists all OSUpgrades in the indexer for a given namespace.
+	// Objects returned here must be treated as read-only.
+	List(selector labels.Selector) (ret []*monok8sv1alpha1.OSUpgrade, err error)
+	// Get retrieves the OSUpgrade from the indexer for a given namespace and name.
+	// Objects returned here must be treated as read-only.
+	Get(name string) (*monok8sv1alpha1.OSUpgrade, error)
+	OSUpgradeNamespaceListerExpansion
+}
+
+// oSUpgradeNamespaceLister implements the OSUpgradeNamespaceLister
+// interface.
+type oSUpgradeNamespaceLister struct {
+	listers.ResourceIndexer[*monok8sv1alpha1.OSUpgrade]
+}

clitools/pkg/generated/listers/monok8s/v1alpha1/osupgradeprogress.go

@@ -0,0 +1,56 @@
+/* MIT License */
+
+// Code generated by lister-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	labels "k8s.io/apimachinery/pkg/labels"
+	listers "k8s.io/client-go/listers"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// OSUpgradeProgressLister helps list OSUpgradeProgresses.
+// All objects returned here must be treated as read-only.
+type OSUpgradeProgressLister interface {
+	// List lists all OSUpgradeProgresses in the indexer.
+	// Objects returned here must be treated as read-only.
+	List(selector labels.Selector) (ret []*monok8sv1alpha1.OSUpgradeProgress, err error)
+	// OSUpgradeProgresses returns an object that can list and get OSUpgradeProgresses.
+	OSUpgradeProgresses(namespace string) OSUpgradeProgressNamespaceLister
+	OSUpgradeProgressListerExpansion
+}
+
+// oSUpgradeProgressLister implements the OSUpgradeProgressLister interface.
+type oSUpgradeProgressLister struct {
+	listers.ResourceIndexer[*monok8sv1alpha1.OSUpgradeProgress]
+}
+
+// NewOSUpgradeProgressLister returns a new OSUpgradeProgressLister.
+func NewOSUpgradeProgressLister(indexer cache.Indexer) OSUpgradeProgressLister {
+	return &oSUpgradeProgressLister{listers.New[*monok8sv1alpha1.OSUpgradeProgress](indexer, monok8sv1alpha1.Resource("osupgradeprogress"))}
+}
+
+// OSUpgradeProgresses returns an object that can list and get OSUpgradeProgresses.
+func (s *oSUpgradeProgressLister) OSUpgradeProgresses(namespace string) OSUpgradeProgressNamespaceLister {
+	return oSUpgradeProgressNamespaceLister{listers.NewNamespaced[*monok8sv1alpha1.OSUpgradeProgress](s.ResourceIndexer, namespace)}
+}
+
+// OSUpgradeProgressNamespaceLister helps list and get OSUpgradeProgresses.
+// All objects returned here must be treated as read-only.
+type OSUpgradeProgressNamespaceLister interface {
+	// List lists all OSUpgradeProgresses in the indexer for a given namespace.
+	// Objects returned here must be treated as read-only.
+	List(selector labels.Selector) (ret []*monok8sv1alpha1.OSUpgradeProgress, err error)
+	// Get retrieves the OSUpgradeProgress from the indexer for a given namespace and name.
+	// Objects returned here must be treated as read-only.
+	Get(name string) (*monok8sv1alpha1.OSUpgradeProgress, error)
+	OSUpgradeProgressNamespaceListerExpansion
+}
+
+// oSUpgradeProgressNamespaceLister implements the OSUpgradeProgressNamespaceLister
+// interface.
+type oSUpgradeProgressNamespaceLister struct {
+	listers.ResourceIndexer[*monok8sv1alpha1.OSUpgradeProgress]
+}

clitools/pkg/kube/clients.go

@@ -3,11 +3,12 @@ package kube
 import (
 	"fmt"
 
+	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	monoclientset "example.com/monok8s/pkg/generated/clientset/versioned"
 	apiextensionsclientset "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
 	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
 	kubernetes "k8s.io/client-go/kubernetes"
 	clientgoscheme "k8s.io/client-go/kubernetes/scheme"
-	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
 
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/cli-runtime/pkg/genericclioptions"
@@ -21,6 +22,7 @@ type Clients struct {
 	Dynamic          dynamic.Interface
 	APIExtensions    apiextensionsclientset.Interface
 	RESTClientGetter genericclioptions.RESTClientGetter
+	MonoKS           monoclientset.Interface
 }
 
 func NewClients(flags *genericclioptions.ConfigFlags) (*Clients, error) {
@@ -40,7 +42,19 @@ func NewClients(flags *genericclioptions.ConfigFlags) (*Clients, error) {
 	if err != nil {
 		return nil, fmt.Errorf("build apiextensions client: %w", err)
 	}
-	return &Clients{Config: cfg, Kubernetes: kubeClient, Dynamic: dyn, APIExtensions: ext, RESTClientGetter: flags}, nil
+	mono, err := monoclientset.NewForConfig(cfg)
+	if err != nil {
+		return nil, fmt.Errorf("build monok8s client: %w", err)
+	}
+
+	return &Clients{
+		Config:           cfg,
+		Kubernetes:       kubeClient,
+		Dynamic:          dyn,
+		APIExtensions:    ext,
+		RESTClientGetter: flags,
+		MonoKS:           mono,
+	}, nil
 }
 
 func NewClientsFromKubeconfig(kubeconfigPath string) (*Clients, error) {

clitools/pkg/node/agent.go

@@ -3,39 +3,27 @@ package node
 import (
 	"context"
 	"fmt"
-	"reflect"
 	"strings"
 
-	appsv1 "k8s.io/api/apps/v1"
-	corev1 "k8s.io/api/core/v1"
-	rbacv1 "k8s.io/api/rbac/v1"
-	apierrors "k8s.io/apimachinery/pkg/api/errors"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/client-go/kubernetes"
 	"k8s.io/klog/v2"
 
-	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
 	"example.com/monok8s/pkg/kube"
+	"example.com/monok8s/pkg/render"
 	templates "example.com/monok8s/pkg/templates"
 )
 
-const (
-	controlAgentName              = "control-agent"
-	controlAgentNodeSelectorValue = "true"
-	controlAgentImage             = "localhost/monok8s/control-agent:dev"
-	kubeconfig                    = "/etc/kubernetes/admin.conf"
-)
+const kubeconfig = "/etc/kubernetes/admin.conf"
 
-func ApplyControlAgentDaemonSetResources(ctx context.Context, n *NodeContext) error {
-	// Only the control-plane should bootstrap this DaemonSet definition.
-	// And only when the feature is enabled.
-	if strings.TrimSpace(n.Config.Spec.ClusterRole) != "control-plane" || !n.Config.Spec.EnableControlAgent {
-		klog.InfoS("skipped for", "clusterRole", n.Config.Spec.ClusterRole, "enableControlAgent", n.Config.Spec.EnableControlAgent)
+func ApplyNodeControlDaemonSetResources(ctx context.Context, n *NodeContext) error {
+	if strings.TrimSpace(n.Config.Spec.ClusterRole) != "control-plane" || !n.Config.Spec.EnableNodeControl {
+		klog.InfoS("skipped for",
+			"clusterRole", n.Config.Spec.ClusterRole,
+			"enableNodeAgent", n.Config.Spec.EnableNodeControl,
+		)
 		return nil
 	}
 
-	err := ApplyCRDs(ctx, n)
-	if err != nil {
+	if err := ApplyCRDs(ctx, n); err != nil {
 		return err
 	}
 
@@ -49,320 +37,13 @@ func ApplyControlAgentDaemonSetResources(ctx context.Context, n *NodeContext) er
 		return fmt.Errorf("build kube clients from %s: %w", kubeconfig, err)
 	}
 
-	labels := map[string]string{
-		"app.kubernetes.io/name":       controlAgentName,
-		"app.kubernetes.io/component":  "agent",
-		"app.kubernetes.io/part-of":    "monok8s",
-		"app.kubernetes.io/managed-by": "ctl",
+	conf := render.AgentConf{
+		Namespace: namespace,
 	}
 
-	kubeClient := clients.Kubernetes
-
-	if err := applyControlAgentServiceAccount(ctx, kubeClient, namespace, labels); err != nil {
-		return fmt.Errorf("apply serviceaccount: %w", err)
-	}
-	if err := applyControlAgentClusterRole(ctx, kubeClient, labels); err != nil {
-		return fmt.Errorf("apply clusterrole: %w", err)
-	}
-	if err := applyControlAgentClusterRoleBinding(ctx, kubeClient, namespace, labels); err != nil {
-		return fmt.Errorf("apply clusterrolebinding: %w", err)
-	}
-	if err := applyControlAgentDaemonSet(ctx, kubeClient, namespace, labels); err != nil {
-		return fmt.Errorf("apply daemonset: %w", err)
+	if err := render.ApplyAgentDaemonSets(ctx, clients.Kubernetes, conf); err != nil {
+		return fmt.Errorf("apply node agent daemonset resources: %w", err)
 	}
 
 	return nil
 }
-
-func applyControlAgentServiceAccount(ctx context.Context, kubeClient kubernetes.Interface, namespace string, labels map[string]string) error {
-	want := &corev1.ServiceAccount{
-		ObjectMeta: metav1.ObjectMeta{
-			Name:      controlAgentName,
-			Namespace: namespace,
-			Labels:    labels,
-		},
-	}
-
-	existing, err := kubeClient.CoreV1().ServiceAccounts(namespace).Get(ctx, controlAgentName, metav1.GetOptions{})
-	if apierrors.IsNotFound(err) {
-		_, err = kubeClient.CoreV1().ServiceAccounts(namespace).Create(ctx, want, metav1.CreateOptions{})
-		return err
-	}
-	if err != nil {
-		return err
-	}
-
-	changed := false
-	if !reflect.DeepEqual(existing.Labels, want.Labels) {
-		existing.Labels = want.Labels
-		changed = true
-	}
-
-	if !changed {
-		return nil
-	}
-
-	_, err = kubeClient.CoreV1().ServiceAccounts(namespace).Update(ctx, existing, metav1.UpdateOptions{})
-	return err
-}
-
-func applyControlAgentClusterRole(ctx context.Context, kubeClient kubernetes.Interface, labels map[string]string) error {
-	wantRules := []rbacv1.PolicyRule{
-		{
-			APIGroups: []string{monov1alpha1.Group},
-			Resources: []string{"osupgrades"},
-			Verbs:     []string{"get", "list", "watch"},
-		},
-		{
-			APIGroups: []string{monov1alpha1.Group},
-			Resources: []string{"osupgrades/status"},
-			Verbs:     []string{"get", "patch", "update"},
-		},
-		{
-			APIGroups: []string{""},
-			Resources: []string{"nodes"},
-			Verbs:     []string{"get", "list", "watch"},
-		},
-	}
-
-	want := &rbacv1.ClusterRole{
-		ObjectMeta: metav1.ObjectMeta{
-			Name:   controlAgentName,
-			Labels: labels,
-		},
-		Rules: wantRules,
-	}
-
-	existing, err := kubeClient.RbacV1().ClusterRoles().Get(ctx, controlAgentName, metav1.GetOptions{})
-	if apierrors.IsNotFound(err) {
-		_, err = kubeClient.RbacV1().ClusterRoles().Create(ctx, want, metav1.CreateOptions{})
-		return err
-	}
-	if err != nil {
-		return err
-	}
-
-	changed := false
-	if !reflect.DeepEqual(existing.Labels, want.Labels) {
-		existing.Labels = want.Labels
-		changed = true
-	}
-	if !reflect.DeepEqual(existing.Rules, want.Rules) {
-		existing.Rules = want.Rules
-		changed = true
-	}
-
-	if !changed {
-		return nil
-	}
-
-	_, err = kubeClient.RbacV1().ClusterRoles().Update(ctx, existing, metav1.UpdateOptions{})
-	return err
-}
-
-func applyControlAgentClusterRoleBinding(ctx context.Context, kubeClient kubernetes.Interface, namespace string, labels map[string]string) error {
-	wantRoleRef := rbacv1.RoleRef{
-		APIGroup: rbacv1.GroupName,
-		Kind:     "ClusterRole",
-		Name:     controlAgentName,
-	}
-	wantSubjects := []rbacv1.Subject{
-		{
-			Kind:      "ServiceAccount",
-			Name:      controlAgentName,
-			Namespace: namespace,
-		},
-	}
-
-	want := &rbacv1.ClusterRoleBinding{
-		ObjectMeta: metav1.ObjectMeta{
-			Name:   controlAgentName,
-			Labels: labels,
-		},
-		RoleRef:  wantRoleRef,
-		Subjects: wantSubjects,
-	}
-
-	existing, err := kubeClient.RbacV1().ClusterRoleBindings().Get(ctx, controlAgentName, metav1.GetOptions{})
-	if apierrors.IsNotFound(err) {
-		_, err = kubeClient.RbacV1().ClusterRoleBindings().Create(ctx, want, metav1.CreateOptions{})
-		return err
-	}
-	if err != nil {
-		return err
-	}
-
-	// roleRef is immutable. If it differs, fail loudly instead of pretending we can patch it.
-	if !reflect.DeepEqual(existing.RoleRef, want.RoleRef) {
-		return fmt.Errorf("existing ClusterRoleBinding %q has different roleRef and must be recreated", controlAgentName)
-	}
-
-	changed := false
-	if !reflect.DeepEqual(existing.Labels, want.Labels) {
-		existing.Labels = want.Labels
-		changed = true
-	}
-	if !reflect.DeepEqual(existing.Subjects, want.Subjects) {
-		existing.Subjects = want.Subjects
-		changed = true
-	}
-
-	if !changed {
-		return nil
-	}
-
-	_, err = kubeClient.RbacV1().ClusterRoleBindings().Update(ctx, existing, metav1.UpdateOptions{})
-	return err
-}
-
-func applyControlAgentDaemonSet(ctx context.Context, kubeClient kubernetes.Interface, namespace string, labels map[string]string) error {
-	privileged := true
-
-	dsLabels := map[string]string{
-		"app.kubernetes.io/name":       controlAgentName,
-		"app.kubernetes.io/component":  "agent",
-		"app.kubernetes.io/part-of":    "monok8s",
-		"app.kubernetes.io/managed-by": "ctl",
-	}
-
-	want := &appsv1.DaemonSet{
-		ObjectMeta: metav1.ObjectMeta{
-			Name:      controlAgentName,
-			Namespace: namespace,
-			Labels:    labels,
-		},
-		Spec: appsv1.DaemonSetSpec{
-			Selector: &metav1.LabelSelector{
-				MatchLabels: map[string]string{
-					"app.kubernetes.io/name": controlAgentName,
-				},
-			},
-			Template: corev1.PodTemplateSpec{
-				ObjectMeta: metav1.ObjectMeta{
-					Labels: dsLabels,
-				},
-				Spec: corev1.PodSpec{
-					ServiceAccountName: controlAgentName,
-					HostNetwork:        true,
-					HostPID:            true,
-					DNSPolicy:          corev1.DNSClusterFirstWithHostNet,
-					NodeSelector: map[string]string{
-						monov1alpha1.ControlAgentKey: controlAgentNodeSelectorValue,
-					},
-					Tolerations: []corev1.Toleration{
-						{Operator: corev1.TolerationOpExists},
-					},
-					Containers: []corev1.Container{
-						{
-							Name:            "agent",
-							Image:           controlAgentImage,
-							ImagePullPolicy: corev1.PullNever,
-							Args:            []string{"agent", "--env-file", "$(CLUSTER_ENV_FILE)"},
-							Env: []corev1.EnvVar{
-								{
-									Name: "NODE_NAME",
-									ValueFrom: &corev1.EnvVarSource{
-										FieldRef: &corev1.ObjectFieldSelector{
-											APIVersion: "v1",
-											FieldPath:  "spec.nodeName",
-										},
-									},
-								},
-								{
-									Name:  "CLUSTER_ENV_FILE",
-									Value: "/host/opt/monok8s/config/cluster.env",
-								},
-								{
-									Name:  "FW_ENV_CONFIG_FILE",
-									Value: "/host/etc/fw_env.config",
-								},
-							},
-							SecurityContext: &corev1.SecurityContext{
-								Privileged: &privileged,
-							},
-							VolumeMounts: []corev1.VolumeMount{
-								{
-									Name:      "host-dev",
-									MountPath: "/dev",
-								},
-								{
-									Name:      "host-etc",
-									MountPath: "/host/etc",
-									ReadOnly:  true,
-								},
-								{
-									Name:      "host-config",
-									MountPath: "/host/opt/monok8s/config",
-									ReadOnly:  true,
-								},
-							},
-						},
-					},
-					Volumes: []corev1.Volume{
-						{
-							Name: "host-dev",
-							VolumeSource: corev1.VolumeSource{
-								HostPath: &corev1.HostPathVolumeSource{
-									Path: "/dev",
-									Type: hostPathType(corev1.HostPathDirectory),
-								},
-							},
-						},
-						{
-							Name: "host-etc",
-							VolumeSource: corev1.VolumeSource{
-								HostPath: &corev1.HostPathVolumeSource{
-									Path: "/etc",
-									Type: hostPathType(corev1.HostPathDirectory),
-								},
-							},
-						},
-						{
-							Name: "host-config",
-							VolumeSource: corev1.VolumeSource{
-								HostPath: &corev1.HostPathVolumeSource{
-									Path: "/opt/monok8s/config",
-									Type: hostPathType(corev1.HostPathDirectory),
-								},
-							},
-						},
-					},
-				},
-			},
-		},
-	}
-
-	existing, err := kubeClient.AppsV1().DaemonSets(namespace).Get(ctx, controlAgentName, metav1.GetOptions{})
-	if apierrors.IsNotFound(err) {
-		_, err = kubeClient.AppsV1().DaemonSets(namespace).Create(ctx, want, metav1.CreateOptions{})
-		return err
-	}
-	if err != nil {
-		return err
-	}
-
-	changed := false
-	if !reflect.DeepEqual(existing.Labels, want.Labels) {
-		existing.Labels = want.Labels
-		changed = true
-	}
-	if !reflect.DeepEqual(existing.Spec, want.Spec) {
-		existing.Spec = want.Spec
-		changed = true
-	}
-
-	if !changed {
-		return nil
-	}
-
-	_, err = kubeClient.AppsV1().DaemonSets(namespace).Update(ctx, existing, metav1.UpdateOptions{})
-	return err
-}
-
-func hostPathType(t corev1.HostPathType) *corev1.HostPathType {
-	return &t
-}
-
-func mountPropagationMode(m corev1.MountPropagationMode) *corev1.MountPropagationMode {
-	return &m
-}

clitools/pkg/node/crio.go

@@ -5,9 +5,14 @@ import (
 	"fmt"
 	"os"
 	"strings"
+	"time"
 
-	"k8s.io/klog/v2"
 	system "example.com/monok8s/pkg/system"
+	"k8s.io/klog/v2"
+)
+
+const (
+	storageConfPath = "/etc/containers/storage.conf"
 )
 
 func ConfigureDefaultCNI(ctx context.Context, n *NodeContext) error {
@@ -60,3 +65,56 @@ func ConfigureDefaultCNI(ctx context.Context, n *NodeContext) error {
 func StartCRIO(ctx context.Context, n *NodeContext) error {
 	return system.EnsureServiceRunning(ctx, n.SystemRunner, "crio")
 }
+
+func RestartCRIO(ctx context.Context, nctx *NodeContext) error {
+	_, err := nctx.SystemRunner.RunWithOptions(
+		ctx,
+		"rc-service",
+		[]string{"crio", "restart"},
+		system.RunOptions{
+			Timeout:      60 * time.Second,
+			OnStdoutLine: func(line string) { klog.Infof("[crio] %s", line) },
+			OnStderrLine: func(line string) { klog.Infof("[crio] %s", line) },
+		},
+	)
+	return err
+}
+
+func writeCRIOStorageConfig(ctx context.Context, nctx *NodeContext, altSource string) error {
+
+	additionalStores := []string{
+		"/usr/lib/monok8s/imagestore",
+	}
+
+	if altSource != "" {
+		additionalStores = append(additionalStores, altSource)
+	}
+
+	var b strings.Builder
+	b.WriteString("# Generated File. DO NOT MODIFY.\n")
+	b.WriteString("[storage]\n")
+	b.WriteString("driver = \"overlay\"\n")
+	b.WriteString("runroot = \"/run/containers/storage\"\n")
+	b.WriteString("graphroot = \"/var/lib/containers/storage\"\n\n")
+	b.WriteString("[storage.options]\n")
+	b.WriteString("additionalimagestores = [\n")
+	for _, s := range additionalStores {
+		b.WriteString(fmt.Sprintf("  %q,\n", s))
+	}
+	b.WriteString("]\n")
+
+	content := b.String()
+
+	path := storageConfPath
+	tmp := path + ".tmp"
+	if err := os.WriteFile(tmp, []byte(content), 0o644); err != nil {
+		return fmt.Errorf("write temp storage.conf: %w", err)
+	}
+	if err := os.Rename(tmp, path); err != nil {
+		_ = os.Remove(tmp)
+		return fmt.Errorf("replace storage.conf: %w", err)
+	}
+
+	klog.InfoS("wrote CRI-O storage config", "path", path, "additionalImageStores", additionalStores)
+	return nil
+}

clitools/pkg/node/fs.go

@@ -0,0 +1,66 @@
+package node
+
+import (
+	"context"
+	"fmt"
+	"os"
+	"path/filepath"
+
+	"k8s.io/klog/v2"
+
+	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	"example.com/monok8s/pkg/controller/osimage"
+)
+
+func EngageControlGate(ctx context.Context, nctx *NodeContext) error {
+	gateFile := filepath.Join(monov1alpha1.EnvConfigDir, ".control-gate")
+
+	f, err := os.OpenFile(gateFile, os.O_CREATE|os.O_WRONLY, 0o644)
+	if err != nil {
+		return fmt.Errorf("touch %s: %w", gateFile, err)
+	}
+	defer f.Close()
+
+	return nil
+}
+
+func ReleaseControlGate(ctx context.Context, nctx *NodeContext) error {
+	gateFile := filepath.Join(monov1alpha1.EnvConfigDir, ".control-gate")
+
+	if err := os.Remove(gateFile); err != nil {
+		return fmt.Errorf("release control gate: %w", err)
+	}
+
+	return WriteLastState(ctx, nctx)
+}
+
+// Required for detecting bootslot changes
+func WriteLastState(ctx context.Context, nctx *NodeContext) error {
+	stBootPart := filepath.Join(monov1alpha1.EnvConfigDir, ".bootpart")
+
+	state, err := osimage.ReadBootState()
+	if err != nil {
+		return fmt.Errorf("read boot state: %w", err)
+	}
+
+	bootPart := state["BOOT_PART"]
+	if bootPart == "" {
+		return fmt.Errorf("BOOT_PART missing")
+	}
+
+	klog.Infof("Writing last state: %+v", bootPart)
+
+	tmp := stBootPart + ".tmp"
+	if err := os.WriteFile(tmp, []byte(bootPart+"\n"), 0o644); err != nil {
+		return err
+	}
+	if err := os.Rename(tmp, stBootPart); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func DiagTestDiskWrite(ctx context.Context, nctx *NodeContext) error {
+	return osimage.TestStreamToTarget(ctx, monov1alpha1.AltPartDeviceLink)
+}

clitools/pkg/node/kubeadm.go

@@ -7,13 +7,11 @@ import (
 	"fmt"
 	"net"
 	"os"
+	"sort"
 	"strings"
 	"time"
 
 	"gopkg.in/yaml.v3"
-	"k8s.io/client-go/discovery"
-	"k8s.io/client-go/kubernetes"
-	"k8s.io/client-go/tools/clientcmd"
 	"k8s.io/klog/v2"
 
 	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
@@ -26,6 +24,16 @@ const (
 	tmpKubeadmInitConf    = "/tmp/kubeadm-init.yaml"
 )
 
+func chooseVersionKubeconfig(state *LocalClusterState) string {
+	if state.HasAdminKubeconfig {
+		return adminKubeconfigPath
+	}
+	if state.HasKubeletKubeconfig {
+		return kubeletKubeconfigPath
+	}
+	return ""
+}
+
 func DetectLocalClusterState(ctx context.Context, nctx *NodeContext) error {
 	_ = ctx
 
@@ -258,110 +266,6 @@ func waitForAPIViaKubeconfig(ctx context.Context, kubeconfigPath string, timeout
 	}
 }
 
-func getServerVersion(ctx context.Context, kubeconfigPath string) (string, error) {
-	restCfg, err := clientcmd.BuildConfigFromFlags("", kubeconfigPath)
-	if err != nil {
-		return "", fmt.Errorf("build kubeconfig %s: %w", kubeconfigPath, err)
-	}
-
-	// Keep this short. This is a probe, not a long-running client.
-	restCfg.Timeout = 5 * time.Second
-
-	clientset, err := kubernetes.NewForConfig(restCfg)
-	if err != nil {
-		return "", fmt.Errorf("create clientset: %w", err)
-	}
-
-	disc := clientset.Discovery()
-	return discoverServerVersion(ctx, disc)
-}
-
-func discoverServerVersion(ctx context.Context, disc discovery.DiscoveryInterface) (string, error) {
-	info, err := disc.ServerVersion()
-	if err != nil {
-		return "", err
-	}
-	if info == nil || strings.TrimSpace(info.GitVersion) == "" {
-		return "", errors.New("server version is empty")
-	}
-	return normalizeKubeVersion(info.GitVersion), nil
-}
-
-type kubeVersion struct {
-	Major int
-	Minor int
-	Patch int
-}
-
-func parseKubeVersion(s string) (kubeVersion, error) {
-	s = strings.TrimSpace(s)
-	s = strings.TrimPrefix(s, "v")
-
-	var v kubeVersion
-	n, err := fmt.Sscanf(s, "%d.%d.%d", &v.Major, &v.Minor, &v.Patch)
-	// Accepts "1.29" or "1.29.3"
-	if err != nil || n < 2 {
-		return kubeVersion{}, fmt.Errorf("invalid kubernetes version %q", s)
-	}
-	return v, nil
-}
-
-// Control-plane: keep this strict.
-// Accept same version, or a one-minor step where the node binary is newer than the current cluster.
-// That covers normal control-plane upgrade flow but blocks nonsense.
-func isSupportedControlPlaneSkew(clusterVersion, nodeVersion string) bool {
-	cv, err := parseKubeVersion(clusterVersion)
-	if err != nil {
-		return false
-	}
-	nv, err := parseKubeVersion(nodeVersion)
-	if err != nil {
-		return false
-	}
-
-	if cv.Major != nv.Major {
-		return false
-	}
-	if cv.Minor == nv.Minor {
-		return true
-	}
-	if nv.Minor == cv.Minor+1 {
-		return true
-	}
-	return false
-}
-
-// Worker: kubelet generally must not be newer than the apiserver.
-// Older kubelets are allowed within supported skew range.
-// Your requirement says unsupported worker skew should still proceed, so this
-// only classifies support status and must NOT be used to block this function.
-func isSupportedWorkerSkew(clusterVersion, nodeVersion string) bool {
-	cv, err := parseKubeVersion(clusterVersion)
-	if err != nil {
-		return false
-	}
-	nv, err := parseKubeVersion(nodeVersion)
-	if err != nil {
-		return false
-	}
-
-	if cv.Major != nv.Major {
-		return false
-	}
-
-	// kubelet newer than apiserver => unsupported
-	if nv.Minor > cv.Minor {
-		return false
-	}
-
-	// kubelet up to 3 minors older than apiserver => supported
-	if cv.Minor-nv.Minor <= 3 {
-		return true
-	}
-
-	return false
-}
-
 func ValidateRequiredImagesPresent(ctx context.Context, n *NodeContext) error {
 	if n.Config.Spec.SkipImageCheck {
 		klog.Infof("skipping image check (skipImageCheck=true)")
@@ -409,8 +313,8 @@ func checkImagePresent(ctx context.Context, n *NodeContext, image string) error
 
 	// crictl inspecti exits non-zero when the image is absent.
 	_, err := n.SystemRunner.RunRetry(ctx, system.RetryOptions{
-		Attempts: 3,
-		Delay:    1 * system.DefaultSecond,
+		Attempts: 6,
+		Delay:    2 * system.DefaultSecond,
 	}, "crictl", "inspecti", image)
 	if err != nil {
 		return fmt.Errorf("image %q not present: %w", image, err)
@@ -418,31 +322,6 @@ func checkImagePresent(ctx context.Context, n *NodeContext, image string) error
 	return nil
 }
 
-func chooseVersionKubeconfig(state *LocalClusterState) string {
-	if state.HasAdminKubeconfig {
-		return adminKubeconfigPath
-	}
-	if state.HasKubeletKubeconfig {
-		return kubeletKubeconfigPath
-	}
-	return ""
-}
-
-func versionEq(a, b string) bool {
-	return normalizeKubeVersion(a) == normalizeKubeVersion(b)
-}
-
-func normalizeKubeVersion(v string) string {
-	v = strings.TrimSpace(v)
-	if v == "" {
-		return ""
-	}
-	if !strings.HasPrefix(v, "v") {
-		v = "v" + v
-	}
-	return v
-}
-
 func buildNodeRegistration(spec monov1alpha1.MonoKSConfigSpec) NodeRegistrationOptions {
 	nodeName := strings.TrimSpace(spec.NodeName)
 	criSocket := strings.TrimSpace(spec.ContainerRuntimeEndpoint)
@@ -464,9 +343,57 @@ func buildNodeRegistration(spec monov1alpha1.MonoKSConfigSpec) NodeRegistrationO
 		)
 	}
 
+	labels := effectiveNodeLabels(spec)
+	if nodeLabels := buildNodeLabelsArg(labels); nodeLabels != "" {
+		nr.KubeletExtraArgs = append(nr.KubeletExtraArgs,
+			KubeadmArg{Name: "node-labels", Value: nodeLabels},
+		)
+	}
+
 	return nr
 }
 
+func effectiveNodeLabels(spec monov1alpha1.MonoKSConfigSpec) map[string]string {
+	if len(spec.NodeLabels) == 0 && !spec.EnableNodeControl {
+		return nil
+	}
+
+	labels := make(map[string]string, len(spec.NodeLabels)+1)
+	for k, v := range spec.NodeLabels {
+		labels[k] = v
+	}
+
+	if spec.EnableNodeControl {
+		labels[monov1alpha1.NodeControlKey] = "true"
+	}
+
+	return labels
+}
+
+func buildNodeLabelsArg(labels map[string]string) string {
+	if len(labels) == 0 {
+		return ""
+	}
+
+	keys := make([]string, 0, len(labels))
+	for k := range labels {
+		k = strings.TrimSpace(k)
+		if k == "" {
+			continue
+		}
+		keys = append(keys, k)
+	}
+	sort.Strings(keys)
+
+	parts := make([]string, 0, len(keys))
+	for _, k := range keys {
+		v := strings.TrimSpace(labels[k])
+		parts = append(parts, k+"="+v)
+	}
+
+	return strings.Join(parts, ",")
+}
+
 func maybeAddBootstrapTaint(nr *NodeRegistrationOptions, role string) {
 	if strings.TrimSpace(role) != "worker" {
 		return
@@ -732,16 +659,6 @@ func RunKubeadmJoin(ctx context.Context, nctx *NodeContext) error {
 	return nil
 }
 
-func RunKubeadmUpgradeApply(context.Context, *NodeContext) error {
-	klog.Info("run_kubeadm_upgrade_apply: TODO implement kubeadm upgrade apply")
-	return nil
-}
-
-func RunKubeadmUpgradeNode(context.Context, *NodeContext) error {
-	klog.Info("run_kubeadm_upgrade_node: TODO implement kubeadm upgrade node")
-	return nil
-}
-
 func ReconcileControlPlane(ctx context.Context, nctx *NodeContext) error {
 	if nctx.BootstrapState == nil {
 		return errors.New("BootstrapState is nil, call ClassifyBootstrapAction() first")

clitools/pkg/node/kubeadm_compat.go

@@ -0,0 +1,108 @@
+package node
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"os"
+	"strings"
+
+	"k8s.io/klog/v2"
+
+	"example.com/monok8s/pkg/system"
+)
+
+const kubeadmUpgradeNodeHostnameBugFixedIn = "v1.35.0"
+
+// COMPAT(kubeadm-upgrade-node-hostname)
+// Affects: Kubernetes/kubeadm < v1.35.0
+// Upstream: kubernetes/kubeadm#3244, kubernetes/kubernetes#134319
+// RemoveWhen: minimum supported Kubernetes version >= v1.35.0
+//
+// Affected kubeadm versions can derive the target Node name for
+// `kubeadm upgrade node` from the local OS hostname instead of the existing
+// kubeadm NodeRegistration / kubelet --hostname-override state.
+func needsKubeadmUpgradeNodeHostnameWorkaround(kubeadmVersion string) bool {
+	lt, err := versionLt(kubeadmVersion, kubeadmUpgradeNodeHostnameBugFixedIn)
+	if err != nil {
+		klog.Warningf(
+			"could not parse kubeadm version %q; enabling kubeadm upgrade node hostname workaround: %v",
+			kubeadmVersion,
+			err,
+		)
+		return true
+	}
+	return lt
+}
+
+// runWithTemporaryHostname works around kubernetes/kubeadm#3244, fixed by
+// kubernetes/kubernetes#134319 in Kubernetes v1.35.0.
+//
+// Affected kubeadm versions can derive the target Node name for
+// `kubeadm upgrade node` from the local OS hostname instead of the existing
+// kubeadm NodeRegistration / kubelet --hostname-override state. That breaks
+// valid setups where the machine hostname differs from the Kubernetes Node
+// name: kubeadm may authenticate as one node but try to get/patch another Node,
+// and the Node authorizer correctly rejects it.
+//
+// Keep this workaround scoped to affected kubeadm versions only. Set the
+// temporary hostname to the Kubernetes Node name, run kubeadm, then restore the
+// configured machine hostname immediately afterward.
+func runWithTemporaryHostname(ctx context.Context, nctx *NodeContext, fn func(context.Context) error) error {
+	if nctx == nil {
+		return errors.New("node context is nil")
+	}
+
+	temporaryHostname := strings.TrimSpace(nctx.Config.Spec.NodeName)
+	if temporaryHostname == "" {
+		return errors.New("temporary hostname is required")
+	}
+
+	originalHostname, err := os.Hostname()
+	if err != nil {
+		return fmt.Errorf("get current hostname: %w", err)
+	}
+
+	if originalHostname == temporaryHostname {
+		return fn(ctx)
+	}
+
+	restoreHostname := strings.TrimSpace(nctx.Config.Spec.Network.Hostname)
+	if restoreHostname == "" {
+		restoreHostname = originalHostname
+	}
+
+	klog.Warningf(
+		"temporarily changing hostname for kubeadm upgrade node: current=%q temporary=%q restore=%q",
+		originalHostname,
+		temporaryHostname,
+		restoreHostname,
+	)
+
+	if err := system.SetHostname(temporaryHostname); err != nil {
+		return fmt.Errorf("set temporary hostname to %q: %w", temporaryHostname, err)
+	}
+
+	defer func() {
+		if err := system.SetHostname(restoreHostname); err != nil {
+			klog.Errorf("failed to restore hostname to %q: %v", restoreHostname, err)
+		}
+	}()
+
+	return fn(ctx)
+}
+
+// COMPAT(kubeadm-upgrade-node-hostname)
+// RemoveWhen: minimum supported Kubernetes version >= v1.35.0
+func runKubeadmUpgradeNodeWithCompat(
+	ctx context.Context,
+	nctx *NodeContext,
+	kubeadmVersion string,
+	fn func(context.Context) error,
+) error {
+	if needsKubeadmUpgradeNodeHostnameWorkaround(kubeadmVersion) {
+		return runWithTemporaryHostname(ctx, nctx, fn)
+	}
+
+	return fn(ctx)
+}

clitools/pkg/node/kubeadm_upgrade.go

@@ -0,0 +1,358 @@
+package node
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"strings"
+	"time"
+
+	corev1 "k8s.io/api/core/v1"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/util/wait"
+	"k8s.io/client-go/kubernetes"
+	"k8s.io/klog/v2"
+
+	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	"example.com/monok8s/pkg/kube"
+	"example.com/monok8s/pkg/system"
+)
+
+const (
+	healthCheckNamespace = "kube-system"
+	healthCheckTimeout   = 60 * time.Second
+)
+
+func RunKubeadmUpgradeApply(ctx context.Context, nctx *NodeContext) error {
+	if nctx.BootstrapState == nil {
+		return errors.New("BootstrapState is nil. Please run earlier steps first")
+	}
+
+	if nctx.BootstrapState.Action != BootstrapActionUpgradeControlPlane {
+		klog.V(4).Infof("skipped for %s", nctx.BootstrapState.Action)
+		return nil
+	}
+
+	if strings.TrimSpace(tmpKubeadmInitConf) == "" {
+		return fmt.Errorf("tmp kubeadm config path is empty")
+	}
+
+	pauseImage, err := resolvePauseImage(ctx, nctx, nctx.Config.Spec.KubernetesVersion)
+	if err != nil {
+		return fmt.Errorf("resolve pause image: %w", err)
+	}
+	klog.InfoS("resolved kubeadm pause image", "image", pauseImage)
+
+	clients, err := kube.NewClientsFromKubeconfig(adminKubeconfigPath)
+	if err != nil {
+		return fmt.Errorf("build kube clients from %s: %w", adminKubeconfigPath, err)
+	}
+
+	if err := runUpgradeSelfHealthCheck(ctx, clients.Kubernetes, pauseImage); err != nil {
+		return fmt.Errorf("pre-upgrade self health check failed: %w", err)
+	}
+
+	args := []string{
+		"upgrade", "apply", "-y",
+		nctx.Config.Spec.KubernetesVersion,
+		"--ignore-preflight-errors=CreateJob",
+	}
+
+	_, err = nctx.SystemRunner.RunWithOptions(
+		ctx,
+		"kubeadm",
+		args,
+		system.RunOptions{
+			Timeout: 15 * time.Minute,
+			OnStdoutLine: func(line string) {
+				klog.Infof("[kubeadm] %s", line)
+			},
+			OnStderrLine: func(line string) {
+				klog.Infof("[kubeadm] %s", line)
+			},
+		},
+	)
+	if err != nil {
+		return fmt.Errorf("run kubeadm upgrade apply: %w", err)
+	}
+
+	return nil
+}
+
+func resolvePauseImage(ctx context.Context, nctx *NodeContext, kubeVersion string) (string, error) {
+	result, err := nctx.SystemRunner.Run(
+		ctx,
+		"kubeadm",
+		"config", "images", "list",
+		"--kubernetes-version", kubeVersion,
+	)
+	if err != nil {
+		return "", fmt.Errorf("kubeadm config images list: %w", err)
+	}
+
+	for _, line := range strings.Split(result.Stdout, "\n") {
+		line = strings.TrimSpace(line)
+		if line == "" {
+			continue
+		}
+
+		if strings.Contains(line, "/pause:") || strings.HasPrefix(line, "pause:") {
+			return line, nil
+		}
+	}
+
+	return "", fmt.Errorf("pause image not found in kubeadm image list output")
+}
+
+func runUpgradeSelfHealthCheck(ctx context.Context, kubeClient kubernetes.Interface, pauseImage string) error {
+	name := fmt.Sprintf("preupgrade-health-check-%d", time.Now().UnixMilli())
+
+	pod := &corev1.Pod{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      name,
+			Namespace: healthCheckNamespace,
+			Labels: map[string]string{
+				"app.kubernetes.io/name":       "preupgrade-health-check",
+				"app.kubernetes.io/managed-by": monov1alpha1.NodeControlName,
+			},
+		},
+		Spec: corev1.PodSpec{
+			RestartPolicy: corev1.RestartPolicyNever,
+			Tolerations: []corev1.Toleration{
+				{
+					Operator: corev1.TolerationOpExists,
+				},
+			},
+			Containers: []corev1.Container{
+				{
+					Name:            "check",
+					Image:           pauseImage,
+					ImagePullPolicy: corev1.PullIfNotPresent,
+				},
+			},
+		},
+	}
+
+	klog.InfoS("creating pre-upgrade health-check pod", "namespace", pod.Namespace, "name", pod.Name, "image", pauseImage)
+
+	created, err := kubeClient.CoreV1().Pods(pod.Namespace).Create(ctx, pod, metav1.CreateOptions{})
+	if err != nil {
+		return fmt.Errorf("create health-check pod %s/%s: %w", pod.Namespace, pod.Name, err)
+	}
+
+	defer func() {
+		delCtx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
+		defer cancel()
+
+		propagation := metav1.DeletePropagationBackground
+		err := kubeClient.CoreV1().Pods(created.Namespace).Delete(delCtx, created.Name, metav1.DeleteOptions{
+			PropagationPolicy: &propagation,
+		})
+		if err != nil && !apierrors.IsNotFound(err) {
+			klog.ErrorS(err, "failed to delete health-check pod", "namespace", created.Namespace, "name", created.Name)
+		}
+	}()
+
+	waitCtx, cancel := context.WithTimeout(ctx, healthCheckTimeout)
+	defer cancel()
+
+	err = wait.PollUntilContextCancel(waitCtx, 1*time.Second, true, func(ctx context.Context) (bool, error) {
+		cur, err := kubeClient.CoreV1().Pods(created.Namespace).Get(ctx, created.Name, metav1.GetOptions{})
+		if err != nil {
+			if apierrors.IsNotFound(err) {
+				return false, nil
+			}
+			return false, err
+		}
+
+		switch cur.Status.Phase {
+		case corev1.PodRunning:
+			if isPodReady(cur) {
+				klog.InfoS("pre-upgrade health-check pod is ready", "namespace", cur.Namespace, "name", cur.Name, "node", cur.Spec.NodeName)
+				return true, nil
+			}
+			return false, nil
+
+		case corev1.PodSucceeded:
+			// unlikely for pause, but fine if it somehow happens
+			klog.InfoS("pre-upgrade health-check pod succeeded", "namespace", cur.Namespace, "name", cur.Name, "node", cur.Spec.NodeName)
+			return true, nil
+
+		case corev1.PodFailed:
+			return false, fmt.Errorf("health-check pod failed: reason=%q message=%q", cur.Status.Reason, cur.Status.Message)
+
+		default:
+			return false, nil
+		}
+	})
+	if err != nil {
+		descErr := describeHealthCheckFailure(ctx, kubeClient, created.Namespace, created.Name)
+		if descErr != nil {
+			klog.ErrorS(descErr, "failed to collect health-check diagnostics", "namespace", created.Namespace, "name", created.Name)
+		}
+		return fmt.Errorf("wait for health-check pod readiness: %w", err)
+	}
+
+	return nil
+}
+
+func isPodReady(pod *corev1.Pod) bool {
+	for _, cond := range pod.Status.Conditions {
+		if cond.Type == corev1.PodReady {
+			return cond.Status == corev1.ConditionTrue
+		}
+	}
+	return false
+}
+
+func describeHealthCheckFailure(ctx context.Context, kubeClient kubernetes.Interface, namespace, name string) error {
+	pod, err := kubeClient.CoreV1().Pods(namespace).Get(ctx, name, metav1.GetOptions{})
+	if err != nil {
+		return fmt.Errorf("get failed health-check pod: %w", err)
+	}
+
+	klog.ErrorS(nil, "health-check pod did not become ready",
+		"namespace", pod.Namespace,
+		"name", pod.Name,
+		"phase", pod.Status.Phase,
+		"reason", pod.Status.Reason,
+		"message", pod.Status.Message,
+		"node", pod.Spec.NodeName,
+	)
+
+	for _, cs := range pod.Status.ContainerStatuses {
+		if cs.State.Waiting != nil {
+			klog.ErrorS(nil, "container waiting",
+				"container", cs.Name,
+				"reason", cs.State.Waiting.Reason,
+				"message", cs.State.Waiting.Message,
+			)
+		}
+		if cs.State.Terminated != nil {
+			klog.ErrorS(nil, "container terminated",
+				"container", cs.Name,
+				"reason", cs.State.Terminated.Reason,
+				"message", cs.State.Terminated.Message,
+				"exitCode", cs.State.Terminated.ExitCode,
+			)
+		}
+	}
+
+	events, err := kubeClient.CoreV1().Events(namespace).List(ctx, metav1.ListOptions{
+		FieldSelector: fmt.Sprintf("involvedObject.kind=Pod,involvedObject.name=%s", name),
+	})
+	if err != nil {
+		return fmt.Errorf("list pod events: %w", err)
+	}
+
+	for _, ev := range events.Items {
+		klog.ErrorS(nil, "health-check pod event",
+			"type", ev.Type,
+			"reason", ev.Reason,
+			"message", ev.Message,
+			"count", ev.Count,
+		)
+	}
+
+	return nil
+}
+
+func RunKubeadmUpgradeNode(ctx context.Context, nctx *NodeContext) error {
+	if nctx == nil {
+		return errors.New("node context is nil")
+	}
+	if nctx.Config == nil {
+		return errors.New("node config is nil")
+	}
+	if nctx.LocalClusterState == nil {
+		return errors.New("LocalClusterState is nil. Please run earlier steps first")
+	}
+	if nctx.BootstrapState == nil {
+		return errors.New("BootstrapState is nil. Please run earlier steps first")
+	}
+
+	switch nctx.BootstrapState.Action {
+	case BootstrapActionUpgradeWorker:
+		// continue
+	default:
+		klog.V(4).Infof("RunKubeadmUpgradeNode skipped for action %q", nctx.BootstrapState.Action)
+		return nil
+	}
+
+	wantVersion := normalizeKubeVersion(strings.TrimSpace(nctx.Config.Spec.KubernetesVersion))
+	if wantVersion == "" {
+		return errors.New("spec.kubernetesVersion is required")
+	}
+
+	kubeconfigPath := chooseVersionKubeconfig(nctx.LocalClusterState)
+	if kubeconfigPath == "" {
+		return errors.New("no kubeconfig available for detecting cluster version")
+	}
+
+	clusterVersion := strings.TrimSpace(nctx.BootstrapState.DetectedClusterVersion)
+	if clusterVersion == "" {
+		var err error
+		clusterVersion, err = getServerVersion(ctx, kubeconfigPath)
+		if err != nil {
+			if nctx.BootstrapState.UnsupportedWorkerVersionSkew {
+				klog.Warningf(
+					"cluster version unavailable but worker skew was marked unsupported/permissive, continuing: reason=%s",
+					nctx.BootstrapState.VersionSkewReason,
+				)
+			} else {
+				return fmt.Errorf("get cluster version via %s: %w", kubeconfigPath, err)
+			}
+		}
+	}
+
+	if clusterVersion != "" && !isSupportedWorkerSkew(clusterVersion, wantVersion) {
+		klog.Warningf(
+			"unsupported worker version skew detected, continuing anyway: cluster=%s node=%s",
+			clusterVersion,
+			wantVersion,
+		)
+	}
+
+	klog.Infof(
+		"running kubeadm upgrade node: role=%s clusterVersion=%s nodeVersion=%s kubeconfig=%s",
+		strings.TrimSpace(nctx.Config.Spec.ClusterRole),
+		clusterVersion,
+		wantVersion,
+		kubeconfigPath,
+	)
+
+	args := []string{
+		"upgrade",
+		"node",
+		"--kubeconfig",
+		kubeconfigPath,
+	}
+
+	runKubeadm := func(ctx context.Context) error {
+		_, err := nctx.SystemRunner.RunWithOptions(
+			ctx,
+			"kubeadm",
+			args,
+			system.RunOptions{
+				Timeout: 10 * time.Minute,
+				OnStdoutLine: func(line string) {
+					klog.Infof("[kubeadm] %s", line)
+				},
+				OnStderrLine: func(line string) {
+					klog.Infof("[kubeadm] %s", line)
+				},
+			},
+		)
+		return err
+	}
+
+	// COMPAT(kubeadm-upgrade-node-hostname)
+	// RemoveWhen: minimum supported Kubernetes version >= v1.35.0
+	// Replace this wrapper with direct runKubeadm(ctx).
+	if err := runKubeadmUpgradeNodeWithCompat(ctx, nctx, wantVersion, runKubeadm); err != nil {
+		return fmt.Errorf("run kubeadm upgrade node: %w", err)
+	}
+
+	return nil
+}

clitools/pkg/node/metadata.go

@@ -17,8 +17,8 @@ import (
 func ApplyLocalNodeMetadataIfPossible(ctx context.Context, nctx *NodeContext) error {
 	spec := nctx.Config.Spec
 
-	if len(spec.NodeAnnotations) == 0 && len(spec.NodeLabels) == 0 {
-		klog.V(4).Infof("No annotations or labels was defined")
+	if len(spec.NodeLabels) == 0 {
+		klog.V(4).Infof("No labels was defined")
 		return nil // nothing to do
 	}
 
@@ -53,9 +53,6 @@ func ApplyLocalNodeMetadataIfPossible(ctx context.Context, nctx *NodeContext) er
 	if node.Labels == nil {
 		node.Labels = make(map[string]string)
 	}
-	if node.Annotations == nil {
-		node.Annotations = make(map[string]string)
-	}
 
 	// Apply labels
 	for k, v := range spec.NodeLabels {
@@ -63,13 +60,8 @@ func ApplyLocalNodeMetadataIfPossible(ctx context.Context, nctx *NodeContext) er
 	}
 
 	// Additional Labels
-	if spec.EnableControlAgent {
-		node.Labels[monov1alpah1.ControlAgentKey] = controlAgentNodeSelectorValue
-	}
-
-	// Apply annotations
-	for k, v := range spec.NodeAnnotations {
-		node.Annotations[k] = v
+	if spec.EnableNodeControl {
+		node.Labels[monov1alpah1.NodeControlKey] = "true"
 	}
 
 	_, err = client.CoreV1().Nodes().Update(ctx, node, metav1.UpdateOptions{})
@@ -77,6 +69,6 @@ func ApplyLocalNodeMetadataIfPossible(ctx context.Context, nctx *NodeContext) er
 		return fmt.Errorf("update node metadata: %w", err)
 	}
 
-	klog.Infof("applied labels/annotations to node %q", nodeName)
+	klog.Infof("applied labels to node %q", nodeName)
 	return nil
 }

clitools/pkg/node/mounts.go

@@ -0,0 +1,194 @@
+package node
+
+import (
+	"bufio"
+	"context"
+	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
+	"time"
+
+	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	system "example.com/monok8s/pkg/system"
+	"k8s.io/klog/v2"
+)
+
+const (
+	altRootMount     = "/run/altrootfs"
+	altImageStoreSrc = "/run/altrootfs/usr/lib/monok8s/imagestore"
+)
+
+func MountAltImageStore(ctx context.Context, nctx *NodeContext) error {
+	// Make mountpoints first.
+	if err := os.MkdirAll(altRootMount, 0o755); err != nil {
+		return fmt.Errorf("mkdir %s: %w", altRootMount, err)
+	}
+
+	altDev := monov1alpha1.AltPartDeviceLink
+	if mounted, err := isMounted(altRootMount); err != nil {
+		return fmt.Errorf("check mount %s: %w", altRootMount, err)
+	} else if !mounted {
+		klog.InfoS("mounting alt rootfs", "device", altDev, "target", altRootMount)
+		if _, err := nctx.SystemRunner.RunWithOptions(
+			ctx,
+			"mount",
+			[]string{"-o", "ro", altDev, altRootMount},
+			system.RunOptions{Timeout: 30 * time.Second},
+		); err != nil {
+			klog.Errorf("mount alt rootfs %s on %s: %w", altDev, altRootMount, err)
+		}
+	}
+
+	// If the alt imagestore doesn't exist, don't fail hard unless you want strict behavior.
+	st, err := os.Stat(altImageStoreSrc)
+	if err != nil {
+
+		// Unmount immediately
+		_ = safeUnmount(ctx, nctx, altRootMount)
+
+		if os.IsNotExist(err) {
+			klog.InfoS(
+				"alt imagestore not found; proceeding without it",
+				"path", altImageStoreSrc,
+			)
+			err = writeCRIOStorageConfig(ctx, nctx, "")
+			if err != nil {
+				return err
+			}
+			return nil
+		}
+
+		return fmt.Errorf("stat alt imagestore %s: %w", altImageStoreSrc, err)
+	}
+
+	if !st.IsDir() {
+		return fmt.Errorf("alt imagestore exists but is not a directory: %s", altImageStoreSrc)
+	}
+
+	err = writeCRIOStorageConfig(ctx, nctx, altImageStoreSrc)
+	if err != nil {
+		return err
+	}
+
+	return nil
+}
+
+func UnmountAltImageStore(ctx context.Context, nctx *NodeContext) error {
+
+	mounted, err := isMounted(altRootMount)
+	if err != nil {
+		return err
+	}
+
+	if mounted {
+		if err := writeCRIOStorageConfig(ctx, nctx, ""); err != nil {
+			return err
+		}
+		err := RestartCRIO(ctx, nctx)
+		if err != nil {
+			return err
+		}
+	}
+
+	var errs []string
+
+	if err := safeUnmount(ctx, nctx, altRootMount); err != nil {
+		errs = append(errs, err.Error())
+	}
+
+	if len(errs) > 0 {
+		return fmt.Errorf(strings.Join(errs, "; "))
+	}
+	return nil
+}
+
+func isMounted(path string) (bool, error) {
+	f, err := os.Open("/proc/self/mountinfo")
+	if err != nil {
+		return false, fmt.Errorf("open /proc/self/mountinfo: %w", err)
+	}
+	defer f.Close()
+
+	target := filepath.Clean(path)
+
+	scanner := bufio.NewScanner(f)
+	for scanner.Scan() {
+		line := scanner.Text()
+		fields := strings.Fields(line)
+		if len(fields) < 5 {
+			continue
+		}
+
+		mountPoint, err := unescapeMountInfo(fields[4])
+		if err != nil {
+			return false, fmt.Errorf("decode mountpoint %q: %w", fields[4], err)
+		}
+
+		if filepath.Clean(mountPoint) == target {
+			return true, nil
+		}
+	}
+
+	if err := scanner.Err(); err != nil {
+		return false, fmt.Errorf("scan /proc/self/mountinfo: %w", err)
+	}
+
+	return false, nil
+}
+
+func safeUnmount(ctx context.Context, nctx *NodeContext, path string) error {
+	mounted, err := isMounted(path)
+	if err != nil {
+		return fmt.Errorf("check mountpoint %s: %w", path, err)
+	}
+	if !mounted {
+		return nil
+	}
+
+	klog.InfoS("unmounting", "target", path)
+	_, err = nctx.SystemRunner.RunWithOptions(
+		ctx,
+		"umount",
+		[]string{path},
+		system.RunOptions{Timeout: 30 * time.Second},
+	)
+	if err != nil {
+		return fmt.Errorf("umount %s: %w", path, err)
+	}
+	return nil
+}
+
+func unescapeMountInfo(s string) (string, error) {
+	var b strings.Builder
+	b.Grow(len(s))
+
+	for i := 0; i < len(s); i++ {
+		if s[i] != '\\' {
+			b.WriteByte(s[i])
+			continue
+		}
+
+		if i+3 >= len(s) {
+			return "", fmt.Errorf("truncated escape sequence")
+		}
+
+		esc := s[i+1 : i+4]
+		switch esc {
+		case "040":
+			b.WriteByte(' ')
+		case "011":
+			b.WriteByte('\t')
+		case "012":
+			b.WriteByte('\n')
+		case "134":
+			b.WriteByte('\\')
+		default:
+			return "", fmt.Errorf("unsupported escape sequence \\%s", esc)
+		}
+
+		i += 3
+	}
+
+	return b.String(), nil
+}

clitools/pkg/node/prereqs.go

@@ -8,9 +8,18 @@ import (
 	"strings"
 	"time"
 
+	"k8s.io/client-go/discovery"
+	"k8s.io/client-go/kubernetes"
+	"k8s.io/client-go/tools/clientcmd"
 	"k8s.io/klog/v2"
 )
 
+type kubeVersion struct {
+	Major int
+	Minor int
+	Patch int
+}
+
 func ValidateNodeIPAndAPIServerReachability(ctx context.Context, nct *NodeContext) error {
 	requireLocalIP := func(wantedIP string) error {
 		wantedIP = strings.TrimSpace(wantedIP)
@@ -189,3 +198,136 @@ func CheckForVersionSkew(ctx context.Context, nctx *NodeContext) error {
 
 	return nil
 }
+
+func versionEq(a, b string) bool {
+	return normalizeKubeVersion(a) == normalizeKubeVersion(b)
+}
+
+func versionLt(a, b string) (bool, error) {
+	av, err := parseKubeVersion(a)
+	if err != nil {
+		return false, err
+	}
+
+	bv, err := parseKubeVersion(b)
+	if err != nil {
+		return false, err
+	}
+
+	if av.Major != bv.Major {
+		return av.Major < bv.Major, nil
+	}
+	if av.Minor != bv.Minor {
+		return av.Minor < bv.Minor, nil
+	}
+	return av.Patch < bv.Patch, nil
+}
+
+func normalizeKubeVersion(v string) string {
+	v = strings.TrimSpace(v)
+	if v == "" {
+		return ""
+	}
+	if !strings.HasPrefix(v, "v") {
+		v = "v" + v
+	}
+	return v
+}
+
+func parseKubeVersion(s string) (kubeVersion, error) {
+	s = strings.TrimSpace(s)
+	s = strings.TrimPrefix(s, "v")
+
+	var v kubeVersion
+	n, err := fmt.Sscanf(s, "%d.%d.%d", &v.Major, &v.Minor, &v.Patch)
+	// Accepts "1.29" or "1.29.3"
+	if err != nil || n < 2 {
+		return kubeVersion{}, fmt.Errorf("invalid kubernetes version %q", s)
+	}
+	return v, nil
+}
+
+// Control-plane: keep this strict.
+// Accept same version, or a one-minor step where the node binary is newer than the current cluster.
+// That covers normal control-plane upgrade flow but blocks nonsense.
+func isSupportedControlPlaneSkew(clusterVersion, nodeVersion string) bool {
+	cv, err := parseKubeVersion(clusterVersion)
+	if err != nil {
+		return false
+	}
+	nv, err := parseKubeVersion(nodeVersion)
+	if err != nil {
+		return false
+	}
+
+	if cv.Major != nv.Major {
+		return false
+	}
+	if cv.Minor == nv.Minor {
+		return true
+	}
+	if nv.Minor == cv.Minor+1 {
+		return true
+	}
+	return false
+}
+
+// Worker: kubelet generally must not be newer than the apiserver.
+// Older kubelets are allowed within supported skew range.
+// Your requirement says unsupported worker skew should still proceed, so this
+// only classifies support status and must NOT be used to block this function.
+func isSupportedWorkerSkew(clusterVersion, nodeVersion string) bool {
+	cv, err := parseKubeVersion(clusterVersion)
+	if err != nil {
+		return false
+	}
+	nv, err := parseKubeVersion(nodeVersion)
+	if err != nil {
+		return false
+	}
+
+	if cv.Major != nv.Major {
+		return false
+	}
+
+	// kubelet newer than apiserver => unsupported
+	if nv.Minor > cv.Minor {
+		return false
+	}
+
+	// kubelet up to 3 minors older than apiserver => supported
+	if cv.Minor-nv.Minor <= 3 {
+		return true
+	}
+
+	return false
+}
+
+func getServerVersion(ctx context.Context, kubeconfigPath string) (string, error) {
+	restCfg, err := clientcmd.BuildConfigFromFlags("", kubeconfigPath)
+	if err != nil {
+		return "", fmt.Errorf("build kubeconfig %s: %w", kubeconfigPath, err)
+	}
+
+	// Keep this short. This is a probe, not a long-running client.
+	restCfg.Timeout = 5 * time.Second
+
+	clientset, err := kubernetes.NewForConfig(restCfg)
+	if err != nil {
+		return "", fmt.Errorf("create clientset: %w", err)
+	}
+
+	disc := clientset.Discovery()
+	return discoverServerVersion(ctx, disc)
+}
+
+func discoverServerVersion(ctx context.Context, disc discovery.DiscoveryInterface) (string, error) {
+	info, err := disc.ServerVersion()
+	if err != nil {
+		return "", err
+	}
+	if info == nil || strings.TrimSpace(info.GitVersion) == "" {
+		return "", errors.New("server version is empty")
+	}
+	return normalizeKubeVersion(info.GitVersion), nil
+}

clitools/pkg/node/uboot/ab.go

@@ -2,9 +2,14 @@ package uboot
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"os"
+	"path/filepath"
+	"strings"
 
+	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	"example.com/monok8s/pkg/controller/osimage"
 	"example.com/monok8s/pkg/node"
 )
 
@@ -15,16 +20,97 @@ func ConfigureABBoot(ctx context.Context, nctx *node.NodeContext) error {
 		return fmt.Errorf("get current executable path: %w", err)
 	}
 
+	doWrite := false
 	writer := NewFWEnvWriter(HostFWEnvCfgPath, exePath)
 
-	// TODO: configurable from cluster.env
+	bootPart, err := writer.GetEnv(ctx, "boot_part")
+	if err != nil {
+		if errors.Is(err, ErrEnvNotFound) {
+
+			state, err := osimage.ReadBootState()
+			if err != nil {
+				return fmt.Errorf("read boot state: %w", err)
+			}
+
+			bootPart = state["BOOT_PART"]
+			if bootPart == "" {
+				return fmt.Errorf("BOOT_PART missing")
+			}
+
+			doWrite = true
+		} else {
+			return fmt.Errorf("get boot_part: %w", err)
+		}
+	}
+
+	bootSource, err := writer.GetEnv(ctx, "boot_source")
+	if err != nil {
+		if errors.Is(err, ErrEnvNotFound) {
+			target, err := os.Readlink(monov1alpha1.AltPartDeviceLink)
+			if err != nil {
+				return fmt.Errorf("readlink %q: %w", monov1alpha1.AltPartDeviceLink, err)
+			}
+
+			if !filepath.IsAbs(target) {
+				target = filepath.Join(filepath.Dir(monov1alpha1.AltPartDeviceLink), target)
+			}
+
+			resolved, err := filepath.EvalSymlinks(target)
+			if err != nil {
+				return fmt.Errorf("resolve %q: %w", target, err)
+			}
+
+			base := filepath.Base(resolved)
+
+			if strings.HasPrefix(base, "mmcblk") {
+				bootSource = "emmc"
+			} else {
+				bootSource = "usb"
+			}
+			doWrite = true
+		} else {
+			return fmt.Errorf("get boot_source: %w", err)
+		}
+	}
+
+	if doWrite {
+		return writer.EnsureBootEnv(ctx, BootEnvConfig{
+			BootSource: bootSource,
+			BootPart:   bootPart,
+		})
+	}
+
+	return nil
+}
+
+// This is called by the agent controller/osupgrade/handler.go
+func ConfigureNextBoot(ctx context.Context, fwEnvCfgPath string) error {
+
+	exePath, err := os.Executable()
+	if err != nil {
+		return fmt.Errorf("get current executable path: %w", err)
+	}
+
+	writer := NewFWEnvWriter(fwEnvCfgPath, exePath)
+
+	currBootPart, err := writer.GetEnv(ctx, "boot_part")
+	if err != nil {
+		return fmt.Errorf("get boot_part: %w", err)
+	}
+
+	next := "A"
+	if currBootPart == "A" {
+		next = "B"
+
+	}
+
+	currBootSource, err := writer.GetEnv(ctx, "boot_source")
+	if err != nil {
+		return fmt.Errorf("get boot_source: %w", err)
+	}
+
 	return writer.EnsureBootEnv(ctx, BootEnvConfig{
-		BootSource:      "usb",
-		BootPart:        "A",
-		BootDisk:        0,
-		RootfsAPartNum:  2,
-		RootfsBPartNum:  3,
-		DataPartNum:     4,
-		LinuxRootPrefix: "/dev/sda",
+		BootSource: currBootSource,
+		BootPart:   next,
 	})
 }

clitools/pkg/node/uboot/bootcmd_test.go

@@ -0,0 +1,12 @@
+package uboot
+
+import (
+	"fmt"
+	"testing"
+)
+
+func TestBootCmd(_ *testing.T) {
+
+	cfg := BootEnvConfig{}
+	fmt.Println(cfg.bootCmdOrDefault())
+}

clitools/pkg/node/uboot/bootenvconfig.go

@@ -18,22 +18,6 @@ func (c BootEnvConfig) Validate() error {
 		return fmt.Errorf("invalid boot part %q", c.BootPart)
 	}
 
-	if c.BootDisk < 0 {
-		return fmt.Errorf("invalid boot disk %d", c.BootDisk)
-	}
-	if c.RootfsAPartNum <= 0 {
-		return fmt.Errorf("invalid rootfs A part %d", c.RootfsAPartNum)
-	}
-	if c.RootfsBPartNum <= 0 {
-		return fmt.Errorf("invalid rootfs B part %d", c.RootfsBPartNum)
-	}
-	if c.DataPartNum <= 0 {
-		return fmt.Errorf("invalid data part %d", c.DataPartNum)
-	}
-	if strings.TrimSpace(c.LinuxRootPrefix) == "" {
-		return fmt.Errorf("linux root prefix is required")
-	}
-
 	return nil
 }
 

clitools/pkg/node/uboot/env_writer.go

@@ -2,6 +2,7 @@ package uboot
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"strings"
 	"time"
@@ -11,6 +12,8 @@ import (
 	"example.com/monok8s/pkg/system"
 )
 
+var ErrEnvNotFound = errors.New("fw env not found")
+
 func NewFWEnvWriter(configPath string, ctlPath string) *FWEnvWriter {
 	return &FWEnvWriter{
 		Runner: system.NewRunner(system.RunnerConfig{
@@ -38,8 +41,12 @@ func (w *FWEnvWriter) GetEnv(ctx context.Context, key string) (string, error) {
 	res, err := w.Runner.RunWithOptions(ctx, w.CtlPath, args, system.RunOptions{Quiet: true})
 	if err != nil {
 		if res != nil {
-			return "", fmt.Errorf("fw-printenv %q: %w (stdout=%q stderr=%q)",
-				key, err, strings.TrimSpace(res.Stdout), strings.TrimSpace(res.Stderr))
+			if strings.Contains(res.Stderr, "not defined") {
+				return "", ErrEnvNotFound
+			} else {
+				return "", fmt.Errorf("fw-printenv %q: %w (stdout=%q stderr=%q)",
+					key, err, strings.TrimSpace(res.Stdout), strings.TrimSpace(res.Stderr))
+			}
 		}
 		return "", fmt.Errorf("fw-printenv %q: %w", key, err)
 	}
@@ -126,11 +133,6 @@ func (w *FWEnvWriter) EnsureBootEnv(ctx context.Context, cfg BootEnvConfig) erro
 		{"bootcmd", cfg.bootCmdOrDefault()},
 		{"boot_source", string(cfg.BootSource)},
 		{"boot_part", string(cfg.BootPart)},
-		{"boot_disk", fmt.Sprintf("%d", cfg.BootDisk)},
-		{"rootfs_a_partnum", fmt.Sprintf("%d", cfg.RootfsAPartNum)},
-		{"rootfs_b_partnum", fmt.Sprintf("%d", cfg.RootfsBPartNum)},
-		{"data_partnum", fmt.Sprintf("%d", cfg.DataPartNum)},
-		{"linux_root_prefix", cfg.LinuxRootPrefix},
 	}
 
 	for _, kv := range envs {

clitools/pkg/node/uboot/init.go

@@ -3,6 +3,7 @@ package uboot
 import (
 	"bytes"
 	"context"
+	"errors"
 	"fmt"
 	"k8s.io/klog/v2"
 	"os"
@@ -155,7 +156,7 @@ func ConfigureUBootCommands(ctx context.Context, n *node.NodeContext) error {
 			b.WriteString(trimOutput(a.output, 600))
 			b.WriteString("\n")
 		}
-		return fmt.Errorf(b.String())
+		return errors.New(b.String())
 	}
 
 	klog.Infof("Using: %s", best.config)

clitools/pkg/node/uboot/types.go

@@ -12,14 +12,9 @@ const (
 )
 
 type BootEnvConfig struct {
-	BootSource      string // usb or emmc
-	BootPart        string // A or B
-	BootDisk        int
-	RootfsAPartNum  int
-	RootfsBPartNum  int
-	DataPartNum     int
-	LinuxRootPrefix string // /dev/sda or /dev/mmcblk0p
-	BootCmd         string // optional; defaults to DefaultBootCmd
+	BootSource string // usb or emmc
+	BootPart   string // A or B
+	BootCmd    string // optional; defaults to DefaultBootCmd
 }
 
 type FWEnvWriter struct {
@@ -29,32 +24,52 @@ type FWEnvWriter struct {
 }
 
 const defaultBootCmdTemplate = `
+setenv kernel_addr_r 0xa0000000;
+
+if usb start; then
+    if usb dev 0; then
+        echo "Trying generic USB boot...";
+
+        # Prefer extlinux-style boot
+        if test -e usb 0:1 /boot/extlinux/extlinux.conf; then
+            echo "Found extlinux config on USB";
+            sysboot usb 0:1 any ${kernel_addr_r} /boot/extlinux/extlinux.conf;
+        fi;
+
+        # Fallback: U-Boot script image
+		if test -e usb 0:1 /boot/boot.scr; then
+			echo "Found boot.scr on USB";
+			if load usb 0:1 ${kernel_addr_r} /boot/boot.scr; then
+				source ${kernel_addr_r};
+			fi;
+		fi;
+    fi;
+fi;
+
 if test "${boot_source}" = "usb"; then
     usb start || exit;
     setenv boot_iface usb;
+    usb dev 0 || exit;
 elif test "${boot_source}" = "emmc"; then
-    mmc dev ${boot_disk} || exit;
+    mmc dev 0 || exit;
     setenv boot_iface mmc;
 else
     echo "unsupported boot_source: ${boot_source}";
     exit;
 fi;
 
-setenv kernel_addr_r 0xa0000000;
-
 if test "${boot_part}" = "A"; then
-    setenv rootpart ${rootfs_a_partnum};
+    setenv rootpart 2;
 elif test "${boot_part}" = "B"; then
-    setenv rootpart ${rootfs_b_partnum};
+    setenv rootpart 3;
 else
     echo "unsupported boot_part: ${boot_part}";
     exit;
 fi;
 
-setenv bootdev ${boot_disk}:${rootpart};
-setenv rootdev ${linux_root_prefix}${rootpart};
-setenv datadev ${linux_root_prefix}${data_partnum};
+setenv bootdev 0:${rootpart};
+setenv rootdev ${boot_source}:${rootpart};
 
-setenv bootargs "${bootargs_console} root=${rootdev} data=${datadev} rw rootwait rootfstype=ext4";
+setenv bootargs "${bootargs_console} root=${rootdev} bootpart=${boot_part} rw rootwait rootfstype=ext4 delayacct";
 ext4load ${boot_iface} ${bootdev} ${kernel_addr_r} /boot/kernel.itb && bootm ${kernel_addr_r};
 `

clitools/pkg/render/agent.go

@@ -0,0 +1,284 @@
+package render
+
+import (
+	"fmt"
+	"strings"
+
+	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
+	rbacv1 "k8s.io/api/rbac/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+
+	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	buildinfo "example.com/monok8s/pkg/buildinfo"
+)
+
+type AgentConf struct {
+	Namespace        string
+	Image            string
+	ImagePullSecrets []string
+	Labels           map[string]string
+}
+
+func RenderAgentDaemonSets(conf AgentConf) (string, error) {
+	objs, err := buildAgentDaemonSetObjects(conf)
+	if err != nil {
+		return "", err
+	}
+
+	return renderObjects(objs)
+}
+
+func buildAgentDaemonSetObjects(conf AgentConf) ([]runtime.Object, error) {
+	if strings.TrimSpace(conf.Namespace) == "" {
+		return nil, fmt.Errorf("namespace is required")
+	}
+
+	conf.Labels = map[string]string{
+		"app.kubernetes.io/name":       monov1alpha1.NodeAgentName,
+		"app.kubernetes.io/component":  "agent",
+		"app.kubernetes.io/part-of":    "monok8s",
+		"app.kubernetes.io/managed-by": monov1alpha1.NodeControlName,
+	}
+
+	return []runtime.Object{
+		buildAgentServiceAccount(conf),
+		buildAgentClusterRole(conf),
+		buildAgentClusterRoleBinding(conf),
+		buildAgentDaemonSet(conf),
+	}, nil
+}
+
+func buildAgentNamespace(conf AgentConf) *corev1.Namespace {
+	return &corev1.Namespace{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "v1",
+			Kind:       "Namespace",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:   conf.Namespace,
+			Labels: copyStringMap(conf.Labels),
+		},
+	}
+}
+
+func buildAgentServiceAccount(conf AgentConf) *corev1.ServiceAccount {
+	return &corev1.ServiceAccount{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "v1",
+			Kind:       "ServiceAccount",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      monov1alpha1.NodeAgentName,
+			Namespace: conf.Namespace,
+			Labels:    copyStringMap(conf.Labels),
+		},
+	}
+}
+
+func buildAgentClusterRole(conf AgentConf) *rbacv1.ClusterRole {
+	wantRules := []rbacv1.PolicyRule{
+		{
+			APIGroups: []string{monov1alpha1.Group},
+			Resources: []string{"osupgrades"},
+			Verbs:     []string{"get"},
+		},
+		{
+			APIGroups: []string{monov1alpha1.Group},
+			Resources: []string{"osupgradeprogresses"},
+			Verbs:     []string{"get", "list", "watch", "create", "patch", "update"},
+		},
+		{
+			APIGroups: []string{monov1alpha1.Group},
+			Resources: []string{"osupgradeprogresses/status"},
+			Verbs:     []string{"get", "list", "watch", "create", "patch", "update"},
+		},
+		{
+			APIGroups: []string{""},
+			Resources: []string{"nodes"},
+			Verbs:     []string{"get", "list", "watch"},
+		},
+	}
+
+	return &rbacv1.ClusterRole{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "rbac.authorization.k8s.io/v1",
+			Kind:       "ClusterRole",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:   monov1alpha1.NodeAgentName,
+			Labels: copyStringMap(conf.Labels),
+		},
+		Rules: wantRules,
+	}
+}
+
+func buildAgentClusterRoleBinding(conf AgentConf) *rbacv1.ClusterRoleBinding {
+	return &rbacv1.ClusterRoleBinding{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "rbac.authorization.k8s.io/v1",
+			Kind:       "ClusterRoleBinding",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:   monov1alpha1.NodeAgentName,
+			Labels: copyStringMap(conf.Labels),
+		},
+		RoleRef: rbacv1.RoleRef{
+			APIGroup: rbacv1.GroupName,
+			Kind:     "ClusterRole",
+			Name:     monov1alpha1.NodeAgentName,
+		},
+		Subjects: []rbacv1.Subject{
+			{
+				Kind:      "ServiceAccount",
+				Name:      monov1alpha1.NodeAgentName,
+				Namespace: conf.Namespace,
+			},
+		},
+	}
+}
+
+func buildAgentDaemonSet(conf AgentConf) *appsv1.DaemonSet {
+	privileged := true
+	dsLabels := monov1alpha1.NodeAgentLabels()
+
+	image, pullPolicy := agentImage(conf)
+
+	return &appsv1.DaemonSet{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "apps/v1",
+			Kind:       "DaemonSet",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      monov1alpha1.NodeAgentName,
+			Namespace: conf.Namespace,
+			Labels:    copyStringMap(conf.Labels),
+		},
+		Spec: appsv1.DaemonSetSpec{
+			Selector: &metav1.LabelSelector{
+				MatchLabels: map[string]string{
+					"app.kubernetes.io/name": monov1alpha1.NodeAgentName,
+				},
+			},
+			Template: corev1.PodTemplateSpec{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: dsLabels,
+				},
+				Spec: corev1.PodSpec{
+					ServiceAccountName: monov1alpha1.NodeAgentName,
+					HostNetwork:        true,
+					HostPID:            true,
+					DNSPolicy:          corev1.DNSClusterFirstWithHostNet,
+					ImagePullSecrets:   imagePullSecrets(conf.ImagePullSecrets),
+					NodeSelector: map[string]string{
+						monov1alpha1.NodeControlKey: "true",
+					},
+					Tolerations: []corev1.Toleration{
+						{Operator: corev1.TolerationOpExists},
+					},
+					Containers: []corev1.Container{
+						{
+							Name:            "agent",
+							Image:           image,
+							ImagePullPolicy: pullPolicy,
+							Args:            []string{"agent", "--env-file", "$(CLUSTER_ENV_FILE)"},
+							Env: []corev1.EnvVar{
+								{
+									Name: "NODE_NAME",
+									ValueFrom: &corev1.EnvVarSource{
+										FieldRef: &corev1.ObjectFieldSelector{
+											APIVersion: "v1",
+											FieldPath:  "spec.nodeName",
+										},
+									},
+								},
+								{
+									Name:  "CLUSTER_ENV_FILE",
+									Value: "/host/opt/monok8s/config/cluster.env",
+								},
+								{
+									Name:  "FW_ENV_CONFIG_FILE",
+									Value: "/host/etc/fw_env.config",
+								},
+							},
+							SecurityContext: &corev1.SecurityContext{
+								Privileged: &privileged,
+							},
+							VolumeMounts: []corev1.VolumeMount{
+								{
+									Name:      "host-dev",
+									MountPath: "/dev",
+								},
+								{
+									Name:      "host-etc",
+									MountPath: "/host/etc",
+									ReadOnly:  true,
+								},
+								{
+									Name:      "host-config",
+									MountPath: "/host/opt/monok8s/config",
+									ReadOnly:  true,
+								},
+							},
+						},
+					},
+					Volumes: []corev1.Volume{
+						{
+							Name: "host-dev",
+							VolumeSource: corev1.VolumeSource{
+								HostPath: &corev1.HostPathVolumeSource{
+									Path: "/dev",
+									Type: hostPathType(corev1.HostPathDirectory),
+								},
+							},
+						},
+						{
+							Name: "host-etc",
+							VolumeSource: corev1.VolumeSource{
+								HostPath: &corev1.HostPathVolumeSource{
+									Path: "/etc",
+									Type: hostPathType(corev1.HostPathDirectory),
+								},
+							},
+						},
+						{
+							Name: "host-config",
+							VolumeSource: corev1.VolumeSource{
+								HostPath: &corev1.HostPathVolumeSource{
+									Path: "/opt/monok8s/config",
+									Type: hostPathType(corev1.HostPathDirectory),
+								},
+							},
+						},
+					},
+				},
+			},
+		},
+	}
+}
+
+func agentImage(conf AgentConf) (string, corev1.PullPolicy) {
+	if conf.Image != "" {
+		return conf.Image, corev1.PullIfNotPresent
+	}
+
+	return fmt.Sprintf("localhost/monok8s/node-control:%s", buildinfo.Version), corev1.PullNever
+}
+
+func copyStringMap(in map[string]string) map[string]string {
+	if len(in) == 0 {
+		return nil
+	}
+
+	out := make(map[string]string, len(in))
+	for k, v := range in {
+		out[k] = v
+	}
+
+	return out
+}
+
+func hostPathType(t corev1.HostPathType) *corev1.HostPathType {
+	return &t
+}

clitools/pkg/render/agent_apply.go

@@ -0,0 +1,203 @@
+package render
+
+import (
+	"context"
+	"fmt"
+	"reflect"
+
+	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
+	rbacv1 "k8s.io/api/rbac/v1"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/client-go/kubernetes"
+)
+
+func ApplyAgentDaemonSets(ctx context.Context, kubeClient kubernetes.Interface, conf AgentConf) error {
+	objs, err := buildAgentDaemonSetObjects(conf)
+	if err != nil {
+		return err
+	}
+
+	if err := applyAgentNamespace(ctx, kubeClient, buildAgentNamespace(conf)); err != nil {
+		return fmt.Errorf("apply namespace: %w", err)
+	}
+
+	for _, obj := range objs {
+		if err := applyAgentObject(ctx, kubeClient, obj); err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func applyAgentObject(ctx context.Context, kubeClient kubernetes.Interface, obj runtime.Object) error {
+	switch want := obj.(type) {
+	case *corev1.ServiceAccount:
+		return applyAgentServiceAccount(ctx, kubeClient, want)
+	case *rbacv1.ClusterRole:
+		return applyAgentClusterRole(ctx, kubeClient, want)
+	case *rbacv1.ClusterRoleBinding:
+		return applyAgentClusterRoleBinding(ctx, kubeClient, want)
+	case *appsv1.DaemonSet:
+		return applyAgentDaemonSet(ctx, kubeClient, want)
+	default:
+		return fmt.Errorf("unsupported agent object type %T", obj)
+	}
+}
+
+func applyAgentNamespace(ctx context.Context, kubeClient kubernetes.Interface, want *corev1.Namespace) error {
+	existing, err := kubeClient.CoreV1().Namespaces().Get(ctx, want.Name, metav1.GetOptions{})
+	if apierrors.IsNotFound(err) {
+		_, err = kubeClient.CoreV1().Namespaces().Create(ctx, want, metav1.CreateOptions{})
+		return err
+	}
+	if err != nil {
+		return err
+	}
+
+	labels, changed := mergeStringMapsInto(existing.Labels, want.Labels)
+	if !changed {
+		return nil
+	}
+
+	existing.Labels = labels
+	_, err = kubeClient.CoreV1().Namespaces().Update(ctx, existing, metav1.UpdateOptions{})
+	return err
+}
+
+func applyAgentServiceAccount(ctx context.Context, kubeClient kubernetes.Interface, want *corev1.ServiceAccount) error {
+	existing, err := kubeClient.CoreV1().ServiceAccounts(want.Namespace).Get(ctx, want.Name, metav1.GetOptions{})
+	if apierrors.IsNotFound(err) {
+		_, err = kubeClient.CoreV1().ServiceAccounts(want.Namespace).Create(ctx, want, metav1.CreateOptions{})
+		return err
+	}
+	if err != nil {
+		return err
+	}
+
+	changed := false
+	if !reflect.DeepEqual(existing.Labels, want.Labels) {
+		existing.Labels = want.Labels
+		changed = true
+	}
+
+	if !changed {
+		return nil
+	}
+
+	_, err = kubeClient.CoreV1().ServiceAccounts(want.Namespace).Update(ctx, existing, metav1.UpdateOptions{})
+	return err
+}
+
+func applyAgentClusterRole(ctx context.Context, kubeClient kubernetes.Interface, want *rbacv1.ClusterRole) error {
+	existing, err := kubeClient.RbacV1().ClusterRoles().Get(ctx, want.Name, metav1.GetOptions{})
+	if apierrors.IsNotFound(err) {
+		_, err = kubeClient.RbacV1().ClusterRoles().Create(ctx, want, metav1.CreateOptions{})
+		return err
+	}
+	if err != nil {
+		return err
+	}
+
+	changed := false
+	if !reflect.DeepEqual(existing.Labels, want.Labels) {
+		existing.Labels = want.Labels
+		changed = true
+	}
+	if !reflect.DeepEqual(existing.Rules, want.Rules) {
+		existing.Rules = want.Rules
+		changed = true
+	}
+
+	if !changed {
+		return nil
+	}
+
+	_, err = kubeClient.RbacV1().ClusterRoles().Update(ctx, existing, metav1.UpdateOptions{})
+	return err
+}
+
+func applyAgentClusterRoleBinding(ctx context.Context, kubeClient kubernetes.Interface, want *rbacv1.ClusterRoleBinding) error {
+	existing, err := kubeClient.RbacV1().ClusterRoleBindings().Get(ctx, want.Name, metav1.GetOptions{})
+	if apierrors.IsNotFound(err) {
+		_, err = kubeClient.RbacV1().ClusterRoleBindings().Create(ctx, want, metav1.CreateOptions{})
+		return err
+	}
+	if err != nil {
+		return err
+	}
+
+	// roleRef is immutable. If it differs, fail loudly instead of pretending we can patch it.
+	if !reflect.DeepEqual(existing.RoleRef, want.RoleRef) {
+		return fmt.Errorf("existing ClusterRoleBinding %q has different roleRef and must be recreated", want.Name)
+	}
+
+	changed := false
+	if !reflect.DeepEqual(existing.Labels, want.Labels) {
+		existing.Labels = want.Labels
+		changed = true
+	}
+	if !reflect.DeepEqual(existing.Subjects, want.Subjects) {
+		existing.Subjects = want.Subjects
+		changed = true
+	}
+
+	if !changed {
+		return nil
+	}
+
+	_, err = kubeClient.RbacV1().ClusterRoleBindings().Update(ctx, existing, metav1.UpdateOptions{})
+	return err
+}
+
+func applyAgentDaemonSet(ctx context.Context, kubeClient kubernetes.Interface, want *appsv1.DaemonSet) error {
+	existing, err := kubeClient.AppsV1().DaemonSets(want.Namespace).Get(ctx, want.Name, metav1.GetOptions{})
+	if apierrors.IsNotFound(err) {
+		_, err = kubeClient.AppsV1().DaemonSets(want.Namespace).Create(ctx, want, metav1.CreateOptions{})
+		return err
+	}
+	if err != nil {
+		return err
+	}
+
+	changed := false
+	if !reflect.DeepEqual(existing.Labels, want.Labels) {
+		existing.Labels = want.Labels
+		changed = true
+	}
+	if !reflect.DeepEqual(existing.Spec, want.Spec) {
+		existing.Spec = want.Spec
+		changed = true
+	}
+
+	if !changed {
+		return nil
+	}
+
+	_, err = kubeClient.AppsV1().DaemonSets(want.Namespace).Update(ctx, existing, metav1.UpdateOptions{})
+	return err
+}
+
+func mergeStringMapsInto(dst map[string]string, src map[string]string) (map[string]string, bool) {
+	if len(src) == 0 {
+		return dst, false
+	}
+
+	changed := false
+	if dst == nil {
+		dst = map[string]string{}
+		changed = true
+	}
+
+	for k, v := range src {
+		if dst[k] != v {
+			dst[k] = v
+			changed = true
+		}
+	}
+
+	return dst, changed
+}