Added missing kconfig

.gitignore

@@ -1,3 +1,6 @@
+build.env.work
+cluster.env.work
+.DS_Store
 clitools/bin
 packages/
 out/

README.md

@@ -1,76 +1,195 @@
 # monok8s
-An Alpine-based Kubernetes cluster image for Mono's Gateway Development Kit
 
-https://docs.mono.si/gateway-development-kit/getting-started
+This is an Alpine-based Kubernetes image for Mono's Gateway Development Kit.
 
-## Features
- * A/B deployment
- * Read-only RootFS
- * k8s style OS upgrade (see Upgrading)
+It gives you a ready-to-boot Kubernetes control-plane image so you can get your device running first, then learn and customize from there.
 
-## DISCLAIMER
+Project/device docs: <https://docs.mono.si/gateway-development-kit/getting-started>
 
- * This is not your everyday linux image! It is best suited for users that is already familiar
-   with k8s. For first-timers, you may want to try the default config that gives you a ready-to-use
-   cluster and get yourself started from there
+---
 
- * USE AT YOUR OWN RISKS. I leverage ChatGPT heavily for this.
+## What you get
 
-## Table of Contents
-1. Flashing
-   - [USB](docs/flashing-usb.md)
-   - [Over network (tftp)](docs/flashing-network.md)
-2. [Upgrading](docs/ota.md)
-3. Getting shell access to the host
-   - UART. The thing you did in first time flashing.
-   - [Install an ssh pod](docs/installing-ssh-pod.md) (Recommended)
+The default image boots into a small Kubernetes control-plane environment with:
 
-## Build
+- Alpine Linux
+- Kubernetes initialized through `kubeadm`
+- read-only root filesystem layout
+- A/B rootfs layout for safer OS upgrades
+- a Kubernetes-style OS upgrade path through `OSUpgrade`
 
-Prerequisites
-* make
-* Docker
-* curl (downloading dependency packages, kubelet, crio, etc)
-* go (building clitools, control-agent)
-   * controller-gen (see [clitools readme](clitools/README.md))
-* git (cloning uboot repo because uboot does not provide direct downloads)
+You do **not** need to know Go or understand the internal build system to try the image.
 
+---
+
+## Before you start
+
+You need:
+
+- a Linux build machine or VM
+- Docker
+- `make`
+- basic command-line comfort
+
+If you are building on a fresh Debian machine, you can install the usual build dependencies with:
+
+```bash
+devtools/setup-build-host.sh
 ```
+
+Or install the minimum packages yourself:
+
+```bash
+sudo apt-get update
+sudo apt-get install -y docker.io make qemu-user-static binfmt-support
+```
+
+Make sure your user can run Docker, or use `sudo` where needed.
+
+---
+
+## Fast path: build an image
+
+Download the project tarball, extract it, then run:
+
+```bash
 make release
 ```
 
-The default configuration will boot as a first time control-plane.
+When the build finishes, check the `out/` directory for the generated image artifacts.
 
-For control-plane
-```
-make cluster-config \
-  MKS_HOSTNAME=monok8s-master \
-  MKS_CLUSTER_ROLE=control-plane \
-  MKS_INIT_CONTROL_PLANE=true \
-  MKS_MGMT_ADDRESS=10.0.0.10/24 \
-  MKS_APISERVER_ADVERTISE_ADDRESS=10.0.0.10
-```
+That is the main path most users should try first.
 
-For worker
-```
-make cluster-config \
-  MKS_HOSTNAME=monok8s-worker \
-  MKS_CLUSTER_ROLE=worker \
-  MKS_INIT_CONTROL_PLANE=no \
-  MKS_MGMT_ADDRESS=10.0.0.10/24 \
-  MKS_APISERVER_ADVERTISE_ADDRESS=10.0.0.10 \
-  MKS_API_SERVER_ENDPOINT=10.0.0.1:6443 \
-  MKS_CNI_PLUGIN=none \
-  MKS_BOOTSTRAP_TOKEN=abcd12.ef3456789abcdef0 \
-  MKS_DISCOVERY_TOKEN_CA_CERT_HASH=sha256:9f1c2b3a4d5e6f7890abc1234567890abcdef1234567890abcdef1234567890ab
-```
+---
 
-Check inside [configs/cluster.env.default](configs/cluster.env.default) for configuration details
+## Flash the image
+
+After building, flash the generated image to your device.
+
+Start with one of these guides:
+
+- [Flash over USB](docs/flashing-usb.md)
+- [Flash over network / TFTP](docs/flashing-network.md)
+
+USB flashing is usually the easiest path when you are setting up the device for the first time.
+
+---
+
+## First boot
+
+The default configuration is intended to boot as a first-time Kubernetes control-plane node.
+
+Default-style control-plane configuration looks like this:
 
-### Making sub stages
 ```bash
-make build-base # The image that builds the kernel and everything
-make kernel     # Builds our kernel from NXP
-make initramfs
-make itb        # Builds out/board.itb (contains the kernel and the initramfs)
+make cluster-config \
+	MKS_HOSTNAME=monok8s-master \
+	MKS_CLUSTER_ROLE=control-plane \
+	MKS_INIT_CONTROL_PLANE=true \
+	MKS_MGMT_ADDRESS=10.0.0.10/24 \
+	MKS_APISERVER_ADVERTISE_ADDRESS=10.0.0.10
 ```
+
+If you are just trying the image for the first time, start with the default control-plane setup. Worker-node setup is still incomplete.
+
+For all available configuration values, see:
+
+- [configs/cluster.env.default](configs/cluster.env.default)
+
+For worker node
+```
+make cluster-config \
+	MKS_HOSTNAME=monok8s-worker \
+	MKS_CLUSTER_ROLE=worker \
+	MKS_INIT_CONTROL_PLANE=no \
+	MKS_MGMT_ADDRESS=10.0.0.10/24 \
+	MKS_APISERVER_ADVERTISE_ADDRESS=10.0.0.10 \
+	MKS_API_SERVER_ENDPOINT=10.0.0.1:6443 \
+	MKS_CNI_PLUGIN=none \
+	MKS_BOOTSTRAP_TOKEN=abcd12.ef3456789abcdef0 \
+	MKS_DISCOVERY_TOKEN_CA_CERT_HASH=sha256:9f1c2b3a4d5e6f7890abc1234567890abcdef1234567890abcdef1234567890ab
+```
+
+---
+
+## Getting shell access
+
+For first-time setup, UART is the most direct option because it is already part of the flashing process.
+
+After the device is running, the recommended path is:
+
+- [Install an SSH pod](docs/installing-ssh-pod.md)
+
+---
+
+## Upgrading
+
+monok8s includes a Kubernetes-style OS upgrade flow using the `OSUpgrade` custom resource.
+
+See:
+
+- [OTA upgrade guide](docs/ota.md)
+
+The currently tested upgrade chain is:
+
+- `1.33.3 -> 1.33.10`
+- `1.33.10 -> 1.34.6`
+- `1.34.6 -> 1.35.3`
+
+Tested worker node upgrade chain:
+
+- `1.33.3 -> 1.34.1`
+- `1.33.1 -> 1.35.3`
+
+---
+
+## Current status
+
+This project is usable for experimenting with a single control-plane device image, but it is still a development project.
+
+Working today:
+
+- initramfs boot flow
+- Alpine boot
+- Kubernetes control-plane bootstrap
+- Kubernetes worker-node
+- default bridge CNI
+- Cilium
+
+Still in progress:
+
+- VPP/DPAA networking experiments
+
+---
+
+## Common build issue
+
+### `chroot: failed to run command '/bin/sh': Exec format error`
+
+This usually means the build host cannot run ARM64 binaries.
+
+On Debian, install ARM64 emulation support:
+
+```bash
+sudo apt-get install -y qemu-user-static binfmt-support
+```
+
+Then run the build again:
+
+```bash
+make release
+```
+
+---
+
+## Notes
+
+This is not a general-purpose Linux distribution. It is a device image for experimenting with Kubernetes on Mono's Gateway Development Kit.
+
+The safest path is:
+
+1. build the default image,
+2. flash it,
+3. boot the control-plane,
+4. confirm Kubernetes is running,
+5. customize only after the base image works.

alpine/build-rootfs.sh

@@ -2,9 +2,11 @@
 
 set -euo pipefail
 
+source /utils.sh
+
 /preload-k8s-images.sh || exit 1
 
-export CTL_BIN_LAYER=$( skopeo inspect docker-daemon:localhost/monok8s/control-agent:dev | jq -r '.Layers[0] | sub("^sha256:"; "")' )
+export CTL_BIN_LAYER=$( skopeo inspect docker-daemon:localhost/monok8s/node-control:$TAG | jq -r '.Layers[0] | sub("^sha256:"; "")' )
 
 mkdir -p \
   "$ROOTFS/dev" \
@@ -17,6 +19,7 @@ mkdir -p \
   "$ROOTFS/build" \
   "$ROOTFS/var/cache/apk" \
   "$ROOTFS/usr/lib/monok8s/crds" \
+  "$ROOTFS/usr/lib/monok8s/migrations.d/k8s" \
   "$ROOTFS/opt/monok8s/config"
 
 mount --bind /var/cache/apk "$ROOTFS/var/cache/apk"
@@ -30,6 +33,15 @@ cp /etc/resolv.conf "$ROOTFS/etc/resolv.conf"
 cp /build/crio.tar.gz "$ROOTFS/build/"
 cp /build/crds/*.yaml "$ROOTFS/usr/lib/monok8s/crds"
 
+KUBE_MINOR=$(printf '%s\n' "$KUBE_VERSION" | sed -E 's/^v?([0-9]+\.[0-9]+).*/\1/')
+MIG_SRC="/build/migrations/k8s/$KUBE_MINOR"
+MIG_DST="$ROOTFS/usr/lib/monok8s/migrations.d/k8s/$KUBE_MINOR"
+
+if [ -d "$MIG_SRC" ]; then
+    mkdir -p "$MIG_DST"
+    cp -a "$MIG_SRC"/. "$MIG_DST"/
+fi
+
 chroot "$ROOTFS" /bin/sh -c "ln -s /var/cache/apk /etc/apk/cache"
 # chroot "$ROOTFS" /bin/sh -c "apk update"
 chroot "$ROOTFS" /bin/sh -c "apk add bash curl"
@@ -64,7 +76,7 @@ DISK_SIZE="${DISK_SIZE:-8G}"
 
 ROOTFS_IMG="${ROOTFS_IMG:-rootfs.ext4}"
 ROOTFS_IMG_ZST="${ROOTFS_IMG_ZST:-rootfs.ext4.zst}"
-ROOTFS_PART_SIZE_MIB="${ROOTFS_PART_SIZE_MIB:-2048}"
+ROOTFS_PART_SIZE_MIB="${ROOTFS_PART_SIZE_MIB:-2560}"
 
 FAKE_DEV="/tmp/dev"
 MNT_ROOTFS_IMG="/mnt/rootfs-img"
@@ -107,6 +119,8 @@ mkdir -p "$FAKE_DEV" "$MNT_ROOTFS_IMG" "$MNT_DATA"
 
 echo "##################################################### Packaging RootFS $(du -sh "$ROOTFS" | awk '{print $1}')"
 
+ensure_loop_ready
+
 ###############################################################################
 # 1. Build reusable rootfs ext4 image once
 ###############################################################################
@@ -183,8 +197,8 @@ truncate -s "$DISK_SIZE" "$IMG"
 
 sgdisk -o "$IMG" \
   -n 1:2048:+64M  -t 1:0700 -c 1:config \
-  -n 2:0:+2G      -t 2:8300 -c 2:rootfsA \
-  -n 3:0:+2G      -t 3:8300 -c 3:rootfsB \
+  -n 2:0:+2560M   -t 2:8300 -c 2:rootfsA \
+  -n 3:0:+2560M   -t 3:8300 -c 3:rootfsB \
   -n 4:0:0        -t 4:8300 -c 4:data
 
 losetup -D

alpine/migrations/k8s/1.35/10-drop-removed-kubelet-pause-flag.sh

@@ -0,0 +1,35 @@
+#!/bin/sh
+set -eu
+
+# Kubernetes removed the kubelet flag:
+#   --pod-infra-container-image
+#
+# Timeline:
+# - Deprecated before v1.27
+# - Removed in newer kubelet versions (>=1.27+)
+# - kubeadm may still write it into:
+#     /var/lib/kubelet/kubeadm-flags.env
+#
+# This causes kubelet to fail with:
+#   "unknown flag: --pod-infra-container-image"
+#
+# References:
+# - https://github.com/kubernetes/kubeadm/issues/3281
+# - https://github.com/kubernetes/kubernetes/pull/122739
+# - https://kubernetes.io/blog/2022/04/07/upcoming-changes-in-kubernetes-1-24/
+#
+# Root cause:
+# - Sandbox (pause) image is now managed by CRI (containerd/CRI-O),
+#   not kubelet flags.
+#
+# Fix:
+# - Strip the flag from kubeadm-flags.env during upgrade
+
+FILE=/var/lib/kubelet/kubeadm-flags.env
+
+[ -f "$FILE" ] || exit 0
+grep -q -- '--pod-infra-container-image=' "$FILE" || exit 0
+
+sed -i 's/ --pod-infra-container-image=[^"]*//g' "$FILE"
+
+echo "Removed deprecated kubelet flag --pod-infra-container-image from $FILE"

alpine/preload-k8s-images.sh

@@ -26,7 +26,7 @@ FUSE_OVERLAYFS="${FUSE_OVERLAYFS:-/usr/bin/fuse-overlayfs}"
 #   )
 EXTRA_IMAGES=(
 	"${EXTRA_IMAGES[@]:-}"
-	"docker-daemon:localhost/monok8s/control-agent:$TAG"
+	"docker-daemon:localhost/monok8s/node-control:$TAG"
 )
 
 # Keep archive cache version/arch scoped so downloads do not get mixed.

alpine/rootfs-extra/etc/local.d/monok8s.start

@@ -1,4 +1,41 @@
 #!/bin/sh
+set -eu
+
+exec >>/var/log/monok8s/boot.log 2>&1
 
 export PATH="/usr/local/bin:/usr/local/sbin:$PATH"
-/usr/local/bin/ctl init --env-file /opt/monok8s/config/cluster.env >>/var/log/monok8s/bootstrap.log 2>&1 &
+
+MIGRATIONS_LIB=/usr/lib/monok8s/lib/migrations.sh
+CONFIG_DIR=/opt/monok8s/config
+BOOT_STATE=/run/monok8s/boot-state.env
+BOOTPART_FILE="$CONFIG_DIR/.bootpart"
+MIGRATION_STATE_DIR="$CONFIG_DIR/migration-state"
+
+mkdir -p /dev/hugepages
+mountpoint -q /dev/hugepages || mount -t hugetlbfs none /dev/hugepages
+echo 256 > /proc/sys/vm/nr_hugepages
+
+CUR=$(grep '^BOOT_PART=' "$BOOT_STATE" | cut -d= -f2-)
+LAST=$(cat "$BOOTPART_FILE" 2>/dev/null || true)
+
+slot_changed=0
+if [ "$CUR" != "$LAST" ]; then
+	slot_changed=1
+	echo "Slot changed ($LAST -> $CUR)"
+fi
+
+# shellcheck source=/dev/null
+. "$MIGRATIONS_LIB"
+
+if [ "$slot_changed" -eq 1 ]; then
+	monok8s_cleanup_runtime_state
+fi
+
+K8S_MINOR="$(monok8s_detect_k8s_minor || true)"
+if [ -n "$K8S_MINOR" ]; then
+	monok8s_run_migration_dir \
+	"/usr/lib/monok8s/migrations.d/k8s/$K8S_MINOR" \
+	"$MIGRATION_STATE_DIR/k8s/$K8S_MINOR"
+fi
+
+/usr/lib/monok8s/lib/supervised-init.sh &

alpine/rootfs-extra/usr/lib/monok8s/lib/migrations.sh

@@ -0,0 +1,58 @@
+#!/bin/sh
+
+monok8s_detect_k8s_minor() {
+	local env_file=${1:-}
+	local version major_minor
+
+	version=$(/usr/local/bin/ctl version -k 2>/dev/null || true)
+
+	[ -n "$version" ] || return 1
+
+	version=${version#v}
+	major_minor=$(printf '%s\n' "$version" | cut -d. -f1,2)
+
+	[ -n "$major_minor" ] || return 1
+	printf '%s\n' "$major_minor"
+}
+
+monok8s_cleanup_runtime_state() {
+	echo "Cleaning runtime state"
+
+	rm -rf \
+		/var/lib/containers \
+		/var/lib/cni \
+		/var/lib/kubelet/pods \
+		/var/lib/kubelet/plugins \
+		/var/lib/kubelet/plugins_registry \
+		/var/lib/kubelet/device-plugins \
+		/run/containers \
+		/run/netns
+
+	mkdir -p \
+		/var/lib/containers \
+		/var/lib/kubelet \
+		/var/lib/cni
+}
+
+monok8s_run_migration_dir() {
+	dir=$1
+	state_dir=$2
+
+	[ -d "$dir" ] || return 0
+	mkdir -p "$state_dir"
+
+	for script in "$dir"/*.sh; do
+		[ -e "$script" ] || continue
+
+		name=$(basename "$script")
+		stamp="$state_dir/$name.done"
+
+		if [ -e "$stamp" ]; then
+			continue
+		fi
+
+		echo "Running migration: $script"
+		sh "$script"
+		: > "$stamp"
+	done
+}

alpine/rootfs-extra/usr/lib/monok8s/lib/supervised-init.sh

@@ -0,0 +1,57 @@
+#!/bin/sh
+set -eu
+
+CONFIG_DIR=/opt/monok8s/config
+LOG=/var/log/monok8s/bootstrap.log
+STATE_DIR=/run/monok8s
+FAIL_COUNT_FILE="$STATE_DIR/bootstrap-fail-count"
+LOCK_DIR="$STATE_DIR/supervised-init.lock"
+
+# For debugging
+HOLD_FILE="$CONFIG_DIR/bootstrap.hold"
+
+mkdir -p "$STATE_DIR" /var/log/monok8s
+
+if ! mkdir "$LOCK_DIR" 2>/dev/null; then
+	echo "[$(date -Is)] supervised-init already running" >> "$LOG"
+	exit 0
+fi
+
+trap 'rmdir "$LOCK_DIR"' EXIT INT TERM
+
+fail_count=0
+if [ -f "$FAIL_COUNT_FILE" ]; then
+	fail_count="$(cat "$FAIL_COUNT_FILE" 2>/dev/null || echo 0)"
+	case "$fail_count" in
+		''|*[!0-9]*) fail_count=0 ;;
+	esac
+fi
+
+while true; do
+	if [ -f "$HOLD_FILE" ]; then
+		echo "[$(date -Is)] bootstrap held by $HOLD_FILE" >> "$LOG"
+		sleep 300
+		continue
+	fi
+
+	echo "[$(date -Is)] starting ctl init" >> "$LOG"
+
+	if /usr/local/bin/ctl init --env-file "$CONFIG_DIR/cluster.env" >> "$LOG" 2>&1; then
+		echo "[$(date -Is)] ctl init succeeded" >> "$LOG"
+		rm -f "$FAIL_COUNT_FILE"
+		exit 0
+	fi
+
+	fail_count=$((fail_count + 1))
+	echo "$fail_count" > "$FAIL_COUNT_FILE"
+
+	echo "[$(date -Is)] ctl init failed, count=$fail_count" >> "$LOG"
+
+	case "$fail_count" in
+		1) sleep 10 ;;
+		2) sleep 30 ;;
+		3) sleep 60 ;;
+		4) sleep 120 ;;
+		*) sleep 300 ;;
+	esac
+done

alpine/utils.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+
+ensure_loop_ready() {
+    # The loop kernel module is host-side. This only works if the container
+    # has permission and modprobe exists; otherwise the host must load it.
+    if ! grep -qw loop /proc/modules 2>/dev/null; then
+        modprobe loop 2>/dev/null || true
+    fi
+
+    # /dev/loop-control: char device 10:237
+    if [ ! -e /dev/loop-control ]; then
+        echo "Creating missing /dev/loop-control" >&2
+        mknod /dev/loop-control c 10 237 || {
+            echo "ERROR: cannot create /dev/loop-control" >&2
+            echo "Run container with --privileged, or pass --device=/dev/loop-control and loop devices." >&2
+            exit 1
+        }
+        chmod 600 /dev/loop-control || true
+    fi
+
+    if [ ! -c /dev/loop-control ]; then
+        echo "ERROR: /dev/loop-control exists but is not a character device" >&2
+        ls -l /dev/loop-control >&2 || true
+        exit 1
+    fi
+
+    # Create a reasonable pool of loop block devices.
+    # loopN block devices are major 7, minor N.
+    for i in $(seq 0 31); do
+        if [ ! -e "/dev/loop$i" ]; then
+            echo "Creating missing /dev/loop$i" >&2
+            mknod "/dev/loop$i" b 7 "$i" || {
+                echo "ERROR: cannot create /dev/loop$i" >&2
+                echo "Run container with --privileged, or pre-create/pass loop devices." >&2
+                exit 1
+            }
+            chmod 660 "/dev/loop$i" || true
+        fi
+
+        if [ ! -b "/dev/loop$i" ]; then
+            echo "ERROR: /dev/loop$i exists but is not a block device" >&2
+            ls -l "/dev/loop$i" >&2 || true
+            exit 1
+        fi
+    done
+
+    # Smoke test: ask losetup for a free loop device.
+    if ! losetup -f >/dev/null 2>&1; then
+        echo "ERROR: losetup cannot find/use a loop device" >&2
+        echo "Debug info:" >&2
+        ls -l /dev/loop-control /dev/loop* >&2 || true
+        grep -w loop /proc/modules >&2 || true
+        echo >&2
+        echo "Docker likely needs --privileged, or at minimum CAP_SYS_ADMIN plus loop devices." >&2
+        exit 1
+    fi
+}

build.env

@@ -3,16 +3,33 @@ DOCKER_IMAGE_ROOT=monok8s
 # Image tag
 TAG=dev
 
-# The Linux kernel, from NXP
-NXP_VERSION=lf-6.18.2-1.0.0
-CRIO_VERSION=cri-o.arm64.v1.33.3
-KUBE_VERSION=v1.33.3
+# NXP's Linux Factory
+LINUX_FACTORY=6.12.49-2.2.0
+NXP_VERSION=lf-$(LINUX_FACTORY)
+FMLIB_VERSION=lf-$(LINUX_FACTORY)
+FMC_VERSION=lf-$(LINUX_FACTORY)
+DPDK_VERSION=lf-$(LINUX_FACTORY)
+VPP_VERSION=lf-$(LINUX_FACTORY)
+VPP_UPSTREAM_VERSION=23.10
+
+# ASK's deps
+MONO_ASK_VERSION=mt-$(LINUX_FACTORY)
+LIBNFNETLINK_VERSION=1.0.2
+LIBMNL_VERSION=1.0.5
+LIBNFCT_VERSION=1.1.0
+LIBCLI_VERSION=1.10.7
+# Check the package version for Debian trixies (what ASK uses)
+LIBXML2_VERSION=2.11.7
+TCLAP_VERSION=1.2.5
+LIBPCAP_VERSION=1.10.4
+
+CRIO_VERSION=cri-o.arm64.v1.35.2
+KUBE_VERSION=v1.35.3
 
 # Mono's tutorial said fsl-ls1046a-rdb.dtb but our shipped board is not that one
 # We need fsl-ls1046a-rdb-sdk.dtb here
 DEVICE_TREE_TARGET=mono-gateway-dk-sdk
 
-
 # Arch, should always be arm64 for our board. This is here in case branching off to other devices
 ARCH=arm64
 
@@ -32,3 +49,11 @@ ALPINE_HOSTNAME=monok8s-hostname
 
 # Upper case [A-Z]_ only, used for naming env vars
 BUILD_TAG=MONOK8S
+
+# Optional apt cache
+#   example: apt-cacher-ng.eco-system.svc.cluster.local:3142
+APT_PROXY=
+
+# remote image repository prefix to push to
+# e.g. ghcr.io/monok8s
+IMAGE_REPOSITORY=

clitools/devtools/run.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+SCRIPT_DIR="$(dirname "${BASH_SOURCE[0]}")"
+PROJ_ROOT="$( realpath "$SCRIPT_DIR"/../ )"
+OUT_DIR="$PROJ_ROOT"/out
+
+if [ "$1" == "controller" ]; then
+	if [ -f "$OUT_DIR/tls.key" ] && [ -f "$OUT_DIR/tls.crt" ]; then
+		echo "Use existing certs"
+	else
+		echo "Generating self signed certs"
+		openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
+		  -keyout "$OUT_DIR"/tls.key -out "$OUT_DIR"/tls.crt \
+		  -subj "/CN=127.0.0.1" \
+		  -addext "subjectAltName=IP:127.0.0.1,DNS:localhost"
+	fi
+	go run "$PROJ_ROOT"/cmd/ctl $@ --tls-cert-file "$OUT_DIR"/tls.crt --tls-private-key-file "$OUT_DIR"/tls.key
+else
+	go run "$PROJ_ROOT"/cmd/ctl $@
+fi

clitools/docker/crdgen.Dockerfile

@@ -0,0 +1,14 @@
+ARG BASE_IMAGE=localhost/monok8s/ctl-build-base:dev
+FROM ${BASE_IMAGE} AS build
+
+WORKDIR /src
+
+RUN GOBIN=/usr/local/bin go install sigs.k8s.io/controller-tools/cmd/controller-gen@v0.20.1
+
+COPY . .
+
+RUN mkdir -p /out && \
+    controller-gen crd paths=./pkg/apis/... output:crd:dir=/out
+
+FROM scratch
+COPY --from=build /out/ /

clitools/docker/ctl-agent.Dockerfile

@@ -1,16 +1,41 @@
+ARG BASE_IMAGE=localhost/monok8s/ctl-build-base:dev
+
+FROM --platform=$BUILDPLATFORM ${BASE_IMAGE} AS build
+
+ARG VERSION=dev
+ARG TARGETOS
+ARG TARGETARCH
+
+WORKDIR /src
+
+COPY . .
+
+RUN test -f pkg/buildinfo/buildinfo_gen.go
+
+RUN mkdir -p /out && \
+    GOOS=${TARGETOS} GOARCH=${TARGETARCH} CGO_ENABLED=0 \
+    go build -trimpath -ldflags="-s -w" \
+      -o /out/ctl ./cmd/ctl
+
+
 FROM alpine:latest AS cacerts
+
+
 FROM scratch
 
 ARG VERSION
+ARG TARGETOS
+ARG TARGETARCH
 ENV VERSION=${VERSION}
 
 WORKDIR /
 
-COPY bin/ctl-linux-aarch64-${VERSION} ./ctl
-COPY out/fw_printenv ./
-COPY out/fw_setenv ./
+COPY --from=build /out/ctl /ctl
 COPY --from=cacerts /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
 
+COPY out/uboot-tools/${TARGETOS}_${TARGETARCH}/fw_printenv /fw_printenv
+COPY out/uboot-tools/${TARGETOS}_${TARGETARCH}/fw_setenv /fw_setenv
+
 ENV PATH=/
 
 ENTRYPOINT ["/ctl"]

clitools/docker/ctl-build-base.Dockerfile

@@ -0,0 +1,8 @@
+FROM golang:1.26-alpine
+
+WORKDIR /src
+
+RUN apk add --no-cache git build-base
+
+COPY go.mod go.sum ./
+RUN go mod download

clitools/docker/download-packages.Dockerfile

@@ -0,0 +1,26 @@
+FROM alpine:3.23.0 AS base
+
+# We seperate the packages so this line can be cached upstream
+RUN apk add --no-cache curl ca-certificates
+RUN apk add --no-cache git
+
+# ---- uboot ----
+FROM base AS uboot
+ARG UBOOT_VERSION
+
+WORKDIR /work
+
+RUN apk add --no-cache git tar gzip
+
+RUN git clone \
+      --depth 1 \
+      --branch "${UBOOT_VERSION}" \
+      --filter=blob:none \
+      https://github.com/u-boot/u-boot.git src
+
+RUN mkdir -p /out && \
+    tar -C /work/src -zcf "/out/uboot-${UBOOT_VERSION}.tar.gz" .
+
+# ---- final exported artifact set ----
+FROM scratch
+COPY --from=uboot /out/ /

clitools/docker/dpdk.Dockerfile

@@ -0,0 +1,51 @@
+ARG TAG=dev
+ARG DOCKER_IMAGE_ROOT=monok8s
+
+FROM alpine:3.22 AS build
+
+RUN apk add --no-cache \
+  bash \
+  build-base \
+  linux-headers \
+  meson \
+  ninja \
+  pkgconf \
+  python3 \
+  py3-elftools \
+  coreutils \
+  file \
+  git \
+  bsd-compat-headers
+
+RUN mkdir /src
+
+WORKDIR /src
+
+ARG DPDK_TAR
+ARG DPDK_VERSION
+
+COPY ${DPDK_TAR} /tmp/
+
+RUN set -eux; \
+    mkdir -p /src/dpdk; \
+    tar -xf "/tmp/$(basename "${DPDK_TAR}")" -C /src/dpdk --strip-components=1
+
+RUN set -eux; \
+    meson setup /src/dpdk/build /src/dpdk \
+      --buildtype=release \
+      -Dplatform=dpaa \
+      -Dtests=false \
+      -Ddisable_drivers=crypto/*,compress/*,baseband/*,dma/*,event/*,regex/*,ml/*,gpu/*,raw/*,net/pcap,net/tap,net/vhost,net/virtio,net/ixgbe,net/i40e,net/txgbe,net/ring,net/af_packet; \
+    meson configure /src/dpdk/build | tee /tmp/meson-config.txt; \
+    grep -Ei 'dpaa|platform|disable_drivers' /tmp/meson-config.txt || true; \
+    ninja -C /src/dpdk/build; \
+    DESTDIR=/out ninja -C /src/dpdk/build install
+
+RUN set -eux; \
+    mkdir -p /artifact/bin; \
+    test -x /src/dpdk/build/app/dpdk-testpmd; \
+    cp -r /src/dpdk/build/app/dpdk-testpmd /artifact/bin/
+
+FROM scratch AS export
+COPY --from=build /out/ /
+COPY --from=build /artifact/ /

clitools/docker/uboot-tools.Dockerfile

@@ -1,4 +1,4 @@
-FROM alpine:3.22 AS build
+FROM alpine:3.23 AS build
 
 RUN apk add --no-cache \
     build-base \

clitools/go.mod

@@ -1,33 +1,39 @@
 module example.com/monok8s
 
-go 1.24.0
+go 1.26.0
 
 require (
+	github.com/emicklei/go-restful/v3 v3.12.2
 	github.com/klauspost/compress v1.18.5
-	github.com/spf13/cobra v1.9.1
-	golang.org/x/sys v0.31.0
+	github.com/spf13/cobra v1.10.2
+	golang.org/x/sys v0.39.0
 	gopkg.in/yaml.v3 v3.0.1
-	k8s.io/api v0.34.0
-	k8s.io/apiextensions-apiserver v0.34.0
-	k8s.io/apimachinery v0.34.0
+	k8s.io/api v0.35.0
+	k8s.io/apiextensions-apiserver v0.35.0
+	k8s.io/apimachinery v0.35.0
+	k8s.io/apiserver v0.35.0
 	k8s.io/cli-runtime v0.34.0
-	k8s.io/client-go v0.34.0
+	k8s.io/client-go v0.35.0
+	k8s.io/code-generator v0.35.0
 	k8s.io/klog/v2 v2.130.1
+	sigs.k8s.io/controller-tools v0.20.1
 	sigs.k8s.io/yaml v1.6.0
 )
 
 require (
 	github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect
+	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/blang/semver/v4 v4.0.0 // indirect
+	github.com/cespare/xxhash/v2 v2.3.0 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
-	github.com/emicklei/go-restful/v3 v3.12.2 // indirect
+	github.com/fatih/color v1.18.0 // indirect
 	github.com/fxamacker/cbor/v2 v2.9.0 // indirect
 	github.com/go-errors/errors v1.4.2 // indirect
-	github.com/go-logr/logr v1.4.2 // indirect
+	github.com/go-logr/logr v1.4.3 // indirect
 	github.com/go-openapi/jsonpointer v0.21.0 // indirect
 	github.com/go-openapi/jsonreference v0.20.2 // indirect
 	github.com/go-openapi/swag v0.23.0 // indirect
-	github.com/gogo/protobuf v1.3.2 // indirect
+	github.com/gobuffalo/flect v1.0.3 // indirect
 	github.com/google/btree v1.1.3 // indirect
 	github.com/google/gnostic-models v0.7.0 // indirect
 	github.com/google/uuid v1.6.0 // indirect
@@ -37,30 +43,42 @@ require (
 	github.com/json-iterator/go v1.1.12 // indirect
 	github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de // indirect
 	github.com/mailru/easyjson v0.7.7 // indirect
+	github.com/mattn/go-colorable v0.1.13 // indirect
+	github.com/mattn/go-isatty v0.0.20 // indirect
 	github.com/moby/term v0.5.0 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect
 	github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
 	github.com/peterbourgon/diskv v2.0.1+incompatible // indirect
-	github.com/pkg/errors v0.9.1 // indirect
-	github.com/spf13/pflag v1.0.6 // indirect
+	github.com/prometheus/client_golang v1.23.2 // indirect
+	github.com/prometheus/client_model v0.6.2 // indirect
+	github.com/prometheus/common v0.66.1 // indirect
+	github.com/prometheus/procfs v0.16.1 // indirect
+	github.com/spf13/pflag v1.0.10 // indirect
 	github.com/x448/float16 v0.8.4 // indirect
 	github.com/xlab/treeprint v1.2.0 // indirect
-	go.yaml.in/yaml/v2 v2.4.2 // indirect
+	go.opentelemetry.io/otel v1.36.0 // indirect
+	go.opentelemetry.io/otel/trace v1.36.0 // indirect
+	go.yaml.in/yaml/v2 v2.4.3 // indirect
 	go.yaml.in/yaml/v3 v3.0.4 // indirect
-	golang.org/x/net v0.38.0 // indirect
-	golang.org/x/oauth2 v0.27.0 // indirect
-	golang.org/x/sync v0.12.0 // indirect
-	golang.org/x/term v0.30.0 // indirect
-	golang.org/x/text v0.23.0 // indirect
+	golang.org/x/mod v0.31.0 // indirect
+	golang.org/x/net v0.48.0 // indirect
+	golang.org/x/oauth2 v0.30.0 // indirect
+	golang.org/x/sync v0.19.0 // indirect
+	golang.org/x/term v0.38.0 // indirect
+	golang.org/x/text v0.32.0 // indirect
 	golang.org/x/time v0.9.0 // indirect
-	google.golang.org/protobuf v1.36.5 // indirect
-	gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
+	golang.org/x/tools v0.40.0 // indirect
+	google.golang.org/protobuf v1.36.8 // indirect
+	gopkg.in/evanphx/json-patch.v4 v4.13.0 // indirect
 	gopkg.in/inf.v0 v0.9.1 // indirect
-	k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b // indirect
-	k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 // indirect
-	sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 // indirect
+	gopkg.in/yaml.v2 v2.4.0 // indirect
+	k8s.io/component-base v0.35.0 // indirect
+	k8s.io/gengo/v2 v2.0.0-20250922181213-ec3ebc5fd46b // indirect
+	k8s.io/kube-openapi v0.0.0-20250910181357-589584f1c912 // indirect
+	k8s.io/utils v0.0.0-20251002143259-bc988d571ff4 // indirect
+	sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 // indirect
 	sigs.k8s.io/kustomize/api v0.20.1 // indirect
 	sigs.k8s.io/kustomize/kyaml v0.20.1 // indirect
 	sigs.k8s.io/randfill v1.0.0 // indirect

clitools/go.sum

@@ -1,7 +1,19 @@
+cel.dev/expr v0.24.0 h1:56OvJKSH3hDGL0ml5uSxZmz3/3Pq4tJ+fb1unVLAFcY=
+cel.dev/expr v0.24.0/go.mod h1:hLPLo1W4QUmuYdA72RBX06QTs6MXw941piREPl3Yfiw=
 github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 h1:L/gRVlceqvL25UVaW/CKtUDjefjrs0SPonmDGUVOYP0=
 github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
+github.com/Masterminds/semver/v3 v3.4.0 h1:Zog+i5UMtVoCU8oKka5P7i9q9HgrJeGzI9SA1Xbatp0=
+github.com/Masterminds/semver/v3 v3.4.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM=
+github.com/antlr4-go/antlr/v4 v4.13.0 h1:lxCg3LAv+EUK6t1i0y1V6/SLeUi0eKEKdhQAlS8TVTI=
+github.com/antlr4-go/antlr/v4 v4.13.0/go.mod h1:pfChB/xh/Unjila75QW7+VU4TSnWnnk9UTnmpPaOR2g=
+github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
+github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
 github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM=
 github.com/blang/semver/v4 v4.0.0/go.mod h1:IbckMUScFkM3pff0VJDNKRiT6TG/YpiHIM2yvyW5YoQ=
+github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
+github.com/cenkalti/backoff/v4 v4.3.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
+github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
+github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
 github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
 github.com/creack/pty v1.1.18 h1:n56/Zwd5o6whRC5PMGretI4IdRLlmBXYNjScPaBgsbY=
@@ -11,12 +23,20 @@ github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/emicklei/go-restful/v3 v3.12.2 h1:DhwDP0vY3k8ZzE0RunuJy8GhNpPL6zqLkDf9B/a0/xU=
 github.com/emicklei/go-restful/v3 v3.12.2/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
+github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM=
+github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU=
+github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
+github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
+github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
+github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
 github.com/fxamacker/cbor/v2 v2.9.0 h1:NpKPmjDBgUfBms6tr6JZkTHtfFGcMKsw3eGcmD/sapM=
 github.com/fxamacker/cbor/v2 v2.9.0/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ=
 github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA=
 github.com/go-errors/errors v1.4.2/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og=
-github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
-github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
+github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
+github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
+github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
+github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
 github.com/go-openapi/jsonpointer v0.19.6/go.mod h1:osyAmYz/mB/C3I+WsTTSgw1ONzaLJoLCyoi6/zppojs=
 github.com/go-openapi/jsonpointer v0.21.0 h1:YgdVicSA9vH5RiHs9TZW5oyafXZFc6+2Vc1rr/O9oNQ=
 github.com/go-openapi/jsonpointer v0.21.0/go.mod h1:IUyH9l/+uyhIYQ/PXVA41Rexl+kOkAPDdXEYns6fzUY=
@@ -27,29 +47,31 @@ github.com/go-openapi/swag v0.23.0 h1:vsEVJDUo2hPJ2tu0/Xc+4noaxyEffXNIs3cOULZ+Gr
 github.com/go-openapi/swag v0.23.0/go.mod h1:esZ8ITTYEsH1V2trKHjAN8Ai7xHb8RV+YSZ577vPjgQ=
 github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1vB6EwHI=
 github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8=
-github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
-github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
+github.com/gobuffalo/flect v1.0.3 h1:xeWBM2nui+qnVvNM4S3foBhCAL2XgPU+a7FdpelbTq4=
+github.com/gobuffalo/flect v1.0.3/go.mod h1:A5msMlrHtLqh9umBSnvabjsMrCcCpAyzglnDvkbYKHs=
 github.com/google/btree v1.1.3 h1:CVpQJjYgC4VbzxeGVHfvZrv1ctoYCAI8vbl07Fcxlyg=
 github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4=
+github.com/google/cel-go v0.26.0 h1:DPGjXackMpJWH680oGY4lZhYjIameYmR+/6RBdDGmaI=
+github.com/google/cel-go v0.26.0/go.mod h1:A9O8OU9rdvrK5MQyrqfIxo1a0u4g3sF8KB6PUIaryMM=
 github.com/google/gnostic-models v0.7.0 h1:qwTtogB15McXDaNqTZdzPJRHvaVJlAl+HVQnLmJEJxo=
 github.com/google/gnostic-models v0.7.0/go.mod h1:whL5G0m6dmc5cPxKc5bdKdEN3UjI7OUGxBlw57miDrQ=
 github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
 github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
 github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
-github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db h1:097atOisP2aRj7vFgYQBbFN4U4JNXUNYpxael3UzMyo=
-github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db/go.mod h1:vavhavw2zAxS5dIdcRluK6cSGGPlZynqzFM8NdvU144=
+github.com/google/pprof v0.0.0-20250403155104-27863c87afa6 h1:BHT72Gu3keYf3ZEu2J0b1vyeLSOYI8bm5wbJM/8yDe8=
+github.com/google/pprof v0.0.0-20250403155104-27863c87afa6/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA=
 github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
 github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79 h1:+ngKgrYPPJrOjhax5N+uePQ0Fh1Z7PheYoUI/0nzkPA=
 github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79/go.mod h1:FecbI9+v66THATjSRHfNgh1IVFe/9kFxbXtjV0ctIMA=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.3 h1:5ZPtiqj0JL5oKWmcsq4VMaAW5ukBEgSGXEN89zeH1Jo=
+github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.3/go.mod h1:ndYquD05frm2vACXE1nsccT4oJzjhw2arTS2cpUD1PI=
 github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
 github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
 github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
 github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
 github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
 github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
-github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
-github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
 github.com/klauspost/compress v1.18.5 h1:/h1gH5Ce+VWNLSWqPzOVn6XBO+vJbCNGvjoaGBFW2IE=
 github.com/klauspost/compress v1.18.5/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
 github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
@@ -59,10 +81,17 @@ github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
 github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
 github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
+github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
+github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
 github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de h1:9TO3cAIGXtEhnIaL+V+BEER86oLrvS+kWobKpbJuye0=
 github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de/go.mod h1:zAbeS9B/r2mtpb6U+EI2rYA5OAXxsYw6wTamcNW+zcE=
 github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0=
 github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
+github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
+github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
+github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
+github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
+github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
 github.com/moby/term v0.5.0 h1:xt8Q1nalod/v7BqbG21f8mQPqH+xAaC9C3N3wfWbVP0=
 github.com/moby/term v0.5.0/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y=
 github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
@@ -75,25 +104,38 @@ github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 h1:n6/
 github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00/go.mod h1:Pm3mSP3c5uWn86xMLZ5Sa7JB9GsEZySvHYXCTK4E9q4=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
-github.com/onsi/ginkgo/v2 v2.21.0 h1:7rg/4f3rB88pb5obDgNZrNHrQ4e6WpjonchcpuBRnZM=
-github.com/onsi/ginkgo/v2 v2.21.0/go.mod h1:7Du3c42kxCUegi0IImZ1wUQzMBVecgIHjR1C+NkhLQo=
-github.com/onsi/gomega v1.35.1 h1:Cwbd75ZBPxFSuZ6T+rN/WCb/gOc6YgFBXLlZLhC7Ds4=
-github.com/onsi/gomega v1.35.1/go.mod h1:PvZbdDc8J6XJEpDK4HCuRBm8a6Fzp9/DmhC9C7yFlog=
+github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE=
+github.com/nxadm/tail v1.4.8/go.mod h1:+ncqLTQzXmGhMZNUePPaPqPvBxHAIsmXswZKocGu+AU=
+github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE=
+github.com/onsi/ginkgo v1.16.5/go.mod h1:+E8gABHa3K6zRBolWtd+ROzc/U5bkGt0FwiG042wbpU=
+github.com/onsi/ginkgo/v2 v2.27.2 h1:LzwLj0b89qtIy6SSASkzlNvX6WktqurSHwkk2ipF/Ns=
+github.com/onsi/ginkgo/v2 v2.27.2/go.mod h1:ArE1D/XhNXBXCBkKOLkbsb2c81dQHCRcF5zwn/ykDRo=
+github.com/onsi/gomega v1.38.3 h1:eTX+W6dobAYfFeGC2PV6RwXRu/MyT+cQguijutvkpSM=
+github.com/onsi/gomega v1.38.3/go.mod h1:ZCU1pkQcXDO5Sl9/VVEGlDyp+zm0m1cmeG5TOzLgdh4=
 github.com/peterbourgon/diskv v2.0.1+incompatible h1:UBdAOUP5p4RWqPBg048CAvpKN+vxiaj6gdUUzhl4XmI=
 github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
-github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
-github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
-github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII=
-github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o=
+github.com/prometheus/client_golang v1.23.2 h1:Je96obch5RDVy3FDMndoUsjAhG5Edi49h0RJWRi/o0o=
+github.com/prometheus/client_golang v1.23.2/go.mod h1:Tb1a6LWHB3/SPIzCoaDXI4I8UHKeFTEQ1YCr+0Gyqmg=
+github.com/prometheus/client_model v0.6.2 h1:oBsgwpGs7iVziMvrGhE53c/GrLUsZdHnqNwqPLxwZyk=
+github.com/prometheus/client_model v0.6.2/go.mod h1:y3m2F6Gdpfy6Ut/GBsUqTWZqCUvMVzSfMLjcu6wAwpE=
+github.com/prometheus/common v0.66.1 h1:h5E0h5/Y8niHc5DlaLlWLArTQI7tMrsfQjHV+d9ZoGs=
+github.com/prometheus/common v0.66.1/go.mod h1:gcaUsgf3KfRSwHY4dIMXLPV0K/Wg1oZ8+SbZk/HH/dA=
+github.com/prometheus/procfs v0.16.1 h1:hZ15bTNuirocR6u0JZ6BAHHmwS1p8B4P6MRqxtzMyRg=
+github.com/prometheus/procfs v0.16.1/go.mod h1:teAbpZRB1iIAJYREa1LsoWUXykVXA1KlTmWl8x/U+Is=
+github.com/rogpeppe/go-internal v1.14.1 h1:UQB4HGPB6osV0SQTLymcB4TgvyWu6ZyliaW0tI/otEQ=
+github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7so1lCWt35ZSgc=
 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
 github.com/sergi/go-diff v1.2.0 h1:XU+rvMAioB0UC3q1MFrIQy4Vo5/4VsRDQQXHsEya6xQ=
 github.com/sergi/go-diff v1.2.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM=
-github.com/spf13/cobra v1.9.1 h1:CXSaggrXdbHK9CF+8ywj8Amf7PBRmPCOJugH954Nnlo=
-github.com/spf13/cobra v1.9.1/go.mod h1:nDyEzZ8ogv936Cinf6g1RU9MRY64Ir93oCnqb9wxYW0=
-github.com/spf13/pflag v1.0.6 h1:jFzHGLGAlb3ruxLB8MhbI6A8+AQX/2eW4qeyNZXNp2o=
-github.com/spf13/pflag v1.0.6/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
+github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
+github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
+github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+github.com/stoewer/go-strcase v1.3.0 h1:g0eASXYtp+yvN9fK8sH94oCIk0fau9uV1/ZdJ0AVEzs=
+github.com/stoewer/go-strcase v1.3.0/go.mod h1:fAH5hQ5pehh+j3nZfvwdk2RgEgQjAoM8wodgtPmh1xo=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
 github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
@@ -104,92 +146,115 @@ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
 github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
 github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
 github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
-github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
-github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
+github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
+github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
 github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
 github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg=
 github.com/xlab/treeprint v1.2.0 h1:HzHnuAF1plUN2zGlAFHbSQP2qJ0ZAD3XF5XD7OesXRQ=
 github.com/xlab/treeprint v1.2.0/go.mod h1:gj5Gd3gPdKtR1ikdDK6fnFLdmIS0X30kTTuNd/WEJu0=
-github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
-github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
+go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=
+go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
+go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0 h1:F7Jx+6hwnZ41NSFTO5q4LYDtJRXBf2PD0rNBkeB/lus=
+go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0/go.mod h1:UHB22Z8QsdRDrnAtX4PntOl36ajSxcdUMt1sF7Y6E7Q=
+go.opentelemetry.io/otel v1.36.0 h1:UumtzIklRBY6cI/lllNZlALOF5nNIzJVb16APdvgTXg=
+go.opentelemetry.io/otel v1.36.0/go.mod h1:/TcFMXYjyRNh8khOAO9ybYkqaDBb/70aVwkNML4pP8E=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.34.0 h1:OeNbIYk/2C15ckl7glBlOBp5+WlYsOElzTNmiPW/x60=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.34.0/go.mod h1:7Bept48yIeqxP2OZ9/AqIpYS94h2or0aB4FypJTc8ZM=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.34.0 h1:tgJ0uaNS4c98WRNUEx5U3aDlrDOI5Rs+1Vifcw4DJ8U=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.34.0/go.mod h1:U7HYyW0zt/a9x5J1Kjs+r1f/d4ZHnYFclhYY2+YbeoE=
+go.opentelemetry.io/otel/metric v1.36.0 h1:MoWPKVhQvJ+eeXWHFBOPoBOi20jh6Iq2CcCREuTYufE=
+go.opentelemetry.io/otel/metric v1.36.0/go.mod h1:zC7Ks+yeyJt4xig9DEw9kuUFe5C3zLbVjV2PzT6qzbs=
+go.opentelemetry.io/otel/sdk v1.36.0 h1:b6SYIuLRs88ztox4EyrvRti80uXIFy+Sqzoh9kFULbs=
+go.opentelemetry.io/otel/sdk v1.36.0/go.mod h1:+lC+mTgD+MUWfjJubi2vvXWcVxyr9rmlshZni72pXeY=
+go.opentelemetry.io/otel/trace v1.36.0 h1:ahxWNuqZjpdiFAyrIoQ4GIiAIhxAunQR6MUoKrsNd4w=
+go.opentelemetry.io/otel/trace v1.36.0/go.mod h1:gQ+OnDZzrybY4k4seLzPAWNwVBBVlF2szhehOBB/tGA=
+go.opentelemetry.io/proto/otlp v1.5.0 h1:xJvq7gMzB31/d406fB8U5CBdyQGw4P399D1aQWU/3i4=
+go.opentelemetry.io/proto/otlp v1.5.0/go.mod h1:keN8WnHxOy8PG0rQZjJJ5A2ebUoafqWp0eVQ4yIXvJ4=
 go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
 go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
-go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI=
-go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU=
+go.yaml.in/yaml/v2 v2.4.3 h1:6gvOSjQoTB3vt1l+CU+tSyi/HOjfOjRLJ4YwYZGwRO0=
+go.yaml.in/yaml/v2 v2.4.3/go.mod h1:zSxWcmIDjOzPXpjlTTbAsKokqkDNAVtZO0WOMiT90s8=
 go.yaml.in/yaml/v3 v3.0.4 h1:tfq32ie2Jv2UxXFdLJdh3jXuOzWiL1fo0bu/FbuKpbc=
 go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
-golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
-golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
-golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
-golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
-golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
-golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
-golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
-golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
-golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
-golang.org/x/net v0.38.0 h1:vRMAPTMaeGqVhG5QyLJHqNDwecKTomGeqbnfZyKlBI8=
-golang.org/x/net v0.38.0/go.mod h1:ivrbrMbzFq5J41QOQh0siUuly180yBYtLp+CKbEaFx8=
-golang.org/x/oauth2 v0.27.0 h1:da9Vo7/tDv5RH/7nZDz1eMGS/q1Vv1N/7FCrBhI9I3M=
-golang.org/x/oauth2 v0.27.0/go.mod h1:onh5ek6nERTohokkhCD/y2cV4Do3fxFHFuAejCkRWT8=
-golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.12.0 h1:MHc5BpPuC30uJk597Ri8TV3CNZcTLu6B6z4lJy+g6Jw=
-golang.org/x/sync v0.12.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
-golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
-golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 h1:2dVuKD2vS7b0QIHQbpyTISPd0LeHDbnYEryqj5Q1ug8=
+golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56/go.mod h1:M4RDyNAINzryxdtnbRXRL/OHtkFuWGRjvuhBJpk2IlY=
+golang.org/x/mod v0.31.0 h1:HaW9xtz0+kOcWKwli0ZXy79Ix+UW/vOfmWI5QVd2tgI=
+golang.org/x/mod v0.31.0/go.mod h1:43JraMp9cGx1Rx3AqioxrbrhNsLl2l/iNAvuBkrezpg=
+golang.org/x/net v0.48.0 h1:zyQRTTrjc33Lhh0fBgT/H3oZq9WuvRR5gPC70xpDiQU=
+golang.org/x/net v0.48.0/go.mod h1:+ndRgGjkh8FGtu1w1FGbEC31if4VrNVMuKTgcAAnQRY=
+golang.org/x/oauth2 v0.30.0 h1:dnDm7JmhM45NNpd8FDDeLhK6FwqbOf4MLCM9zb1BOHI=
+golang.org/x/oauth2 v0.30.0/go.mod h1:B++QgG3ZKulg6sRPGD/mqlHQs5rB3Ml9erfeDY7xKlU=
+golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
+golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
 golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
-golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
-golang.org/x/term v0.30.0 h1:PQ39fJZ+mfadBm0y5WlL4vlM7Sx1Hgf13sMIY2+QS9Y=
-golang.org/x/term v0.30.0/go.mod h1:NYYFdzHoI5wRh/h5tDMdMqCqPJZEuNqVR5xJLd/n67g=
-golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
-golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
-golang.org/x/text v0.23.0 h1:D71I7dUrlY+VX0gQShAThNGHFxZ13dGLBHQLVl1mJlY=
-golang.org/x/text v0.23.0/go.mod h1:/BLNzu4aZCJ1+kcD0DNRotWKage4q2rGVAg4o22unh4=
+golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
+golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
+golang.org/x/term v0.38.0 h1:PQ5pkm/rLO6HnxFR7N2lJHOZX6Kez5Y1gDSJla6jo7Q=
+golang.org/x/term v0.38.0/go.mod h1:bSEAKrOT1W+VSu9TSCMtoGEOUcKxOKgl3LE5QEF/xVg=
+golang.org/x/text v0.32.0 h1:ZD01bjUt1FQ9WJ0ClOL5vxgxOI/sVCNgX1YtKwcY0mU=
+golang.org/x/text v0.32.0/go.mod h1:o/rUWzghvpD5TXrTIBuJU77MTaN0ljMWE47kxGJQ7jY=
 golang.org/x/time v0.9.0 h1:EsRrnYcQiGH+5FfbgvV4AP7qEZstoyrHB0DzarOQ4ZY=
 golang.org/x/time v0.9.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
-golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
-golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
-golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
-golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
-golang.org/x/tools v0.26.0 h1:v/60pFQmzmT9ExmjDv2gGIfi3OqfKoEP6I5+umXlbnQ=
-golang.org/x/tools v0.26.0/go.mod h1:TPVVj70c7JJ3WCazhD8OdXcZg/og+b9+tH/KxylGwH0=
-golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-google.golang.org/protobuf v1.36.5 h1:tPhr+woSbjfYvY6/GPufUoYizxw1cF/yFoxJ2fmpwlM=
-google.golang.org/protobuf v1.36.5/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
+golang.org/x/tools v0.40.0 h1:yLkxfA+Qnul4cs9QA3KnlFu0lVmd8JJfoq+E41uSutA=
+golang.org/x/tools v0.40.0/go.mod h1:Ik/tzLRlbscWpqqMRjyWYDisX8bG13FrdXp3o4Sr9lc=
+golang.org/x/tools/go/expect v0.1.1-deprecated h1:jpBZDwmgPhXsKZC6WhL20P4b/wmnpsEAGHaNy0n/rJM=
+golang.org/x/tools/go/expect v0.1.1-deprecated/go.mod h1:eihoPOH+FgIqa3FpoTwguz/bVUSGBlGQU67vpBeOrBY=
+golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated h1:1h2MnaIAIXISqTFKdENegdpAgUXz6NrPEsbIeWaBRvM=
+golang.org/x/tools/go/packages/packagestest v0.1.1-deprecated/go.mod h1:RVAQXBGNv1ib0J382/DPCRS/BPnsGebyM1Gj5VSDpG8=
+google.golang.org/genproto/googleapis/api v0.0.0-20250303144028-a0af3efb3deb h1:p31xT4yrYrSM/G4Sn2+TNUkVhFCbG9y8itM2S6Th950=
+google.golang.org/genproto/googleapis/api v0.0.0-20250303144028-a0af3efb3deb/go.mod h1:jbe3Bkdp+Dh2IrslsFCklNhweNTBgSYanP1UXhJDhKg=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20250528174236-200df99c418a h1:v2PbRU4K3llS09c7zodFpNePeamkAwG3mPrAery9VeE=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20250528174236-200df99c418a/go.mod h1:qQ0YXyHHx3XkvlzUtpXDkS29lDSafHMZBAZDc03LQ3A=
+google.golang.org/grpc v1.72.2 h1:TdbGzwb82ty4OusHWepvFWGLgIbNo1/SUynEN0ssqv8=
+google.golang.org/grpc v1.72.2/go.mod h1:wH5Aktxcg25y1I3w7H69nHfXdOG3UiadoBtjh3izSDM=
+google.golang.org/protobuf v1.36.8 h1:xHScyCOEuuwZEc6UtSOvPbAT4zRh0xcNRYekJwfqyMc=
+google.golang.org/protobuf v1.36.8/go.mod h1:fuxRtAxBytpl4zzqUh6/eyUujkJdNiuEkXntxiD/uRU=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
-gopkg.in/evanphx/json-patch.v4 v4.12.0 h1:n6jtcsulIzXPJaxegRbvFNNrZDjbij7ny3gmSPG+6V4=
-gopkg.in/evanphx/json-patch.v4 v4.12.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWMG4EsWvDvM72M=
+gopkg.in/evanphx/json-patch.v4 v4.13.0 h1:czT3CmqEaQ1aanPc5SdlgQrrEIb8w/wwCvWWnfEbYzo=
+gopkg.in/evanphx/json-patch.v4 v4.13.0/go.mod h1:p8EYWUEYMpynmqDbY58zCKCFZw8pRWMG4EsWvDvM72M=
 gopkg.in/inf.v0 v0.9.1 h1:73M5CoZyi3ZLMOyDlQh031Cx6N9NDJ2Vvfl76EDAgDc=
 gopkg.in/inf.v0 v0.9.1/go.mod h1:cWUDdTG/fYaXco+Dcufb5Vnc6Gp2YChqWtbxRZE0mXw=
+gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
+gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
+gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
+gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
-k8s.io/api v0.34.0 h1:L+JtP2wDbEYPUeNGbeSa/5GwFtIA662EmT2YSLOkAVE=
-k8s.io/api v0.34.0/go.mod h1:YzgkIzOOlhl9uwWCZNqpw6RJy9L2FK4dlJeayUoydug=
-k8s.io/apiextensions-apiserver v0.34.0 h1:B3hiB32jV7BcyKcMU5fDaDxk882YrJ1KU+ZSkA9Qxoc=
-k8s.io/apiextensions-apiserver v0.34.0/go.mod h1:hLI4GxE1BDBy9adJKxUxCEHBGZtGfIg98Q+JmTD7+g0=
-k8s.io/apimachinery v0.34.0 h1:eR1WO5fo0HyoQZt1wdISpFDffnWOvFLOOeJ7MgIv4z0=
-k8s.io/apimachinery v0.34.0/go.mod h1:/GwIlEcWuTX9zKIg2mbw0LRFIsXwrfoVxn+ef0X13lw=
+k8s.io/api v0.35.0 h1:iBAU5LTyBI9vw3L5glmat1njFK34srdLmktWwLTprlY=
+k8s.io/api v0.35.0/go.mod h1:AQ0SNTzm4ZAczM03QH42c7l3bih1TbAXYo0DkF8ktnA=
+k8s.io/apiextensions-apiserver v0.35.0 h1:3xHk2rTOdWXXJM+RDQZJvdx0yEOgC0FgQ1PlJatA5T4=
+k8s.io/apiextensions-apiserver v0.35.0/go.mod h1:E1Ahk9SADaLQ4qtzYFkwUqusXTcaV2uw3l14aqpL2LU=
+k8s.io/apimachinery v0.35.0 h1:Z2L3IHvPVv/MJ7xRxHEtk6GoJElaAqDCCU0S6ncYok8=
+k8s.io/apimachinery v0.35.0/go.mod h1:jQCgFZFR1F4Ik7hvr2g84RTJSZegBc8yHgFWKn//hns=
+k8s.io/apiserver v0.35.0 h1:CUGo5o+7hW9GcAEF3x3usT3fX4f9r8xmgQeCBDaOgX4=
+k8s.io/apiserver v0.35.0/go.mod h1:QUy1U4+PrzbJaM3XGu2tQ7U9A4udRRo5cyxkFX0GEds=
 k8s.io/cli-runtime v0.34.0 h1:N2/rUlJg6TMEBgtQ3SDRJwa8XyKUizwjlOknT1mB2Cw=
 k8s.io/cli-runtime v0.34.0/go.mod h1:t/skRecS73Piv+J+FmWIQA2N2/rDjdYSQzEE67LUUs8=
-k8s.io/client-go v0.34.0 h1:YoWv5r7bsBfb0Hs2jh8SOvFbKzzxyNo0nSb0zC19KZo=
-k8s.io/client-go v0.34.0/go.mod h1:ozgMnEKXkRjeMvBZdV1AijMHLTh3pbACPvK7zFR+QQY=
+k8s.io/client-go v0.35.0 h1:IAW0ifFbfQQwQmga0UdoH0yvdqrbwMdq9vIFEhRpxBE=
+k8s.io/client-go v0.35.0/go.mod h1:q2E5AAyqcbeLGPdoRB+Nxe3KYTfPce1Dnu1myQdqz9o=
+k8s.io/code-generator v0.35.0 h1:TvrtfKYZTm9oDF2z+veFKSCcgZE3Igv0svY+ehCmjHQ=
+k8s.io/code-generator v0.35.0/go.mod h1:iS1gvVf3c/T71N5DOGYO+Gt3PdJ6B9LYSvIyQ4FHzgc=
+k8s.io/component-base v0.35.0 h1:+yBrOhzri2S1BVqyVSvcM3PtPyx5GUxCK2tinZz1G94=
+k8s.io/component-base v0.35.0/go.mod h1:85SCX4UCa6SCFt6p3IKAPej7jSnF3L8EbfSyMZayJR0=
+k8s.io/gengo/v2 v2.0.0-20250922181213-ec3ebc5fd46b h1:gMplByicHV/TJBizHd9aVEsTYoJBnnUAT5MHlTkbjhQ=
+k8s.io/gengo/v2 v2.0.0-20250922181213-ec3ebc5fd46b/go.mod h1:CgujABENc3KuTrcsdpGmrrASjtQsWCT7R99mEV4U/fM=
 k8s.io/klog/v2 v2.130.1 h1:n9Xl7H1Xvksem4KFG4PYbdQCQxqc/tTUyrgXaOhHSzk=
 k8s.io/klog/v2 v2.130.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE=
-k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b h1:MloQ9/bdJyIu9lb1PzujOPolHyvO06MXG5TUIj2mNAA=
-k8s.io/kube-openapi v0.0.0-20250710124328-f3f2b991d03b/go.mod h1:UZ2yyWbFTpuhSbFhv24aGNOdoRdJZgsIObGBUaYVsts=
-k8s.io/utils v0.0.0-20250604170112-4c0f3b243397 h1:hwvWFiBzdWw1FhfY1FooPn3kzWuJ8tmbZBHi4zVsl1Y=
-k8s.io/utils v0.0.0-20250604170112-4c0f3b243397/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
-sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8 h1:gBQPwqORJ8d8/YNZWEjoZs7npUVDpVXUUOFfW6CgAqE=
-sigs.k8s.io/json v0.0.0-20241014173422-cfa47c3a1cc8/go.mod h1:mdzfpAEoE6DHQEN0uh9ZbOCuHbLK5wOm7dK4ctXE9Tg=
+k8s.io/kube-openapi v0.0.0-20250910181357-589584f1c912 h1:Y3gxNAuB0OBLImH611+UDZcmKS3g6CthxToOb37KgwE=
+k8s.io/kube-openapi v0.0.0-20250910181357-589584f1c912/go.mod h1:kdmbQkyfwUagLfXIad1y2TdrjPFWp2Q89B3qkRwf/pQ=
+k8s.io/utils v0.0.0-20251002143259-bc988d571ff4 h1:SjGebBtkBqHFOli+05xYbK8YF1Dzkbzn+gDM4X9T4Ck=
+k8s.io/utils v0.0.0-20251002143259-bc988d571ff4/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0=
+sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.31.2 h1:jpcvIRr3GLoUoEKRkHKSmGjxb6lWwrBlJsXc+eUYQHM=
+sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.31.2/go.mod h1:Ve9uj1L+deCXFrPOk1LpFXqTg7LCFzFso6PA48q/XZw=
+sigs.k8s.io/controller-tools v0.20.1 h1:gkfMt9YodI0K85oT8rVi80NTXO/kDmabKR5Ajn5GYxs=
+sigs.k8s.io/controller-tools v0.20.1/go.mod h1:b4qPmjGU3iZwqn34alUU5tILhNa9+VXK+J3QV0fT/uU=
+sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730 h1:IpInykpT6ceI+QxKBbEflcR5EXP7sU1kvOlxwZh5txg=
+sigs.k8s.io/json v0.0.0-20250730193827-2d320260d730/go.mod h1:mdzfpAEoE6DHQEN0uh9ZbOCuHbLK5wOm7dK4ctXE9Tg=
 sigs.k8s.io/kustomize/api v0.20.1 h1:iWP1Ydh3/lmldBnH/S5RXgT98vWYMaTUL1ADcr+Sv7I=
 sigs.k8s.io/kustomize/api v0.20.1/go.mod h1:t6hUFxO+Ph0VxIk1sKp1WS0dOjbPCtLJ4p8aADLwqjM=
 sigs.k8s.io/kustomize/kyaml v0.20.1 h1:PCMnA2mrVbRP3NIB6v9kYCAc38uvFLVs8j/CD567A78=

clitools/hack/boilerplate.go.txt

@@ -0,0 +1 @@
+/* MIT License */

clitools/hack/tool.go

@@ -0,0 +1,9 @@
+//go:build tools
+// +build tools
+
+package tools
+
+import (
+	_ "k8s.io/code-generator"
+	_ "sigs.k8s.io/controller-tools/cmd/controller-gen"
+)

clitools/hack/update-codegen.sh

@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+MODULE_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+echo "MODULE ROOT: ${MODULE_ROOT}"
+
+cd "${MODULE_ROOT}"
+mkdir -p \
+	"${MODULE_ROOT}/pkg/generated/clientset" \
+	"${MODULE_ROOT}/pkg/generated/listers" \
+	"${MODULE_ROOT}/pkg/generated/informers"
+
+controller-gen \
+  object:headerFile=hack/boilerplate.go.txt \
+  paths=./pkg/apis/...
+
+MODULE="$(go list -m -f '{{.Path}}')"
+echo "MODULE: ${MODULE}"
+
+CODEGEN_PKG="$(go list -f '{{.Dir}}' -m k8s.io/code-generator)"
+echo "CODEGEN PKG: ${CODEGEN_PKG}"
+
+source "${CODEGEN_PKG}/kube_codegen.sh"
+
+mkdir -p "${MODULE_ROOT}/pkg/generated"
+
+kube::codegen::gen_helpers \
+  --boilerplate "${MODULE_ROOT}/hack/boilerplate.go.txt" \
+  "${MODULE_ROOT}/pkg/apis"
+
+kube::codegen::gen_client \
+  --with-watch \
+  --output-dir "${MODULE_ROOT}/pkg/generated" \
+  --output-pkg "${MODULE}/pkg/generated" \
+  --boilerplate "${MODULE_ROOT}/hack/boilerplate.go.txt" \
+  "${MODULE_ROOT}/pkg/apis"

clitools/makefile

@@ -1,25 +1,45 @@
+include ../build.env
+-include ../build.env.work
+export
+
+BUILD_PLATFORM ?= linux/amd64
+
 # Should be the same as upstream version in production
 VERSION ?= dev
-UBOOT_VERSION=v2026.01
+UBOOT_VERSION ?= v2026.01
 
 # Target kube version
 KUBE_VERSION ?= v1.33.3
 
 GIT_REV := $(shell git rev-parse HEAD)
 
-PACKAGES_DIR := packages
-BIN_DIR      := bin
-OUT_DIR      := out
-
-UBOOT_TAR     := $(PACKAGES_DIR)/uboot-$(UBOOT_VERSION).tar.gz
+PACKAGES_DIR    := packages
+OUT_DIR         := out
+UBOOT_TOOLS_OUT := $(OUT_DIR)/uboot-tools
 
+UBOOT_TAR      := $(PACKAGES_DIR)/uboot-$(UBOOT_VERSION).tar.gz
 BUILDINFO_FILE := pkg/buildinfo/buildinfo_gen.go
-CRD_PATHS := ./pkg/apis/...
+ASSETS_PATH      := ./pkg/assets
+
+BUILDX_BUILDER      := container-builder
+LOCAL_REGISTRY      := registry
+LOCAL_REGISTRY_PORT := 5000
+
+CTL_BUILD_BASE_REPO  := localhost:5000/monok8s/ctl-build-base
+CTL_IMAGE_REPO       := localhost:5000/monok8s/node-control
+
+CTL_BUILD_BASE_IMAGE := $(CTL_BUILD_BASE_REPO):$(VERSION)
+CTL_IMAGE            := $(CTL_IMAGE_REPO):$(VERSION)
+
+DOWNLOAD_PACKAGES_STAMP := $(PACKAGES_DIR)/.download-packages.stamp
 
 $(PACKAGES_DIR):
 	mkdir -p $@
 
-# Never cache this
+$(OUT_DIR):
+	mkdir -p $@
+
+# Keep buildinfo host-side since it is just generated source and lets Docker see it in build context.
 .buildinfo:
 	@mkdir -p $(dir $(BUILDINFO_FILE))
 	@printf '%s\n' \
@@ -33,44 +53,131 @@ $(PACKAGES_DIR):
 		')' \
 	> $(BUILDINFO_FILE)
 
-$(UBOOT_TAR): | $(PACKAGES_DIR)
-	git clone --depth 1 --branch v2026.01 --filter=blob:none https://github.com/u-boot/u-boot.git $(OUT_DIR)/u-boot-$(UBOOT_VERSION)
-	tar -C "$(OUT_DIR)/u-boot-$(UBOOT_VERSION)" -zcf "$@" .
-	rm -rf $(OUT_DIR)/u-boot-$(UBOOT_VERSION)
-	test -f $@
+ensure-buildx:
+	@if ! docker buildx inspect $(BUILDX_BUILDER) >/dev/null 2>&1; then \
+		echo "Creating buildx builder $(BUILDX_BUILDER)..."; \
+		docker buildx create \
+			--name $(BUILDX_BUILDER) \
+			--driver docker-container \
+			--driver-opt network=host \
+			--bootstrap --use; \
+	else \
+		echo "Using existing buildx builder $(BUILDX_BUILDER)"; \
+		docker buildx use $(BUILDX_BUILDER); \
+	fi
 
-uboot-tools: $(UBOOT_TAR)
-	docker buildx build --platform linux/arm64 \
+ensure-registry:
+	@if ! docker container inspect $(LOCAL_REGISTRY) >/dev/null 2>&1; then \
+		echo "Creating local registry..."; \
+		docker run -d \
+			--restart=always \
+			-p $(LOCAL_REGISTRY_PORT):5000 \
+			--name $(LOCAL_REGISTRY) \
+			registry:2; \
+	else \
+		if [ "$$(docker inspect -f '{{.State.Running}}' $(LOCAL_REGISTRY))" != "true" ]; then \
+			echo "Starting existing local registry..."; \
+			docker start $(LOCAL_REGISTRY); \
+		fi; \
+	fi
+
+$(DOWNLOAD_PACKAGES_STAMP): docker/download-packages.Dockerfile makefile | $(PACKAGES_DIR)
+	docker build \
+		-f docker/download-packages.Dockerfile \
+		--build-arg UBOOT_VERSION=$(UBOOT_VERSION) \
+		--output type=local,dest=./$(PACKAGES_DIR) .
+	@touch $@
+
+uboot-tools: $(DOWNLOAD_PACKAGES_STAMP)
+	rm -rf "$(UBOOT_TOOLS_OUT)"
+	mkdir -p "$(UBOOT_TOOLS_OUT)"
+	docker buildx build \
+		--platform linux/amd64,linux/arm64 \
 		-f docker/uboot-tools.Dockerfile \
 		--build-arg UBOOT_VERSION=$(UBOOT_VERSION) \
 		--build-arg UBOOT_TAR=$(UBOOT_TAR) \
-		--output type=local,dest=./$(OUT_DIR) .
+		--output type=local,dest=./$(UBOOT_TOOLS_OUT),platform-split=true .
 
-build: .buildinfo
-	mkdir -p $(BIN_DIR) $(OUT_DIR)/crds
-	controller-gen crd paths=$(CRD_PATHS) output:crd:dir=$(OUT_DIR)/crds
-	GOOS=linux GOARCH=arm64 go build -o $(BIN_DIR)/ctl-linux-aarch64-$(VERSION) ./cmd/ctl
+ctl-build-base: ensure-buildx ensure-registry
+	docker buildx build \
+		--platform linux/amd64,linux/arm64 \
+		-f docker/ctl-build-base.Dockerfile \
+		-t $(CTL_BUILD_BASE_IMAGE) \
+		--output type=image,push=true,registry.insecure=true .
 
-build-agent: build uboot-tools
-	docker build \
+build-crds: ctl-build-base | $(OUT_DIR)
+	mkdir -p "$(OUT_DIR)/crds"
+	docker buildx build \
+		--platform $(BUILD_PLATFORM) \
+		-f docker/crdgen.Dockerfile \
+		--build-arg BASE_IMAGE=$(CTL_BUILD_BASE_IMAGE) \
+		--output type=local,dest=./$(OUT_DIR)/crds .
+	rm -rf "$(ASSETS_PATH)/crds"
+	mkdir -p "$(ASSETS_PATH)/crds"
+	cp -R "$(OUT_DIR)/crds/." "$(ASSETS_PATH)/crds/"
+
+build-agent: .buildinfo build-crds uboot-tools
+	docker buildx build \
+		--platform linux/amd64,linux/arm64 \
 		-f docker/ctl-agent.Dockerfile \
-		--platform=linux/arm64 \
+		--build-arg BASE_IMAGE=$(CTL_BUILD_BASE_IMAGE) \
 		--build-arg VERSION=$(VERSION) \
-		-t localhost/monok8s/control-agent:$(VERSION) .
+		-t $(CTL_IMAGE) \
+		--output type=image,push=true,registry.insecure=true .
 
-build-local: .buildinfo
-	mkdir -p $(BIN_DIR)
-	go build -o $(BIN_DIR)/ctl-$(VERSION) ./cmd/ctl
+build-local: .buildinfo build-crds uboot-tools
+	docker buildx build \
+		--platform linux/arm64 \
+		-f docker/ctl-agent.Dockerfile \
+		--build-arg BASE_IMAGE=$(CTL_BUILD_BASE_IMAGE) \
+		--build-arg VERSION=$(VERSION) \
+		--load \
+		-t localhost/monok8s/node-control:$(VERSION) .
+
+push-agent: .buildinfo build-crds uboot-tools
+	test -n "$(IMAGE_REPOSITORY)"
+	docker buildx build \
+		--platform linux/amd64,linux/arm64 \
+		-f docker/ctl-agent.Dockerfile \
+		--build-arg BASE_IMAGE=$(CTL_BUILD_BASE_IMAGE) \
+		--build-arg VERSION=$(VERSION) \
+		-t $(IMAGE_REPOSITORY)/node-control:$(VERSION) \
+		--push .
 
 run-agent:
-	go run -tags dev ./cmd/ctl agent --env-file ./out/cluster.env
+	docker run --rm \
+		-v "$$(pwd)/out:/work/out" \
+		$(CTL_IMAGE) \
+		agent --env-file /work/out/cluster.env
 
 clean:
-	-docker image rm localhost/monok8s/control-agent:$(VERSION)
-	rm -rf $(BIN_DIR) \
-		$(BUILDINFO_FILE) \
-		$(OUT_DIR)/crds
+	-docker image rm localhost/monok8s/node-control:$(VERSION) >/dev/null 2>&1 || true
+	rm -rf \
+		$(OUT_DIR)/crds \
+		$(BUILDINFO_FILE)
 
-all: build build-agent build-local
+distclean: clean
+	rm -rf $(OUT_DIR)
 
-.PHONY: clean all run .buildinfo build build-local build-agent uboot-tools
+dockerclean:
+	@echo "Removing tagged images..."
+	- docker rmi \
+		localhost/monok8s/ctl-build-base:$(VERSION) \
+		localhost/monok8s/node-control:$(VERSION) \
+		localhost/monok8s/crdgen:$(VERSION) \
+		2>/dev/null || true
+
+	@echo "Removing dangling build cache/images..."
+	- docker image prune -f
+	- docker builder prune -f
+
+pkgclean:
+	rm -rf $(PACKAGES_DIR)
+
+all: build-agent build-local
+
+.PHONY: \
+	all clean dockerclean \
+	.buildinfo ensure-buildx ensure-registry \
+	build-crds build-local build-agent build-agent-local push-agent \
+	uboot-tools run-agent run-agent-local

clitools/pkg/apis/monok8s/v1alpha1/groupversion_info.go

@@ -14,10 +14,12 @@ var (
 	APIVersion = "monok8s.io/v1alpha1"
 
 	AltPartDeviceLink = "/dev/mksaltpart"
-	Annotation        = "monok8s.io/annotation"
 	BootStateFile     = "/run/monok8s/boot-state.env"
 	CatalogURL        = "https://example.com/monok8s.io/v1alpha1/catalog.yaml"
-	ControlAgentKey   = "monok8s.io/control-agent"
+	NodeControlKey    = "monok8s.io/node-control"
+	NodeControlName   = "node-control"
+	ControllerName    = "node-controller"
+	NodeAgentName     = "node-agent"
 	EnvConfigDir      = "/opt/monok8s/config"
 	Label             = "monok8s.io/label"
 	MonoKSConfigCRD   = "monoksconfigs.monok8s.io"
@@ -25,18 +27,29 @@ var (
 )
 
 var (
-	GroupVersion  = schema.GroupVersion{Group: Group, Version: Version}
-	SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)
-	AddToScheme   = SchemeBuilder.AddToScheme
+	SchemeGroupVersion = schema.GroupVersion{Group: Group, Version: Version}
+	SchemeBuilder      = runtime.NewSchemeBuilder(addKnownTypes)
+	AddToScheme        = SchemeBuilder.AddToScheme
 )
 
 func addKnownTypes(scheme *runtime.Scheme) error {
-	scheme.AddKnownTypes(GroupVersion,
+	scheme.AddKnownTypes(SchemeGroupVersion,
 		&MonoKSConfig{},
 		&MonoKSConfigList{},
 		&OSUpgrade{},
 		&OSUpgradeList{},
+		&OSUpgradeProgress{},
+		&OSUpgradeProgressList{},
 	)
-	metav1.AddToGroupVersion(scheme, GroupVersion)
+	metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
 	return nil
 }
+
+func NodeAgentLabels() map[string]string {
+	return map[string]string{
+		"app.kubernetes.io/name":       NodeAgentName,
+		"app.kubernetes.io/component":  "agent",
+		"app.kubernetes.io/part-of":    "monok8s",
+		"app.kubernetes.io/managed-by": NodeControlName,
+	}
+}

clitools/pkg/apis/monok8s/v1alpha1/monoksconfig.go

@@ -25,7 +25,7 @@ type MonoKSConfigSpec struct {
 	ClusterDomain                 string            `json:"clusterDomain,omitempty" yaml:"clusterDomain,omitempty"`
 	ClusterRole                   string            `json:"clusterRole,omitempty" yaml:"clusterRole,omitempty"`
 	InitControlPlane              bool              `json:"initControlPlane,omitempty" yaml:"initControlPlane,omitempty"`
-	EnableControlAgent            bool              `json:"enableControlAgent,omitempty" yaml:"enableControlAgent,omitempty"`
+	EnableNodeControl            bool              `json:"enableNodeControl,omitempty" yaml:"enableNodeControl,omitempty"`
 	PodSubnet                     string            `json:"podSubnet,omitempty" yaml:"podSubnet,omitempty"`
 	ServiceSubnet                 string            `json:"serviceSubnet,omitempty" yaml:"serviceSubnet,omitempty"`
 	APIServerAdvertiseAddress     string            `json:"apiServerAdvertiseAddress,omitempty" yaml:"apiServerAdvertiseAddress,omitempty"`
@@ -40,7 +40,6 @@ type MonoKSConfigSpec struct {
 	KubeProxyNodePortAddresses    []string          `json:"kubeProxyNodePortAddresses,omitempty" yaml:"kubeProxyNodePortAddresses,omitempty"`
 	SubjectAltNames               []string          `json:"subjectAltNames,omitempty" yaml:"subjectAltNames,omitempty"`
 	NodeLabels                    map[string]string `json:"nodeLabels,omitempty" yaml:"nodeLabels,omitempty"`
-	NodeAnnotations               map[string]string `json:"nodeAnnotations,omitempty" yaml:"nodeAnnotations,omitempty"`
 	Network                       NetworkSpec       `json:"network,omitempty" yaml:"network,omitempty"`
 }
 

clitools/pkg/apis/monok8s/v1alpha1/osupgrade.go

@@ -2,17 +2,14 @@ package v1alpha1
 
 import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/runtime"
 )
 
 type OSUpgradePhase string
 
 const (
-	OSUpgradePhasePending    OSUpgradePhase = "Pending"
-	OSUpgradePhaseAccepted   OSUpgradePhase = "Accepted"
-	OSUpgradePhaseRollingOut OSUpgradePhase = "RollingOut"
-	OSUpgradePhaseCompleted  OSUpgradePhase = "Completed"
-	OSUpgradePhaseRejected   OSUpgradePhase = "Rejected"
+	OSUpgradePhasePending  OSUpgradePhase = "Pending"
+	OSUpgradePhaseAccepted OSUpgradePhase = "Accepted"
+	OSUpgradePhaseRejected OSUpgradePhase = "Rejected"
 )
 
 type OSUpgradeProgressPhase string
@@ -21,27 +18,35 @@ const (
 	OSUpgradeProgressPhasePending     OSUpgradeProgressPhase = "pending"
 	OSUpgradeProgressPhaseDownloading OSUpgradeProgressPhase = "downloading"
 	OSUpgradeProgressPhaseWriting     OSUpgradeProgressPhase = "writing"
-	OSUpgradeProgressPhaseRebooting   OSUpgradeProgressPhase = "rebooting"
 	OSUpgradeProgressPhaseVerifying   OSUpgradeProgressPhase = "verifying"
 	OSUpgradeProgressPhaseCompleted   OSUpgradeProgressPhase = "completed"
 	OSUpgradeProgressPhaseFailed      OSUpgradeProgressPhase = "failed"
 	OSUpgradeProgressPhaseRejected    OSUpgradeProgressPhase = "rejected"
+
+	// Rebooting is the point-of-no-return phase.
+	//
+	// Once a node reaches Rebooting, the agent may have already changed the boot
+	// environment and requested a reboot. The controller must not supersede,
+	// retry, retarget, or otherwise mutate this progress object until the node
+	// comes back and the agent reports Completed or Failed.
+	OSUpgradeProgressPhaseRebooting OSUpgradeProgressPhase = "rebooting"
 )
 
+// +genclient
 // +kubebuilder:object:root=true
 // +kubebuilder:subresource:status
 // +kubebuilder:resource:scope=Namespaced,shortName=osu
 // +kubebuilder:printcolumn:name="Desired",type=string,JSONPath=`.spec.desiredVersion`
 // +kubebuilder:printcolumn:name="Resolved",type=string,JSONPath=`.status.resolvedVersion`
 // +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
-// +kubebuilder:printcolumn:name="Targets",type=integer,JSONPath=`.status.summary.targetedNodes`
-// +kubebuilder:printcolumn:name="OK",type=integer,JSONPath=`.status.summary.succeededNodes`
-// +kubebuilder:printcolumn:name="Fail",type=integer,JSONPath=`.status.summary.failedNodes`
 type OSUpgrade struct {
 	metav1.TypeMeta   `json:",inline" yaml:",inline"`
 	metav1.ObjectMeta `json:"metadata,omitempty" yaml:"metadata,omitempty"`
 
-	Spec   OSUpgradeSpec    `json:"spec,omitempty" yaml:"spec,omitempty"`
+	// Specification of the desired behavior of the OSUpgrade.
+	Spec OSUpgradeSpec `json:"spec,omitempty" yaml:"spec,omitempty"`
+
+	// Most recently observed status of the OSUpgrade.
 	Status *OSUpgradeStatus `json:"status,omitempty" yaml:"status,omitempty"`
 }
 
@@ -53,40 +58,37 @@ type OSUpgradeList struct {
 }
 
 type OSUpgradeSpec struct {
-	// User request, can be "stable" or an explicit version like "v1.35.3".
+	// +kubebuilder:validation:MinLength=1
 	DesiredVersion string `json:"desiredVersion,omitempty" yaml:"desiredVersion,omitempty"`
 
-	Catalog *VersionCatalogSource `json:"catalog,omitempty"`
+	// +kubebuilder:validation:Enum=fast;balanced;safe
+	// +kubebuilder:default=balanced
+	// Profiles (TODO)
+	//    safe     - api-server can be responsive most of the time
+	//    balanced - api-server can sometimes be unresponsive
+	//    fast     - disable throttling. Good for worker node.
+	FlashProfile string `json:"flashProfile,omitempty" yaml:"flashProfile,omitempty"`
 
+	Catalog      *VersionCatalogSource `json:"catalog,omitempty" yaml:"catalog,omitempty"`
 	NodeSelector *metav1.LabelSelector `json:"nodeSelector,omitempty" yaml:"nodeSelector,omitempty"`
 }
 
 type VersionCatalogSource struct {
-	URL       string `json:"url,omitempty"`
-	Inline    string `json:"inline,omitempty"`
-	ConfigMap string `json:"configMapRef,omitempty"`
+	URL       string `json:"url,omitempty" yaml:"url,omitempty"`
+	Inline    string `json:"inline,omitempty" yaml:"inline,omitempty"`
+	ConfigMap string `json:"configMapRef,omitempty" yaml:"configMapRef,omitempty"`
 }
 
 type OSUpgradeStatus struct {
 	Phase              OSUpgradePhase     `json:"phase,omitempty" yaml:"phase,omitempty"`
 	ResolvedVersion    string             `json:"resolvedVersion,omitempty" yaml:"resolvedVersion,omitempty"`
 	ObservedGeneration int64              `json:"observedGeneration,omitempty" yaml:"observedGeneration,omitempty"`
-	Summary            OSUpgradeSummary   `json:"summary,omitempty" yaml:"summary,omitempty"`
 	Conditions         []metav1.Condition `json:"conditions,omitempty" yaml:"conditions,omitempty"`
-
-	// Optional, useful when rejected.
-	Reason  string `json:"reason,omitempty" yaml:"reason,omitempty"`
-	Message string `json:"message,omitempty" yaml:"message,omitempty"`
-}
-
-type OSUpgradeSummary struct {
-	TargetedNodes  int32 `json:"targetedNodes,omitempty" yaml:"targetedNodes,omitempty"`
-	PendingNodes   int32 `json:"pendingNodes,omitempty" yaml:"pendingNodes,omitempty"`
-	RunningNodes   int32 `json:"runningNodes,omitempty" yaml:"runningNodes,omitempty"`
-	SucceededNodes int32 `json:"succeededNodes,omitempty" yaml:"succeededNodes,omitempty"`
-	FailedNodes    int32 `json:"failedNodes,omitempty" yaml:"failedNodes,omitempty"`
+	Reason             string             `json:"reason,omitempty" yaml:"reason,omitempty"`
+	Message            string             `json:"message,omitempty" yaml:"message,omitempty"`
 }
 
+// +genclient
 // +kubebuilder:object:root=true
 // +kubebuilder:subresource:status
 // +kubebuilder:resource:scope=Namespaced,shortName=osup
@@ -94,12 +96,15 @@ type OSUpgradeSummary struct {
 // +kubebuilder:printcolumn:name="Source",type=string,JSONPath=`.spec.sourceRef.name`
 // +kubebuilder:printcolumn:name="Current",type=string,JSONPath=`.status.currentVersion`
 // +kubebuilder:printcolumn:name="Target",type=string,JSONPath=`.status.targetVersion`
-// +kubebuilder:printcolumn:name="Status",type=string,JSONPath=`.status.phase`
+// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
 type OSUpgradeProgress struct {
 	metav1.TypeMeta   `json:",inline" yaml:",inline"`
 	metav1.ObjectMeta `json:"metadata,omitempty" yaml:"metadata,omitempty"`
 
-	Spec   OSUpgradeProgressSpec    `json:"spec,omitempty" yaml:"spec,omitempty"`
+	// Specification of the desired behavior of the OSUpgradeProgress.
+	Spec OSUpgradeProgressSpec `json:"spec,omitempty" yaml:"spec,omitempty"`
+
+	// Most recently observed status of the OSUpgradeProgress.
 	Status *OSUpgradeProgressStatus `json:"status,omitempty" yaml:"status,omitempty"`
 }
 
@@ -112,60 +117,55 @@ type OSUpgradeProgressList struct {
 
 type OSUpgradeProgressSpec struct {
 	SourceRef OSUpgradeSourceRef `json:"sourceRef,omitempty" yaml:"sourceRef,omitempty"`
-	NodeName  string             `json:"nodeName,omitempty" yaml:"nodeName,omitempty"`
+
+	// RetryNonce triggers a retry when its value changes.
+	// Users can update this field (for example, set it to the current time)
+	// to request a retry of a failed OS upgrade.
+	RetryNonce string `json:"retryNonce,omitempty" yaml:"retryNonce,omitempty"`
+
+	NodeName string `json:"nodeName,omitempty" yaml:"nodeName,omitempty"`
 }
 
 type OSUpgradeSourceRef struct {
-	Name string `json:"name,omitempty" yaml:"name,omitempty"`
+	Name      string `json:"name,omitempty" yaml:"name,omitempty"`
+	Namespace string `json:"namespace,omitempty" yaml:"namespace,omitempty"`
 }
 
 type OSUpgradeProgressStatus struct {
-	CurrentVersion string                 `json:"currentVersion,omitempty" yaml:"currentVersion,omitempty"`
-	TargetVersion  string                 `json:"targetVersion,omitempty" yaml:"targetVersion,omitempty"`
-	Phase          OSUpgradeProgressPhase `json:"phase,omitempty" yaml:"phase,omitempty"`
+	CurrentVersion    string                 `json:"currentVersion,omitempty" yaml:"currentVersion,omitempty"`
+	TargetVersion     string                 `json:"targetVersion,omitempty" yaml:"targetVersion,omitempty"`
+	Phase             OSUpgradeProgressPhase `json:"phase,omitempty" yaml:"phase,omitempty"`
+	StartedAt         *metav1.Time           `json:"startedAt,omitempty" yaml:"startedAt,omitempty"`
+	CompletedAt       *metav1.Time           `json:"completedAt,omitempty" yaml:"completedAt,omitempty"`
+	LastUpdatedAt     *metav1.Time           `json:"lastUpdatedAt,omitempty" yaml:"lastUpdatedAt,omitempty"`
+	RetryCount        int32                  `json:"retryCount,omitempty" yaml:"retryCount,omitempty"`
+	InactivePartition string                 `json:"inactivePartition,omitempty" yaml:"inactivePartition,omitempty"`
+	FailureReason     string                 `json:"failureReason,omitempty" yaml:"failureReason,omitempty"`
+	Message           string                 `json:"message,omitempty" yaml:"message,omitempty"`
+	PlannedPath       []string               `json:"plannedPath,omitempty" yaml:"plannedPath,omitempty"`
+	CurrentStep       int32                  `json:"currentStep,omitempty" yaml:"currentStep,omitempty"`
+	CurrentFrom       string                 `json:"currentFrom,omitempty" yaml:"currentFrom,omitempty"`
+	CurrentTo         string                 `json:"currentTo,omitempty" yaml:"currentTo,omitempty"`
 
-	StartedAt         *metav1.Time `json:"startedAt,omitempty" yaml:"startedAt,omitempty"`
-	CompletedAt       *metav1.Time `json:"completedAt,omitempty" yaml:"completedAt,omitempty"`
-	LastUpdatedAt     *metav1.Time `json:"lastUpdatedAt,omitempty" yaml:"lastUpdatedAt,omitempty"`
-	RetryCount        int32        `json:"retryCount,omitempty" yaml:"retryCount,omitempty"`
-	InactivePartition string       `json:"inactivePartition,omitempty" yaml:"inactivePartition,omitempty"`
-	FailureReason     string       `json:"failureReason,omitempty" yaml:"failureReason,omitempty"`
-	Message           string       `json:"message,omitempty" yaml:"message,omitempty"`
-
-	PlannedPath []string `json:"plannedPath,omitempty"`
-	CurrentStep int32    `json:"currentStep,omitempty"`
-	CurrentFrom string   `json:"currentFrom,omitempty"`
-	CurrentTo   string   `json:"currentTo,omitempty"`
+	// ObservedRetryNonce records the last retryNonce value the agent accepted.
+	// When spec.retryNonce is changed by the user and differs from this value,
+	// the agent may retry a failed upgrade.
+	// +optional
+	ObservedRetryNonce string `json:"observedRetryNonce,omitempty"`
 }
 
-func (in *OSUpgrade) DeepCopyObject() runtime.Object {
-	if in == nil {
-		return nil
+func (osu OSUpgrade) StatusPhase() string {
+	phase := ""
+	if osu.Status != nil {
+		phase = string(osu.Status.Phase)
 	}
-	out := *in
-	return &out
+	return phase
 }
 
-func (in *OSUpgradeList) DeepCopyObject() runtime.Object {
-	if in == nil {
-		return nil
+func (osup OSUpgradeProgress) StatusPhase() string {
+	phase := ""
+	if osup.Status != nil {
+		phase = string(osup.Status.Phase)
 	}
-	out := *in
-	return &out
-}
-
-func (in *OSUpgradeProgress) DeepCopyObject() runtime.Object {
-	if in == nil {
-		return nil
-	}
-	out := *in
-	return &out
-}
-
-func (in *OSUpgradeProgressList) DeepCopyObject() runtime.Object {
-	if in == nil {
-		return nil
-	}
-	out := *in
-	return &out
+	return phase
 }

clitools/pkg/apis/monok8s/v1alpha1/zz_generated.deepcopy.go

@@ -0,0 +1,395 @@
+//go:build !ignore_autogenerated
+
+/* MIT License */
+
+// Code generated by controller-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	"k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+)
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *MonoKSConfig) DeepCopyInto(out *MonoKSConfig) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	in.Spec.DeepCopyInto(&out.Spec)
+	if in.Status != nil {
+		in, out := &in.Status, &out.Status
+		*out = new(MonoKSConfigStatus)
+		(*in).DeepCopyInto(*out)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MonoKSConfig.
+func (in *MonoKSConfig) DeepCopy() *MonoKSConfig {
+	if in == nil {
+		return nil
+	}
+	out := new(MonoKSConfig)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *MonoKSConfigList) DeepCopyInto(out *MonoKSConfigList) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ListMeta.DeepCopyInto(&out.ListMeta)
+	if in.Items != nil {
+		in, out := &in.Items, &out.Items
+		*out = make([]MonoKSConfig, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MonoKSConfigList.
+func (in *MonoKSConfigList) DeepCopy() *MonoKSConfigList {
+	if in == nil {
+		return nil
+	}
+	out := new(MonoKSConfigList)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *MonoKSConfigSpec) DeepCopyInto(out *MonoKSConfigSpec) {
+	*out = *in
+	if in.KubeProxyNodePortAddresses != nil {
+		in, out := &in.KubeProxyNodePortAddresses, &out.KubeProxyNodePortAddresses
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+	if in.SubjectAltNames != nil {
+		in, out := &in.SubjectAltNames, &out.SubjectAltNames
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+	if in.NodeLabels != nil {
+		in, out := &in.NodeLabels, &out.NodeLabels
+		*out = make(map[string]string, len(*in))
+		for key, val := range *in {
+			(*out)[key] = val
+		}
+	}
+	in.Network.DeepCopyInto(&out.Network)
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MonoKSConfigSpec.
+func (in *MonoKSConfigSpec) DeepCopy() *MonoKSConfigSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(MonoKSConfigSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *MonoKSConfigStatus) DeepCopyInto(out *MonoKSConfigStatus) {
+	*out = *in
+	if in.Conditions != nil {
+		in, out := &in.Conditions, &out.Conditions
+		*out = make([]v1.Condition, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+	if in.AppliedSteps != nil {
+		in, out := &in.AppliedSteps, &out.AppliedSteps
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new MonoKSConfigStatus.
+func (in *MonoKSConfigStatus) DeepCopy() *MonoKSConfigStatus {
+	if in == nil {
+		return nil
+	}
+	out := new(MonoKSConfigStatus)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *NetworkSpec) DeepCopyInto(out *NetworkSpec) {
+	*out = *in
+	if in.DNSNameservers != nil {
+		in, out := &in.DNSNameservers, &out.DNSNameservers
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+	if in.DNSSearchDomains != nil {
+		in, out := &in.DNSSearchDomains, &out.DNSSearchDomains
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new NetworkSpec.
+func (in *NetworkSpec) DeepCopy() *NetworkSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(NetworkSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgrade) DeepCopyInto(out *OSUpgrade) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	in.Spec.DeepCopyInto(&out.Spec)
+	if in.Status != nil {
+		in, out := &in.Status, &out.Status
+		*out = new(OSUpgradeStatus)
+		(*in).DeepCopyInto(*out)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgrade.
+func (in *OSUpgrade) DeepCopy() *OSUpgrade {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgrade)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *OSUpgrade) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeList) DeepCopyInto(out *OSUpgradeList) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ListMeta.DeepCopyInto(&out.ListMeta)
+	if in.Items != nil {
+		in, out := &in.Items, &out.Items
+		*out = make([]OSUpgrade, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeList.
+func (in *OSUpgradeList) DeepCopy() *OSUpgradeList {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeList)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *OSUpgradeList) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeProgress) DeepCopyInto(out *OSUpgradeProgress) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	out.Spec = in.Spec
+	if in.Status != nil {
+		in, out := &in.Status, &out.Status
+		*out = new(OSUpgradeProgressStatus)
+		(*in).DeepCopyInto(*out)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeProgress.
+func (in *OSUpgradeProgress) DeepCopy() *OSUpgradeProgress {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeProgress)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *OSUpgradeProgress) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeProgressList) DeepCopyInto(out *OSUpgradeProgressList) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ListMeta.DeepCopyInto(&out.ListMeta)
+	if in.Items != nil {
+		in, out := &in.Items, &out.Items
+		*out = make([]OSUpgradeProgress, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeProgressList.
+func (in *OSUpgradeProgressList) DeepCopy() *OSUpgradeProgressList {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeProgressList)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *OSUpgradeProgressList) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeProgressSpec) DeepCopyInto(out *OSUpgradeProgressSpec) {
+	*out = *in
+	out.SourceRef = in.SourceRef
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeProgressSpec.
+func (in *OSUpgradeProgressSpec) DeepCopy() *OSUpgradeProgressSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeProgressSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeProgressStatus) DeepCopyInto(out *OSUpgradeProgressStatus) {
+	*out = *in
+	if in.StartedAt != nil {
+		in, out := &in.StartedAt, &out.StartedAt
+		*out = (*in).DeepCopy()
+	}
+	if in.CompletedAt != nil {
+		in, out := &in.CompletedAt, &out.CompletedAt
+		*out = (*in).DeepCopy()
+	}
+	if in.LastUpdatedAt != nil {
+		in, out := &in.LastUpdatedAt, &out.LastUpdatedAt
+		*out = (*in).DeepCopy()
+	}
+	if in.PlannedPath != nil {
+		in, out := &in.PlannedPath, &out.PlannedPath
+		*out = make([]string, len(*in))
+		copy(*out, *in)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeProgressStatus.
+func (in *OSUpgradeProgressStatus) DeepCopy() *OSUpgradeProgressStatus {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeProgressStatus)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeSourceRef) DeepCopyInto(out *OSUpgradeSourceRef) {
+	*out = *in
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeSourceRef.
+func (in *OSUpgradeSourceRef) DeepCopy() *OSUpgradeSourceRef {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeSourceRef)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeSpec) DeepCopyInto(out *OSUpgradeSpec) {
+	*out = *in
+	if in.Catalog != nil {
+		in, out := &in.Catalog, &out.Catalog
+		*out = new(VersionCatalogSource)
+		**out = **in
+	}
+	if in.NodeSelector != nil {
+		in, out := &in.NodeSelector, &out.NodeSelector
+		*out = new(v1.LabelSelector)
+		(*in).DeepCopyInto(*out)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeSpec.
+func (in *OSUpgradeSpec) DeepCopy() *OSUpgradeSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *OSUpgradeStatus) DeepCopyInto(out *OSUpgradeStatus) {
+	*out = *in
+	if in.Conditions != nil {
+		in, out := &in.Conditions, &out.Conditions
+		*out = make([]v1.Condition, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new OSUpgradeStatus.
+func (in *OSUpgradeStatus) DeepCopy() *OSUpgradeStatus {
+	if in == nil {
+		return nil
+	}
+	out := new(OSUpgradeStatus)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *VersionCatalogSource) DeepCopyInto(out *VersionCatalogSource) {
+	*out = *in
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new VersionCatalogSource.
+func (in *VersionCatalogSource) DeepCopy() *VersionCatalogSource {
+	if in == nil {
+		return nil
+	}
+	out := new(VersionCatalogSource)
+	in.DeepCopyInto(out)
+	return out
+}

clitools/pkg/assets/crds/monok8s.io_monoksconfigs.yaml

@@ -0,0 +1,178 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.20.1
+  name: monoksconfigs.monok8s.io
+spec:
+  group: monok8s.io
+  names:
+    kind: MonoKSConfig
+    listKind: MonoKSConfigList
+    plural: monoksconfigs
+    singular: monoksconfig
+  scope: Namespaced
+  versions:
+  - name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            properties:
+              allowSchedulingOnControlPlane:
+                type: boolean
+              apiServerAdvertiseAddress:
+                type: string
+              apiServerEndpoint:
+                type: string
+              bootstrapToken:
+                type: string
+              clusterDomain:
+                type: string
+              clusterName:
+                type: string
+              clusterRole:
+                type: string
+              cniPlugin:
+                type: string
+              containerRuntimeEndpoint:
+                type: string
+              controlPlaneCertKey:
+                type: string
+              discoveryTokenCACertHash:
+                type: string
+              enableNodeControl:
+                type: boolean
+              initControlPlane:
+                type: boolean
+              kubeProxyNodePortAddresses:
+                items:
+                  type: string
+                type: array
+              kubernetesVersion:
+                type: string
+              network:
+                properties:
+                  dnsNameservers:
+                    items:
+                      type: string
+                    type: array
+                  dnsSearchDomains:
+                    items:
+                      type: string
+                    type: array
+                  hostname:
+                    type: string
+                  managementCIDR:
+                    type: string
+                  managementGateway:
+                    type: string
+                  managementIface:
+                    type: string
+                type: object
+              nodeLabels:
+                additionalProperties:
+                  type: string
+                type: object
+              nodeName:
+                type: string
+              podSubnet:
+                type: string
+              serviceSubnet:
+                type: string
+              skipImageCheck:
+                type: boolean
+              subjectAltNames:
+                items:
+                  type: string
+                type: array
+            type: object
+          status:
+            properties:
+              appliedSteps:
+                items:
+                  type: string
+                type: array
+              conditions:
+                items:
+                  description: Condition contains details for one aspect of the current
+                    state of this API Resource.
+                  properties:
+                    lastTransitionTime:
+                      description: |-
+                        lastTransitionTime is the last time the condition transitioned from one status to another.
+                        This should be when the underlying condition changed.  If that is not known, then using the time when the API field changed is acceptable.
+                      format: date-time
+                      type: string
+                    message:
+                      description: |-
+                        message is a human readable message indicating details about the transition.
+                        This may be an empty string.
+                      maxLength: 32768
+                      type: string
+                    observedGeneration:
+                      description: |-
+                        observedGeneration represents the .metadata.generation that the condition was set based upon.
+                        For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
+                        with respect to the current state of the instance.
+                      format: int64
+                      minimum: 0
+                      type: integer
+                    reason:
+                      description: |-
+                        reason contains a programmatic identifier indicating the reason for the condition's last transition.
+                        Producers of specific condition types may define expected values and meanings for this field,
+                        and whether the values are considered a guaranteed API.
+                        The value should be a CamelCase string.
+                        This field may not be empty.
+                      maxLength: 1024
+                      minLength: 1
+                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
+                      type: string
+                    status:
+                      description: status of the condition, one of True, False, Unknown.
+                      enum:
+                      - "True"
+                      - "False"
+                      - Unknown
+                      type: string
+                    type:
+                      description: type of condition in CamelCase or in foo.example.com/CamelCase.
+                      maxLength: 316
+                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
+                      type: string
+                  required:
+                  - lastTransitionTime
+                  - message
+                  - reason
+                  - status
+                  - type
+                  type: object
+                type: array
+              observedGeneration:
+                format: int64
+                type: integer
+              phase:
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true

clitools/pkg/assets/crds/monok8s.io_osupgradeprogresses.yaml

@@ -0,0 +1,124 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.20.1
+  name: osupgradeprogresses.monok8s.io
+spec:
+  group: monok8s.io
+  names:
+    kind: OSUpgradeProgress
+    listKind: OSUpgradeProgressList
+    plural: osupgradeprogresses
+    shortNames:
+    - osup
+    singular: osupgradeprogress
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.nodeName
+      name: Node
+      type: string
+    - jsonPath: .spec.sourceRef.name
+      name: Source
+      type: string
+    - jsonPath: .status.currentVersion
+      name: Current
+      type: string
+    - jsonPath: .status.targetVersion
+      name: Target
+      type: string
+    - jsonPath: .status.phase
+      name: Phase
+      type: string
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: Specification of the desired behavior of the OSUpgradeProgress.
+            properties:
+              nodeName:
+                type: string
+              retryNonce:
+                description: |-
+                  RetryNonce triggers a retry when its value changes.
+                  Users can update this field (for example, set it to the current time)
+                  to request a retry of a failed OS upgrade.
+                type: string
+              sourceRef:
+                properties:
+                  name:
+                    type: string
+                  namespace:
+                    type: string
+                type: object
+            type: object
+          status:
+            description: Most recently observed status of the OSUpgradeProgress.
+            properties:
+              completedAt:
+                format: date-time
+                type: string
+              currentFrom:
+                type: string
+              currentStep:
+                format: int32
+                type: integer
+              currentTo:
+                type: string
+              currentVersion:
+                type: string
+              failureReason:
+                type: string
+              inactivePartition:
+                type: string
+              lastUpdatedAt:
+                format: date-time
+                type: string
+              message:
+                type: string
+              observedRetryNonce:
+                description: |-
+                  ObservedRetryNonce records the last retryNonce value the agent accepted.
+                  When spec.retryNonce is changed by the user and differs from this value,
+                  the agent may retry a failed upgrade.
+                type: string
+              phase:
+                type: string
+              plannedPath:
+                items:
+                  type: string
+                type: array
+              retryCount:
+                format: int32
+                type: integer
+              startedAt:
+                format: date-time
+                type: string
+              targetVersion:
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}

clitools/pkg/assets/crds/monok8s.io_osupgrades.yaml

@@ -0,0 +1,202 @@
+---
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    controller-gen.kubebuilder.io/version: v0.20.1
+  name: osupgrades.monok8s.io
+spec:
+  group: monok8s.io
+  names:
+    kind: OSUpgrade
+    listKind: OSUpgradeList
+    plural: osupgrades
+    shortNames:
+    - osu
+    singular: osupgrade
+  scope: Namespaced
+  versions:
+  - additionalPrinterColumns:
+    - jsonPath: .spec.desiredVersion
+      name: Desired
+      type: string
+    - jsonPath: .status.resolvedVersion
+      name: Resolved
+      type: string
+    - jsonPath: .status.phase
+      name: Phase
+      type: string
+    name: v1alpha1
+    schema:
+      openAPIV3Schema:
+        properties:
+          apiVersion:
+            description: |-
+              APIVersion defines the versioned schema of this representation of an object.
+              Servers should convert recognized schemas to the latest internal value, and
+              may reject unrecognized values.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
+            type: string
+          kind:
+            description: |-
+              Kind is a string value representing the REST resource this object represents.
+              Servers may infer this from the endpoint the client submits requests to.
+              Cannot be updated.
+              In CamelCase.
+              More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: Specification of the desired behavior of the OSUpgrade.
+            properties:
+              catalog:
+                properties:
+                  configMapRef:
+                    type: string
+                  inline:
+                    type: string
+                  url:
+                    type: string
+                type: object
+              desiredVersion:
+                minLength: 1
+                type: string
+              flashProfile:
+                default: balanced
+                description: |-
+                  Profiles (TODO)
+                     safe     - api-server can be responsive most of the time
+                     balanced - api-server can sometimes be unresponsive
+                     fast     - disable throttling. Good for worker node.
+                enum:
+                - fast
+                - balanced
+                - safe
+                type: string
+              nodeSelector:
+                description: |-
+                  A label selector is a label query over a set of resources. The result of matchLabels and
+                  matchExpressions are ANDed. An empty label selector matches all objects. A null
+                  label selector matches no objects.
+                properties:
+                  matchExpressions:
+                    description: matchExpressions is a list of label selector requirements.
+                      The requirements are ANDed.
+                    items:
+                      description: |-
+                        A label selector requirement is a selector that contains values, a key, and an operator that
+                        relates the key and values.
+                      properties:
+                        key:
+                          description: key is the label key that the selector applies
+                            to.
+                          type: string
+                        operator:
+                          description: |-
+                            operator represents a key's relationship to a set of values.
+                            Valid operators are In, NotIn, Exists and DoesNotExist.
+                          type: string
+                        values:
+                          description: |-
+                            values is an array of string values. If the operator is In or NotIn,
+                            the values array must be non-empty. If the operator is Exists or DoesNotExist,
+                            the values array must be empty. This array is replaced during a strategic
+                            merge patch.
+                          items:
+                            type: string
+                          type: array
+                          x-kubernetes-list-type: atomic
+                      required:
+                      - key
+                      - operator
+                      type: object
+                    type: array
+                    x-kubernetes-list-type: atomic
+                  matchLabels:
+                    additionalProperties:
+                      type: string
+                    description: |-
+                      matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels
+                      map is equivalent to an element of matchExpressions, whose key field is "key", the
+                      operator is "In", and the values array contains only "value". The requirements are ANDed.
+                    type: object
+                type: object
+                x-kubernetes-map-type: atomic
+            type: object
+          status:
+            description: Most recently observed status of the OSUpgrade.
+            properties:
+              conditions:
+                items:
+                  description: Condition contains details for one aspect of the current
+                    state of this API Resource.
+                  properties:
+                    lastTransitionTime:
+                      description: |-
+                        lastTransitionTime is the last time the condition transitioned from one status to another.
+                        This should be when the underlying condition changed.  If that is not known, then using the time when the API field changed is acceptable.
+                      format: date-time
+                      type: string
+                    message:
+                      description: |-
+                        message is a human readable message indicating details about the transition.
+                        This may be an empty string.
+                      maxLength: 32768
+                      type: string
+                    observedGeneration:
+                      description: |-
+                        observedGeneration represents the .metadata.generation that the condition was set based upon.
+                        For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
+                        with respect to the current state of the instance.
+                      format: int64
+                      minimum: 0
+                      type: integer
+                    reason:
+                      description: |-
+                        reason contains a programmatic identifier indicating the reason for the condition's last transition.
+                        Producers of specific condition types may define expected values and meanings for this field,
+                        and whether the values are considered a guaranteed API.
+                        The value should be a CamelCase string.
+                        This field may not be empty.
+                      maxLength: 1024
+                      minLength: 1
+                      pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
+                      type: string
+                    status:
+                      description: status of the condition, one of True, False, Unknown.
+                      enum:
+                      - "True"
+                      - "False"
+                      - Unknown
+                      type: string
+                    type:
+                      description: type of condition in CamelCase or in foo.example.com/CamelCase.
+                      maxLength: 316
+                      pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
+                      type: string
+                  required:
+                  - lastTransitionTime
+                  - message
+                  - reason
+                  - status
+                  - type
+                  type: object
+                type: array
+              message:
+                type: string
+              observedGeneration:
+                format: int64
+                type: integer
+              phase:
+                type: string
+              reason:
+                type: string
+              resolvedVersion:
+                type: string
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}

clitools/pkg/assets/embed.go

@@ -0,0 +1,6 @@
+package assets
+
+import "embed"
+
+//go:embed crds/*.yaml
+var CRDs embed.FS

clitools/pkg/assets/render.go

@@ -0,0 +1,49 @@
+package assets
+
+import (
+	"fmt"
+	"io"
+	"path/filepath"
+	"sort"
+)
+
+func PrintCRDs(out io.Writer) error {
+	entries, err := CRDs.ReadDir("crds")
+	if err != nil {
+		return err
+	}
+
+	names := make([]string, 0, len(entries))
+	for _, entry := range entries {
+		if entry.IsDir() {
+			continue
+		}
+		if filepath.Ext(entry.Name()) != ".yaml" {
+			continue
+		}
+		names = append(names, entry.Name())
+	}
+
+	sort.Strings(names)
+
+	for _, name := range names {
+		b, err := CRDs.ReadFile("crds/" + name)
+		if err != nil {
+			return err
+		}
+
+		if _, err := fmt.Fprintln(out, "---"); err != nil {
+			return err
+		}
+		if _, err := out.Write(b); err != nil {
+			return err
+		}
+		if len(b) == 0 || b[len(b)-1] != '\n' {
+			if _, err := fmt.Fprintln(out); err != nil {
+				return err
+			}
+		}
+	}
+
+	return nil
+}

clitools/pkg/bootstrap/registry.go

@@ -23,7 +23,7 @@ func NewRegistry(ctx *node.NodeContext) *Registry {
 
 	return &Registry{
 		steps: map[string]node.Step{
-			"ApplyControlAgentDaemonSetResources":    node.ApplyControlAgentDaemonSetResources,
+			"ApplyNodeControlDaemonSetResources":    node.ApplyNodeControlDaemonSetResources,
 			"ApplyLocalNodeMetadataIfPossible":       node.ApplyLocalNodeMetadataIfPossible,
 			"CheckForVersionSkew":                    node.CheckForVersionSkew,
 			"ClassifyBootstrapAction":                node.ClassifyBootstrapAction,

clitools/pkg/bootstrap/runner.go

@@ -70,7 +70,7 @@ func NewRunner(cfg *monov1alpha1.MonoKSConfig) *Runner {
 			{
 				RegKey: "EngageControlGate",
 				Name:   "Engage the control gate",
-				Desc:   "Prevents agent polling resources prematurely",
+				Desc:   "Prevents agent watching resources prematurely",
 			},
 			{
 				RegKey: "StartCRIO",
@@ -112,6 +112,11 @@ func NewRunner(cfg *monov1alpha1.MonoKSConfig) *Runner {
 				Name:   "Wait for existing cluster",
 				Desc:   "Block until control plane is reachable when joining or reconciling an existing cluster",
 			},
+			{
+				RegKey: "CheckForVersionSkew",
+				Name:   "Check for version skew",
+				Desc:   "Validate wether version satisfy the requirements againts current cluster if any",
+			},
 			{
 				RegKey: "ReconcileControlPlane",
 				Name:   "Reconcile control plane",
@@ -122,11 +127,6 @@ func NewRunner(cfg *monov1alpha1.MonoKSConfig) *Runner {
 				Name:   "Reconcile worker node",
 				Desc:   "Reconcile the worker node",
 			},
-			{
-				RegKey: "CheckForVersionSkew",
-				Name:   "Check for version skew",
-				Desc:   "Validate wether version satisfy the requirements againts current cluster if any",
-			},
 			{
 				RegKey: "RunKubeadmUpgradeApply",
 				Name:   "Run kubeadm upgrade apply",
@@ -158,14 +158,14 @@ func NewRunner(cfg *monov1alpha1.MonoKSConfig) *Runner {
 				Desc:   "Make A/B booting possible",
 			},
 			{
-				RegKey: "ApplyControlAgentDaemonSetResources",
+				RegKey: "ApplyNodeControlDaemonSetResources",
 				Name:   "Apply daemonset for control agent",
 				Desc:   "Control agent handles OSUpgrade resources",
 			},
 			{
 				RegKey: "ReleaseControlGate",
 				Name:   "Release the control gate",
-				Desc:   "Allow agent to start polling resources",
+				Desc:   "Allow agent to start watching resources",
 			},
 		},
 	}

clitools/pkg/cmd/agent/agent.go

@@ -9,10 +9,7 @@ import (
 
 	"github.com/spf13/cobra"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
-	"k8s.io/apimachinery/pkg/labels"
-	"k8s.io/apimachinery/pkg/runtime"
-	"k8s.io/apimachinery/pkg/runtime/schema"
+	"k8s.io/apimachinery/pkg/watch"
 	"k8s.io/cli-runtime/pkg/genericclioptions"
 	"k8s.io/klog/v2"
 
@@ -23,19 +20,18 @@ import (
 	"example.com/monok8s/pkg/templates"
 )
 
-const defaultPollInterval = 15 * time.Second
-
-var runtimeDefaultUnstructuredConverter = runtime.DefaultUnstructuredConverter
-
 func NewCmdAgent(flags *genericclioptions.ConfigFlags) *cobra.Command {
-	var namespace string
 	var envFile string
-	var pollInterval time.Duration
 
 	cmd := &cobra.Command{
 		Use:   "agent --env-file path",
-		Short: "Watch OSUpgrade resources and process matching upgrades for this node",
+		Short: "Watch OSUpgradeProgress resources for this node and process upgrades",
 		RunE: func(cmd *cobra.Command, _ []string) error {
+			ns, _, err := flags.ToRawKubeConfigLoader().Namespace()
+			if err != nil {
+				return err
+			}
+
 			if envFile == "" {
 				return fmt.Errorf("--env-file is required")
 			}
@@ -59,9 +55,8 @@ func NewCmdAgent(flags *genericclioptions.ConfigFlags) *cobra.Command {
 
 			klog.InfoS("starting agent",
 				"node", cfg.Spec.NodeName,
-				"namespace", namespace,
+				"namespace", ns,
 				"envFile", envFile,
-				"pollInterval", pollInterval,
 			)
 
 			clients, err := kube.NewClients(flags)
@@ -69,13 +64,11 @@ func NewCmdAgent(flags *genericclioptions.ConfigFlags) *cobra.Command {
 				return fmt.Errorf("create kube clients: %w", err)
 			}
 
-			return runPollLoop(ctx, clients, namespace, cfg.Spec.NodeName, pollInterval)
+			return runWatchLoop(ctx, clients, ns, cfg.Spec.NodeName)
 		},
 	}
 
-	cmd.Flags().StringVar(&namespace, "namespace", "kube-system", "namespace to watch")
 	cmd.Flags().StringVar(&envFile, "env-file", "", "path to env file containing MKS_* variables")
-	cmd.Flags().DurationVar(&pollInterval, "poll-interval", defaultPollInterval, "poll interval for OSUpgrade resources")
 
 	return cmd
 }
@@ -94,9 +87,9 @@ func waitForControlGate(ctx context.Context, envFile string, pollInterval time.D
 	for {
 		_, err := os.Stat(marker)
 		if err == nil {
-			klog.InfoS("Control gate is present; waiting before starting poll loop", "path", marker)
+			klog.InfoS("Control gate is present; waiting before starting watch loop", "path", marker)
 		} else if os.IsNotExist(err) {
-			klog.InfoS("Control gate not present; starting poll loop", "path", marker)
+			klog.InfoS("Control gate not present; starting watch loop", "path", marker)
 			return nil
 		} else {
 			return fmt.Errorf("stat upgrade marker %s: %w", marker, err)
@@ -110,128 +103,150 @@ func waitForControlGate(ctx context.Context, envFile string, pollInterval time.D
 	}
 }
 
-func runPollLoop(ctx context.Context, clients *kube.Clients, namespace, nodeName string, interval time.Duration) error {
-	gvr := schema.GroupVersionResource{
-		Group:    monov1alpha1.Group,
-		Version:  monov1alpha1.Version,
-		Resource: "osupgrades",
-	}
-
-	ticker := time.NewTicker(interval)
-	defer ticker.Stop()
+func runWatchLoop(ctx context.Context, clients *kube.Clients, namespace, nodeName string) error {
+	var resourceVersion string
 
 	for {
-		if err := pollOnce(ctx, clients, gvr, namespace, nodeName); err != nil {
-			klog.ErrorS(err, "poll failed", "namespace", namespace, "node", nodeName)
+		if ctx.Err() != nil {
+			return ctx.Err()
 		}
 
-		select {
-		case <-ctx.Done():
-			return ctx.Err()
-		case <-ticker.C:
+		err := watchOnce(ctx, clients, namespace, nodeName, &resourceVersion)
+		if err != nil {
+			if ctx.Err() != nil {
+				return ctx.Err()
+			}
+			klog.ErrorS(err, "watch failed; retrying",
+				"namespace", namespace,
+				"node", nodeName,
+				"resourceVersion", resourceVersion,
+			)
+			select {
+			case <-ctx.Done():
+				return ctx.Err()
+			case <-time.After(2 * time.Second):
+			}
+			continue
 		}
 	}
 }
 
-func pollOnce(
+func watchOnce(
 	ctx context.Context,
 	clients *kube.Clients,
-	gvr schema.GroupVersionResource,
 	namespace string,
 	nodeName string,
+	resourceVersion *string,
 ) error {
-	list, err := clients.Dynamic.Resource(gvr).Namespace(namespace).List(ctx, metav1.ListOptions{})
+
+	list, err := clients.MonoKS.
+		Monok8sV1alpha1().
+		OSUpgradeProgresses(namespace).
+		List(ctx, metav1.ListOptions{})
 	if err != nil {
-		return fmt.Errorf("list osupgrades: %w", err)
-	}
-
-	klog.InfoS("agent tick", "namespace", namespace, "items", len(list.Items), "node", nodeName)
-
-	nodeLabels := labels.Set{
-		"kubernetes.io/hostname":   nodeName,
-		"monok8s.io/node-name":     nodeName,
-		"monok8s.io/control-agent": "true",
+		return fmt.Errorf("list osupgradeprogresses: %w", err)
 	}
 
 	for i := range list.Items {
 		item := &list.Items[i]
-
-		osu, err := decodeOSUpgrade(item)
-		if err != nil {
-			klog.ErrorS(err, "failed to decode osupgrade",
-				"name", item.GetName(),
-				"resourceVersion", item.GetResourceVersion(),
-			)
+		if !targetsNode(item, nodeName) {
 			continue
 		}
 
-		if !matchesNode(osu, nodeName, nodeLabels) {
-			klog.V(2).InfoS("skipping osupgrade; not targeted to this node",
-				"name", osu.Name,
-				"node", nodeName,
-			)
-			continue
-		}
-
-		klog.InfoS("matched osupgrade",
-			"name", osu.Name,
+		klog.InfoS("found existing osupgradeprogress",
+			"name", item.Name,
 			"node", nodeName,
-			"desiredVersion", osu.Spec.DesiredVersion,
-			"phase", statusPhase(osu.Status),
-			"resourceVersion", osu.ResourceVersion,
+			"phase", item.StatusPhase(),
+			"resourceVersion", item.ResourceVersion,
 		)
 
-		if err := osupgradeController.HandleOSUpgrade(ctx, clients, namespace, nodeName, osu); err != nil {
-			klog.ErrorS(err, "failed to handle osupgrade",
-				"name", osu.Name,
+		if err := osupgradeController.HandleOSUpgradeProgress(ctx, clients, namespace, nodeName, item); err != nil {
+			klog.ErrorS(err, "failed to handle existing osupgradeprogress",
+				"name", item.Name,
 				"node", nodeName,
 			)
-			continue
 		}
 	}
 
-	return nil
-}
+	*resourceVersion = list.ResourceVersion
 
-func decodeOSUpgrade(item *unstructured.Unstructured) (*monov1alpha1.OSUpgrade, error) {
-	var osu monov1alpha1.OSUpgrade
-	if err := runtimeDefaultUnstructuredConverter.FromUnstructured(item.Object, &osu); err != nil {
-		return nil, fmt.Errorf("convert unstructured to OSUpgrade: %w", err)
-	}
-	return &osu, nil
-}
-
-func matchesNode(osu *monov1alpha1.OSUpgrade, nodeName string, nodeLabels labels.Set) bool {
-	if osu == nil {
-		return false
-	}
-
-	sel := osu.Spec.NodeSelector
-	if sel == nil {
-		// No selector means "match all nodes".
-		return true
-	}
-
-	selector, err := metav1.LabelSelectorAsSelector(sel)
+	w, err := clients.MonoKS.
+		Monok8sV1alpha1().
+		OSUpgradeProgresses(namespace).
+		Watch(ctx, metav1.ListOptions{
+			ResourceVersion: *resourceVersion,
+		})
 	if err != nil {
-		klog.ErrorS(err, "invalid node selector on osupgrade", "name", osu.Name)
+		return fmt.Errorf("watch osupgradeprogresses: %w", err)
+	}
+	defer w.Stop()
+
+	klog.InfoS("watching osupgradeprogresses",
+		"namespace", namespace,
+		"node", nodeName,
+		"resourceVersion", *resourceVersion,
+	)
+
+	for {
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+
+		case evt, ok := <-w.ResultChan():
+			if !ok {
+				return fmt.Errorf("watch channel closed")
+			}
+
+			switch evt.Type {
+			case watch.Bookmark:
+				obj, ok := evt.Object.(*monov1alpha1.OSUpgradeProgress)
+				if ok && obj != nil && obj.ResourceVersion != "" {
+					*resourceVersion = obj.ResourceVersion
+				}
+				continue
+
+			case watch.Error:
+				return fmt.Errorf("watch returned error event")
+			}
+
+			osup, ok := evt.Object.(*monov1alpha1.OSUpgradeProgress)
+			if !ok {
+				klog.V(1).InfoS("skipping unexpected watch object type",
+					"type", fmt.Sprintf("%T", evt.Object),
+				)
+				continue
+			}
+
+			if osup.ResourceVersion != "" {
+				*resourceVersion = osup.ResourceVersion
+			}
+
+			if !targetsNode(osup, nodeName) {
+				continue
+			}
+
+			klog.V(4).InfoS("received osupgradeprogress event",
+				"name", osup.Name,
+				"node", nodeName,
+				"phase", osup.StatusPhase(),
+				"eventType", evt.Type,
+				"resourceVersion", osup.ResourceVersion,
+			)
+
+			if err := osupgradeController.HandleOSUpgradeProgress(ctx, clients, namespace, nodeName, osup); err != nil {
+				klog.ErrorS(err, "failed to handle osupgradeprogress",
+					"name", osup.Name,
+					"node", nodeName,
+					"eventType", evt.Type,
+				)
+			}
+		}
+	}
+}
+
+func targetsNode(osup *monov1alpha1.OSUpgradeProgress, nodeName string) bool {
+	if osup == nil {
 		return false
 	}
-
-	if selector.Empty() {
-		return true
-	}
-
-	if selector.Matches(nodeLabels) {
-		return true
-	}
-
-	return false
-}
-
-func statusPhase(st *monov1alpha1.OSUpgradeStatus) string {
-	if st == nil {
-		return ""
-	}
-	return string(st.Phase)
+	return osup.Spec.NodeName == nodeName
 }

clitools/pkg/cmd/controller/controller.go

@@ -0,0 +1,217 @@
+package controller
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"net"
+	"net/http"
+	"os"
+	"time"
+
+	"github.com/spf13/cobra"
+	"k8s.io/cli-runtime/pkg/genericclioptions"
+	"k8s.io/klog/v2"
+
+	mkscontroller "example.com/monok8s/pkg/controller"
+	osupgradectrl "example.com/monok8s/pkg/controller/osupgrade"
+	"example.com/monok8s/pkg/kube"
+)
+
+type ServerConfig struct {
+	Namespace         string `json:"namespace,omitempty"`
+	TLSCertFile       string `json:"tlsCertFile,omitempty"`
+	TLSPrivateKeyFile string `json:"tlsPrivateKeyFile,omitempty"`
+}
+
+func NewCmdController(flags *genericclioptions.ConfigFlags) *cobra.Command {
+	var conf ServerConfig
+
+	cmd := &cobra.Command{
+		Use:   "controller",
+		Short: "Start a controller that handles OSUpgrade resources",
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			ns, _, err := flags.ToRawKubeConfigLoader().Namespace()
+			if err != nil {
+				return err
+			}
+			conf.Namespace = ns
+
+			ctx := cmd.Context()
+
+			klog.InfoS("starting controller", "namespace", conf.Namespace)
+
+			clients, err := kube.NewClients(flags)
+			if err != nil {
+				return err
+			}
+
+			ctx, cancel := context.WithCancel(ctx)
+			defer cancel()
+
+			httpErrCh := make(chan error, 1)
+			watchErrCh := make(chan error, 1)
+
+			go func() {
+				klog.InfoS("starting OSUpgrade watch loop", "namespace", conf.Namespace)
+				watchErrCh <- osupgradectrl.Watch(ctx, clients, conf.Namespace)
+			}()
+
+			go func() {
+				httpErrCh <- listenAndServe(ctx, clients, conf)
+			}()
+
+			select {
+			case <-ctx.Done():
+				klog.InfoS("controller context canceled")
+				return ctx.Err()
+
+			case err := <-watchErrCh:
+				if err != nil && !errors.Is(err, context.Canceled) {
+					cancel()
+					return err
+				}
+				cancel()
+				return nil
+
+			case err := <-httpErrCh:
+				if err != nil && !errors.Is(err, context.Canceled) {
+					cancel()
+					return err
+				}
+				cancel()
+				return nil
+			}
+		},
+	}
+
+	cmd.Flags().StringVar(&conf.TLSCertFile, "tls-cert-file", conf.TLSCertFile,
+		"File containing x509 Certificate used for serving HTTPS (with intermediate certs, if any, concatenated after server cert).")
+	cmd.Flags().StringVar(&conf.TLSPrivateKeyFile, "tls-private-key-file", conf.TLSPrivateKeyFile,
+		"File containing x509 private key matching --tls-cert-file.")
+
+	return cmd
+}
+
+func listenAndServe(ctx context.Context, clients *kube.Clients, conf ServerConfig) error {
+	nodeName := os.Getenv("NODE_NAME")
+
+	controllerServer := mkscontroller.NewServer(ctx, clients, conf.Namespace, nodeName)
+
+	healthMux := http.NewServeMux()
+	healthMux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("ok\n"))
+	})
+	healthMux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		_, _ = w.Write([]byte("ok\n"))
+	})
+
+	healthAddr := net.JoinHostPort("", "8080")
+	controllerAddr := net.JoinHostPort("", "8443")
+
+	healthHTTPServer := &http.Server{
+		Addr:           healthAddr,
+		Handler:        healthMux,
+		IdleTimeout:    90 * time.Second,
+		ReadTimeout:    10 * time.Second,
+		WriteTimeout:   10 * time.Second,
+		MaxHeaderBytes: 1 << 20,
+	}
+
+	controllerHTTPServer := &http.Server{
+		Addr:           controllerAddr,
+		Handler:        controllerServer,
+		IdleTimeout:    90 * time.Second,
+		ReadTimeout:    4 * time.Minute,
+		WriteTimeout:   4 * time.Minute,
+		MaxHeaderBytes: 1 << 20,
+	}
+
+	serverErrCh := make(chan error, 2)
+
+	go func() {
+		klog.InfoS("starting health HTTP server", "addr", healthAddr)
+
+		err := healthHTTPServer.ListenAndServe()
+		if err != nil && !errors.Is(err, http.ErrServerClosed) {
+			serverErrCh <- fmt.Errorf("health HTTP server: %w", err)
+			return
+		}
+
+		serverErrCh <- nil
+	}()
+
+	go func() {
+		if conf.TLSCertFile != "" {
+			klog.InfoS("starting controller HTTPS server",
+				"addr", controllerAddr,
+				"certFile", conf.TLSCertFile,
+				"keyFile", conf.TLSPrivateKeyFile,
+			)
+
+			err := controllerHTTPServer.ListenAndServeTLS(conf.TLSCertFile, conf.TLSPrivateKeyFile)
+			if err != nil && !errors.Is(err, http.ErrServerClosed) {
+				serverErrCh <- fmt.Errorf("controller HTTPS server: %w", err)
+				return
+			}
+
+			serverErrCh <- nil
+			return
+		}
+
+		klog.InfoS("starting controller HTTP server", "addr", controllerAddr)
+
+		err := controllerHTTPServer.ListenAndServe()
+		if err != nil && !errors.Is(err, http.ErrServerClosed) {
+			serverErrCh <- fmt.Errorf("controller HTTP server: %w", err)
+			return
+		}
+
+		serverErrCh <- nil
+	}()
+
+	select {
+	case <-ctx.Done():
+		klog.InfoS("shutting down HTTP servers",
+			"healthAddr", healthAddr,
+			"controllerAddr", controllerAddr,
+		)
+
+		shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+		defer cancel()
+
+		var errs []error
+
+		if err := healthHTTPServer.Shutdown(shutdownCtx); err != nil {
+			errs = append(errs, fmt.Errorf("shutdown health HTTP server: %w", err))
+		}
+
+		if err := controllerHTTPServer.Shutdown(shutdownCtx); err != nil {
+			errs = append(errs, fmt.Errorf("shutdown controller HTTP server: %w", err))
+		}
+
+		for i := 0; i < 2; i++ {
+			if err := <-serverErrCh; err != nil {
+				errs = append(errs, err)
+			}
+		}
+
+		if len(errs) > 0 {
+			return errors.Join(errs...)
+		}
+
+		return context.Canceled
+
+	case err := <-serverErrCh:
+		if err != nil {
+			klog.ErrorS(err, "HTTP server failed")
+			return err
+		}
+
+		// One server exited cleanly unexpectedly. Treat that as failure because
+		// the process should keep both servers alive until ctx is canceled.
+		return fmt.Errorf("HTTP server exited unexpectedly")
+	}
+}

clitools/pkg/cmd/create/create.go

@@ -1,13 +1,18 @@
 package create
 
 import (
+	"bytes"
 	"fmt"
 	"github.com/spf13/cobra"
+	"k8s.io/cli-runtime/pkg/genericclioptions"
+	"os"
+	"strings"
 
+	assets "example.com/monok8s/pkg/assets"
 	render "example.com/monok8s/pkg/render"
 )
 
-func NewCmdCreate() *cobra.Command {
+func NewCmdCreate(flags *genericclioptions.ConfigFlags) *cobra.Command {
 	cmd := &cobra.Command{Use: "create", Short: "Create starter resources"}
 	cmd.AddCommand(
 		&cobra.Command{
@@ -26,7 +31,12 @@ func NewCmdCreate() *cobra.Command {
 			Use:   "osupgrade",
 			Short: "Print an OSUpgrade template",
 			RunE: func(cmd *cobra.Command, _ []string) error {
-				out, err := render.RenderOSUpgrade()
+				ns, _, err := flags.ToRawKubeConfigLoader().Namespace()
+				if err != nil {
+					return err
+				}
+
+				out, err := render.RenderOSUpgrade(ns)
 				if err != nil {
 					return err
 				}
@@ -34,6 +44,143 @@ func NewCmdCreate() *cobra.Command {
 				return err
 			},
 		},
+		&cobra.Command{
+			Use:   "crds",
+			Short: "Print the bundled CRDs",
+			RunE: func(cmd *cobra.Command, _ []string) error {
+				return assets.PrintCRDs(cmd.OutOrStdout())
+			},
+		},
 	)
+
+	var authorizedKeysPath string
+
+	sshdcmd := cobra.Command{
+		Use:   "sshd",
+		Short: "Print sshd deployments template",
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			ns, _, err := flags.ToRawKubeConfigLoader().Namespace()
+			if err != nil {
+				return err
+			}
+
+			authorizedKeys, err := readAuthorizedKeysFile(authorizedKeysPath)
+			if err != nil {
+				return err
+			}
+
+			out, err := render.RenderSSHDDeployments(ns, authorizedKeys)
+			if err != nil {
+				return err
+			}
+
+			_, err = fmt.Fprint(cmd.OutOrStdout(), out)
+			return err
+		},
+	}
+
+	sshdcmd.Flags().StringVar(&authorizedKeysPath, "authkeys", "", "path to authorized_keys file")
+
+	cmd.AddCommand(&sshdcmd)
+
+	cconf := render.ControllerConf{}
+	controllercmd := cobra.Command{
+		Use:   "controller",
+		Short: "Print controller deployments template",
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if len(cconf.ImagePullSecrets) > 0 && strings.TrimSpace(cconf.Image) == "" {
+				return fmt.Errorf("--image-pull-secret requires --image")
+			}
+
+			ns, _, err := flags.ToRawKubeConfigLoader().Namespace()
+			if err != nil {
+				return err
+			}
+
+			cconf.Namespace = ns
+
+			out, err := render.RenderControllerDeployments(cconf)
+			if err != nil {
+				return err
+			}
+
+			_, err = fmt.Fprint(cmd.OutOrStdout(), out)
+			return err
+		},
+	}
+
+	controllercmd.Flags().StringVar(
+		&cconf.Image,
+		"image",
+		"",
+		"Controller image, including optional registry and tag",
+	)
+	controllercmd.Flags().StringSliceVar(
+		&cconf.ImagePullSecrets,
+		"image-pull-secret",
+		nil,
+		"Image pull secret name for the agent image; may be specified multiple times or as a comma-separated list",
+	)
+
+	cmd.AddCommand(&controllercmd)
+
+	aconf := render.AgentConf{}
+	agentcmd := cobra.Command{
+		Use:   "agent",
+		Short: "Print agent daemonsets template",
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			if len(aconf.ImagePullSecrets) > 0 && strings.TrimSpace(aconf.Image) == "" {
+				return fmt.Errorf("--image-pull-secret requires --image")
+			}
+
+			ns, _, err := flags.ToRawKubeConfigLoader().Namespace()
+			if err != nil {
+				return err
+			}
+
+			aconf.Namespace = ns
+
+			out, err := render.RenderAgentDaemonSets(aconf)
+			if err != nil {
+				return err
+			}
+
+			_, err = fmt.Fprint(cmd.OutOrStdout(), out)
+			return err
+		},
+	}
+
+	agentcmd.Flags().StringVar(
+		&aconf.Image,
+		"image",
+		"",
+		"Agent image, including optional registry and tag",
+	)
+	agentcmd.Flags().StringSliceVar(
+		&aconf.ImagePullSecrets,
+		"image-pull-secret",
+		nil,
+		"Image pull secret name for the agent image; may be specified multiple times or as a comma-separated list",
+	)
+
+	cmd.AddCommand(&agentcmd)
+
 	return cmd
 }
+
+func readAuthorizedKeysFile(path string) (string, error) {
+	if path == "" {
+		return "", fmt.Errorf("--authkeys is required")
+	}
+
+	b, err := os.ReadFile(path)
+	if err != nil {
+		return "", fmt.Errorf("read authorized_keys file %q: %w", path, err)
+	}
+
+	if len(bytes.TrimSpace(b)) == 0 {
+		return "", fmt.Errorf("authorized_keys file %q is empty", path)
+	}
+
+	return string(b), nil
+}

clitools/pkg/cmd/root/root.go

@@ -6,6 +6,7 @@ import (
 
 	agentcmd "example.com/monok8s/pkg/cmd/agent"
 	checkconfigcmd "example.com/monok8s/pkg/cmd/checkconfig"
+	controllercmd "example.com/monok8s/pkg/cmd/controller"
 	createcmd "example.com/monok8s/pkg/cmd/create"
 	initcmd "example.com/monok8s/pkg/cmd/initcmd"
 	internalcmd "example.com/monok8s/pkg/cmd/internal"
@@ -18,8 +19,10 @@ import (
 func init() {
 	klog.InitFlags(nil)
 
+	_ = flag.Set("logtostderr", "true")
+
 	if os.Getenv("DEBUG") != "" {
-		_ = flag.Set("v", "4") // debug level
+		_ = flag.Set("v", "4")
 	} else {
 		_ = flag.Set("v", "0")
 	}
@@ -38,14 +41,20 @@ func NewRootCmd() *cobra.Command {
 		},
 	}
 
+	// Expose klog stdlib flags through Cobra/pflag.
+	cmd.PersistentFlags().AddGoFlagSet(flag.CommandLine)
+
 	flags.AddFlags(cmd.PersistentFlags())
+
 	cmd.AddCommand(
 		versioncmd.NewCmdVersion(),
 		initcmd.NewCmdInit(flags),
 		checkconfigcmd.NewCmdCheckConfig(),
-		createcmd.NewCmdCreate(),
+		createcmd.NewCmdCreate(flags),
 		agentcmd.NewCmdAgent(flags),
+		controllercmd.NewCmdController(flags),
 		internalcmd.NewCmdInternal(),
 	)
+
 	return cmd
 }

clitools/pkg/cmd/version/version.go

@@ -1,22 +1,76 @@
-package apply
+package version
 
 import (
+	"encoding/json"
 	"fmt"
 
 	"github.com/spf13/cobra"
 
-	buildInfo "example.com/monok8s/pkg/buildinfo"
+	buildinfo "example.com/monok8s/pkg/buildinfo"
 )
 
+type versionInfo struct {
+	Version     string `json:"version"`
+	GitRevision string `json:"gitRevision"`
+	Timestamp   string `json:"timestamp"`
+	KubeVersion string `json:"kubernetesVersion"`
+}
+
 func NewCmdVersion() *cobra.Command {
+	var (
+		shortOutput      bool
+		jsonOutput       bool
+		kubernetesOutput bool
+	)
+
 	cmd := &cobra.Command{
 		Use:   "version",
-		Short: "Print the version information",
+		Short: "Print version information",
+		Args:  cobra.NoArgs,
 		RunE: func(cmd *cobra.Command, _ []string) error {
+			info := versionInfo{
+				Version:     buildinfo.Version,
+				GitRevision: buildinfo.GitRevision,
+				Timestamp:   buildinfo.Timestamp,
+				KubeVersion: buildinfo.KubeVersion,
+			}
 
-			_, err := fmt.Fprintln(cmd.OutOrStdout(), fmt.Sprintf("%s %s(%s)", buildInfo.Version, buildInfo.GitRevision, buildInfo.Timestamp))
-			return err
+			out := cmd.OutOrStdout()
+
+			switch {
+			case jsonOutput:
+				enc := json.NewEncoder(out)
+				enc.SetIndent("", "  ")
+				return enc.Encode(info)
+
+			case kubernetesOutput:
+				_, err := fmt.Fprintln(out, info.KubeVersion)
+				return err
+
+			case shortOutput:
+				_, err := fmt.Fprintln(out, info.Version)
+				return err
+
+			default:
+				_, err := fmt.Fprintf(
+					out,
+					"Version:    %s\nGit commit: %s\nBuilt at:   %s\nKubernetes: %s\n",
+					info.Version,
+					info.GitRevision,
+					info.Timestamp,
+					info.KubeVersion,
+				)
+				return err
+			}
 		},
 	}
+
+	flags := cmd.Flags()
+	flags.BoolVar(&shortOutput, "short", false, "Show only the application version")
+	flags.BoolVar(&jsonOutput, "json", false, "Show version information as JSON")
+	flags.BoolVarP(&kubernetesOutput, "kubernetes", "k", false, "Show only the Kubernetes version this binary was built for")
+
+	cmd.MarkFlagsMutuallyExclusive("short", "json", "kubernetes")
+
 	return cmd
 }

clitools/pkg/controller/osimage/write_throttle_other.go

@@ -15,4 +15,5 @@ func newNoopAdaptiveWriteController() *adaptiveWriteController {
 }
 
 func (c *adaptiveWriteController) Wait(ctx context.Context, n int) error { return nil }
-func (c *adaptiveWriteController) ObserveWrite(n int, dur interface{})   {}
+func (c *adaptiveWriteController) ObserveWrite(n int)                    {}
+func (c *adaptiveWriteController) ObserveSync()                          {}

clitools/pkg/controller/osupgrade/handler.go

@@ -41,26 +41,62 @@ func (r *UpgradeRunner) Run(fn func() error) error {
 	return fn()
 }
 
-func HandleOSUpgrade(ctx context.Context, clients *kube.Clients,
-	namespace string, nodeName string,
-	osu *monov1alpha1.OSUpgrade,
+func HandleOSUpgradeProgress(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace string,
+	nodeName string,
+	osup *monov1alpha1.OSUpgradeProgress,
 ) error {
 	return r.Run(func() error {
-		return handleOSUpgradeLocked(ctx, clients, namespace, nodeName, osu)
+		return handleOSUpgradeProgressLocked(ctx, clients, namespace, nodeName, osup)
 	})
 }
 
-func handleOSUpgradeLocked(ctx context.Context, clients *kube.Clients,
-	namespace string, nodeName string,
-	osu *monov1alpha1.OSUpgrade,
+func handleOSUpgradeProgressLocked(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace string,
+	nodeName string,
+	osup *monov1alpha1.OSUpgradeProgress,
 ) error {
-	osup, err := ensureProgressHeartbeat(ctx, clients, namespace, nodeName, osu)
-	if err != nil {
-		return err
+	if osup == nil {
+		return fmt.Errorf("osupgradeprogress is nil")
 	}
 
-	klog.InfoS("handling osupgrade",
-		"name", osu.Name,
+	if osup.Spec.NodeName != nodeName {
+		klog.V(4).InfoS("skipping osupgradeprogress due to nodeName mismatch",
+			"name", osup.Name,
+			"node", nodeName,
+			"target", osup.Spec.NodeName,
+		)
+		return nil
+	}
+
+	if !shouldProcessProgress(osup) {
+		klog.V(2).InfoS("skipping osupgradeprogress due to phase",
+			"name", osup.Name,
+			"node", nodeName,
+			"phase", osup.StatusPhase(),
+		)
+		return nil
+	}
+
+	parentName := osup.Spec.SourceRef.Name
+	if parentName == "" {
+		return failProgress(ctx, clients, osup, "resolve parent osupgrade", fmt.Errorf("missing spec.osUpgradeName"))
+	}
+
+	osu, err := clients.MonoKS.Monok8sV1alpha1().
+		OSUpgrades(namespace).
+		Get(ctx, parentName, metav1.GetOptions{})
+	if err != nil {
+		return failProgress(ctx, clients, osup, "resolve parent osupgrade", err)
+	}
+
+	klog.InfoS("handling osupgradeprogress",
+		"name", osup.Name,
+		"osupgrade", osu.Name,
 		"node", nodeName,
 		"desiredVersion", osu.Spec.DesiredVersion,
 	)
@@ -95,7 +131,9 @@ func handleOSUpgradeLocked(ctx context.Context, clients *kube.Clients,
 		now := metav1.Now()
 		cur.Status.CurrentVersion = buildinfo.KubeVersion
 		cur.Status.TargetVersion = plan.ResolvedTarget
+		cur.Status.PlannedPath = plannedPath(plan)
 		cur.Status.Phase = monov1alpha1.OSUpgradeProgressPhaseDownloading
+		cur.Status.ObservedRetryNonce = cur.Spec.RetryNonce
 		cur.Status.Message = fmt.Sprintf("downloading image: %s", first.URL)
 		cur.Status.LastUpdatedAt = &now
 	})
@@ -209,6 +247,27 @@ func handleOSUpgradeLocked(ctx context.Context, clients *kube.Clients,
 	select {}
 }
 
+func shouldProcessProgress(osup *monov1alpha1.OSUpgradeProgress) bool {
+	if osup == nil {
+		return false
+	}
+
+	if osup.Status == nil {
+		return false
+	}
+
+	switch osup.Status.Phase {
+	case "",
+		monov1alpha1.OSUpgradeProgressPhasePending,
+		monov1alpha1.OSUpgradeProgressPhaseRebooting:
+		return true
+	case monov1alpha1.OSUpgradeProgressPhaseFailed:
+		return osup.Spec.RetryNonce != osup.Status.ObservedRetryNonce
+	default:
+		return false
+	}
+}
+
 func triggerReboot() error {
 	_ = os.WriteFile("/proc/sysrq-trigger", []byte("s\n"), 0)
 	_ = os.WriteFile("/proc/sysrq-trigger", []byte("u\n"), 0)

clitools/pkg/controller/osupgrade/planner.go

@@ -224,9 +224,9 @@ func calculatePath(current, target string, available []string) ([]string, error)
 		add(latestCurMinor)
 	}
 
-	// Step 2: walk each intermediate minor using the lowest available patch in that minor.
+	// Step 2: walk each intermediate minor using the latest available patch in that minor.
 	for minor := cur.Minor + 1; minor < tgt.Minor; minor++ {
-		bridge, ok := lowestPatchInMinor(versions, cur.Major, minor)
+		bridge, ok := latestAnyPatchInMinor(versions, cur.Major, minor)
 		if !ok {
 			return nil, fmt.Errorf("no available bridge version for v%d.%d.x", cur.Major, minor)
 		}
@@ -239,6 +239,23 @@ func calculatePath(current, target string, available []string) ([]string, error)
 	return versionsToStrings(path), nil
 }
 
+func latestAnyPatchInMinor(versions []Version, major, minor int) (Version, bool) {
+	var found Version
+	ok := false
+
+	for _, v := range versions {
+		if v.Major != major || v.Minor != minor {
+			continue
+		}
+		if !ok || found.Compare(v) < 0 {
+			found = v
+			ok = true
+		}
+	}
+
+	return found, ok
+}
+
 func parseAndSortVersions(raw []string) ([]Version, error) {
 	out := make([]Version, 0, len(raw))
 	seen := map[string]struct{}{}
@@ -318,6 +335,14 @@ func lowestPatchInMinor(versions []Version, major, minor int) (Version, bool) {
 	return Version{}, false
 }
 
+func plannedPath(plan *Plan) []string {
+	ppath := []string{}
+	for _, img := range plan.Path {
+		ppath = append(ppath, img.Version)
+	}
+	return ppath
+}
+
 func versionsToStrings(vs []Version) []string {
 	out := make([]string, 0, len(vs))
 	for _, v := range vs {

clitools/pkg/controller/osupgrade/planner_test.go

@@ -0,0 +1,149 @@
+package osupgrade
+
+import (
+	"reflect"
+	"testing"
+)
+
+func TestCalculatePath(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name      string
+		current   string
+		target    string
+		available []string
+		want      []string
+		wantErr   bool
+	}{
+		{
+			name:      "same version returns nil path",
+			current:   "v1.34.6",
+			target:    "v1.34.6",
+			available: []string{"v1.34.6"},
+			want:      nil,
+			wantErr:   false,
+		},
+		{
+			name:      "same minor jumps directly to target",
+			current:   "v1.34.1",
+			target:    "v1.34.6",
+			available: []string{"v1.34.1", "v1.34.3", "v1.34.6"},
+			want:      []string{"v1.34.6"},
+			wantErr:   false,
+		},
+		{
+			name:      "next minor direct jump when no current minor patch available",
+			current:   "v1.34.6",
+			target:    "v1.35.3",
+			available: []string{"v1.34.6", "v1.35.1", "v1.35.3"},
+			want:      []string{"v1.35.3"},
+			wantErr:   false,
+		},
+		{
+			name:      "finish current minor then target",
+			current:   "v1.34.1",
+			target:    "v1.35.3",
+			available: []string{"v1.34.1", "v1.34.6", "v1.35.1", "v1.35.3"},
+			want:      []string{"v1.34.6", "v1.35.3"},
+			wantErr:   false,
+		},
+		{
+			name:      "multi minor path uses latest bridge patch",
+			current:   "v1.33.10",
+			target:    "v1.35.3",
+			available: []string{"v1.34.1", "v1.34.6", "v1.35.1", "v1.35.3"},
+			want:      []string{"v1.34.6", "v1.35.3"},
+			wantErr:   false,
+		},
+		{
+			name:      "multi minor path finishes current minor and latest bridge patch",
+			current:   "v1.33.1",
+			target:    "v1.35.3",
+			available: []string{"v1.33.5", "v1.33.9", "v1.34.1", "v1.34.6", "v1.35.3"},
+			want:      []string{"v1.33.9", "v1.34.6", "v1.35.3"},
+			wantErr:   false,
+		},
+		{
+			name:      "duplicates in available are ignored",
+			current:   "v1.33.10",
+			target:    "v1.35.3",
+			available: []string{"v1.34.6", "v1.34.6", "v1.35.3", "v1.35.3"},
+			want:      []string{"v1.34.6", "v1.35.3"},
+			wantErr:   false,
+		},
+		{
+			name:      "target missing returns error",
+			current:   "v1.34.6",
+			target:    "v1.35.3",
+			available: []string{"v1.34.6", "v1.35.1"},
+			wantErr:   true,
+		},
+		{
+			name:      "missing bridge minor returns error",
+			current:   "v1.33.10",
+			target:    "v1.35.3",
+			available: []string{"v1.35.3"},
+			wantErr:   true,
+		},
+		{
+			name:      "downgrade not supported",
+			current:   "v1.35.3",
+			target:    "v1.34.6",
+			available: []string{"v1.34.6", "v1.35.3"},
+			wantErr:   true,
+		},
+		{
+			name:      "cross major not supported",
+			current:   "v1.35.3",
+			target:    "v2.0.0",
+			available: []string{"v1.35.3", "v2.0.0"},
+			wantErr:   true,
+		},
+		{
+			name:      "invalid current version returns error",
+			current:   "garbage",
+			target:    "v1.35.3",
+			available: []string{"v1.35.3"},
+			wantErr:   true,
+		},
+		{
+			name:      "invalid target version returns error",
+			current:   "v1.34.6",
+			target:    "wat",
+			available: []string{"v1.34.6", "v1.35.3"},
+			wantErr:   true,
+		},
+		{
+			name:      "invalid available version returns error",
+			current:   "v1.34.6",
+			target:    "v1.35.3",
+			available: []string{"v1.34.6", "broken", "v1.35.3"},
+			wantErr:   true,
+		},
+	}
+
+	for _, tt := range tests {
+		tt := tt
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+
+			got, err := calculatePath(tt.current, tt.target, tt.available)
+			if tt.wantErr {
+				if err == nil {
+					t.Fatalf("expected error, got nil; path=%v", got)
+				}
+				return
+			}
+
+			if err != nil {
+				t.Fatalf("unexpected error: %v", err)
+			}
+
+			if !reflect.DeepEqual(got, tt.want) {
+				t.Fatalf("calculatePath(%q, %q, %v)\n got:  %v\n want: %v",
+					tt.current, tt.target, tt.available, got, tt.want)
+			}
+		})
+	}
+}

clitools/pkg/controller/osupgrade/progress.go

@@ -7,40 +7,27 @@ import (
 
 	apierrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
-	"k8s.io/apimachinery/pkg/runtime"
-	"k8s.io/apimachinery/pkg/runtime/schema"
 	"k8s.io/client-go/util/retry"
 	"k8s.io/klog/v2"
 
 	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
-	"example.com/monok8s/pkg/buildinfo"
 	"example.com/monok8s/pkg/kube"
 )
 
-var (
-	unstructuredConverter = runtime.DefaultUnstructuredConverter
-
-	osup_gvr = schema.GroupVersionResource{
-		Group:    monov1alpha1.Group,
-		Version:  monov1alpha1.Version,
-		Resource: "osupgradeprogresses",
-	}
-)
-
-func ensureProgressHeartbeat(ctx context.Context, clients *kube.Clients,
-	namespace string, nodeName string,
+func EnsureOSUpgradeProgressForNode(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace string,
+	nodeName string,
 	osu *monov1alpha1.OSUpgrade,
-) (*monov1alpha1.OSUpgradeProgress, error) {
+) error {
+	if osu == nil {
+		return fmt.Errorf("osupgrade is nil")
+	}
+
 	name := fmt.Sprintf("%s-%s", osu.Name, nodeName)
 	now := metav1.Now()
 
-	currentVersion := buildinfo.KubeVersion
-	targetVersion := ""
-	if osu.Status != nil {
-		targetVersion = osu.Status.ResolvedVersion
-	}
-
 	progress := &monov1alpha1.OSUpgradeProgress{
 		TypeMeta: metav1.TypeMeta{
 			APIVersion: monov1alpha1.APIVersion,
@@ -57,83 +44,38 @@ func ensureProgressHeartbeat(ctx context.Context, clients *kube.Clients,
 			},
 		},
 		Status: &monov1alpha1.OSUpgradeProgressStatus{
-			CurrentVersion: currentVersion,
-			TargetVersion:  targetVersion,
-			Phase:          monov1alpha1.OSUpgradeProgressPhasePending,
-			LastUpdatedAt:  &now,
-			Message:        "acknowledged",
+			Phase:         monov1alpha1.OSUpgradeProgressPhasePending,
+			LastUpdatedAt: &now,
 		},
 	}
 
-	created, err := createProgress(ctx, clients, osup_gvr, progress)
+	created, err := createProgress(ctx, clients, progress)
 	if err == nil {
 		klog.InfoS("created osupgradeprogress", "name", created.Name, "namespace", created.Namespace)
-		return created, nil
+		return nil
 	}
 	if !apierrors.IsAlreadyExists(err) {
-		return nil, fmt.Errorf("create OSUpgradeProgress %s/%s: %w", namespace, name, err)
+		return fmt.Errorf("create OSUpgradeProgress %s/%s: %w", namespace, name, err)
 	}
 
-	var out *monov1alpha1.OSUpgradeProgress
-	err = retry.RetryOnConflict(retry.DefaultRetry, func() error {
-		existing, err := getProgress(ctx, clients, osup_gvr, namespace, name)
-		if err != nil {
-			return fmt.Errorf("get existing OSUpgradeProgress %s/%s: %w", namespace, name, err)
-		}
-
-		// Keep spec aligned with source and node.
-		existing.Spec.NodeName = nodeName
-		existing.Spec.SourceRef.Name = osu.Name
-
-		existing, err = updateProgressSpec(ctx, clients, osup_gvr, existing)
-		if err != nil {
-			if isUnknownUpdateResult(err) {
-				latest, getErr := getProgress(ctx, clients, osup_gvr, namespace, name)
-				if getErr == nil {
-					out = latest
-				}
-			}
-			return fmt.Errorf("update OSUpgradeProgress spec %s/%s: %w", namespace, name, err)
-		}
-
-		if existing.Status == nil {
-			existing.Status = &monov1alpha1.OSUpgradeProgressStatus{}
-		}
-
-		existing.Status.CurrentVersion = currentVersion
-		existing.Status.TargetVersion = targetVersion
-		existing.Status.LastUpdatedAt = &now
-
-		if existing.Status.Phase == "" {
-			existing.Status.Phase = monov1alpha1.OSUpgradeProgressPhasePending
-		}
-		if existing.Status.Message == "" {
-			existing.Status.Message = "acknowledged"
-		}
-
-		existing, err = updateProgressStatus(ctx, clients, osup_gvr, existing)
-		if err != nil {
-			if isUnknownUpdateResult(err) {
-				latest, getErr := getProgress(ctx, clients, osup_gvr, namespace, name)
-				if getErr == nil {
-					out = latest
-				}
-			}
-			return fmt.Errorf("update OSUpgradeProgress status %s/%s: %w", namespace, name, err)
-		}
-
-		out = existing
-		return nil
-	})
+	existing, err := getProgress(ctx, clients, namespace, name)
 	if err != nil {
-		if out != nil {
-			return out, nil
-		}
-		return nil, err
+		return fmt.Errorf("get existing OSUpgradeProgress %s/%s: %w", namespace, name, err)
 	}
 
-	klog.InfoS("updated osupgradeprogress", "name", out.Name, "namespace", out.Namespace)
-	return out, nil
+	if existing.Spec.NodeName != nodeName || existing.Spec.SourceRef.Name != osu.Name {
+		return fmt.Errorf(
+			"conflicting OSUpgradeProgress %s/%s already exists: got spec.nodeName=%q spec.sourceRef.name=%q, want nodeName=%q sourceRef.name=%q",
+			namespace,
+			name,
+			existing.Spec.NodeName,
+			existing.Spec.SourceRef.Name,
+			nodeName,
+			osu.Name,
+		)
+	}
+
+	return nil
 }
 
 func updateProgressRobust(
@@ -146,7 +88,7 @@ func updateProgressRobust(
 	var out *monov1alpha1.OSUpgradeProgress
 
 	err := retry.RetryOnConflict(retry.DefaultRetry, func() error {
-		current, err := getProgress(ctx, clients, osup_gvr, namespace, name)
+		current, err := getProgress(ctx, clients, namespace, name)
 		if err != nil {
 			return err
 		}
@@ -157,10 +99,10 @@ func updateProgressRobust(
 
 		mutate(current)
 
-		updated, err := updateProgressStatus(ctx, clients, osup_gvr, current)
+		updated, err := updateProgressStatus(ctx, clients, current)
 		if err != nil {
 			if isUnknownUpdateResult(err) {
-				latest, getErr := getProgress(ctx, clients, osup_gvr, namespace, name)
+				latest, getErr := getProgress(ctx, clients, namespace, name)
 				if getErr == nil {
 					out = latest
 				}
@@ -202,84 +144,55 @@ func isUnknownUpdateResult(err error) bool {
 func createProgress(
 	ctx context.Context,
 	clients *kube.Clients,
-	gvr schema.GroupVersionResource,
 	progress *monov1alpha1.OSUpgradeProgress,
 ) (*monov1alpha1.OSUpgradeProgress, error) {
-	obj, err := toUnstructured(progress)
+	toCreate := progress.DeepCopy()
+	toCreate.Status = nil
+
+	created, err := clients.MonoKS.
+		Monok8sV1alpha1().
+		OSUpgradeProgresses(toCreate.Namespace).
+		Create(ctx, toCreate, metav1.CreateOptions{})
 	if err != nil {
 		return nil, err
 	}
 
-	created, err := clients.Dynamic.
-		Resource(gvr).
-		Namespace(progress.Namespace).
-		Create(ctx, obj, metav1.CreateOptions{})
-	if err != nil {
-		return nil, err
+	if progress.Status != nil {
+		toUpdate := created.DeepCopy()
+		toUpdate.Status = progress.Status
+		return updateProgressStatus(ctx, clients, toUpdate)
 	}
 
-	return fromUnstructuredProgress(created)
-}
-
-func getProgress(
-	ctx context.Context,
-	clients *kube.Clients,
-	gvr schema.GroupVersionResource,
-	namespace, name string,
-) (*monov1alpha1.OSUpgradeProgress, error) {
-	got, err := clients.Dynamic.
-		Resource(gvr).
-		Namespace(namespace).
-		Get(ctx, name, metav1.GetOptions{})
-	if err != nil {
-		return nil, err
-	}
-
-	return fromUnstructuredProgress(got)
-}
-
-func updateProgressSpec(
-	ctx context.Context,
-	clients *kube.Clients,
-	gvr schema.GroupVersionResource,
-	progress *monov1alpha1.OSUpgradeProgress,
-) (*monov1alpha1.OSUpgradeProgress, error) {
-	obj, err := toUnstructured(progress)
-	if err != nil {
-		return nil, err
-	}
-
-	updated, err := clients.Dynamic.
-		Resource(gvr).
-		Namespace(progress.Namespace).
-		Update(ctx, obj, metav1.UpdateOptions{})
-	if err != nil {
-		return nil, err
-	}
-
-	return fromUnstructuredProgress(updated)
+	return created, nil
 }
 
 func updateProgressStatus(
 	ctx context.Context,
 	clients *kube.Clients,
-	gvr schema.GroupVersionResource,
 	progress *monov1alpha1.OSUpgradeProgress,
 ) (*monov1alpha1.OSUpgradeProgress, error) {
-	obj, err := toUnstructured(progress)
+	updated, err := clients.MonoKS.
+		Monok8sV1alpha1().
+		OSUpgradeProgresses(progress.Namespace).
+		UpdateStatus(ctx, progress, metav1.UpdateOptions{})
 	if err != nil {
-		return nil, err
+		return nil, fmt.Errorf(
+			"update status for OSUpgradeProgress %s/%s: %w",
+			progress.Namespace, progress.Name, err,
+		)
 	}
+	return updated, nil
+}
 
-	updated, err := clients.Dynamic.
-		Resource(gvr).
-		Namespace(progress.Namespace).
-		UpdateStatus(ctx, obj, metav1.UpdateOptions{})
-	if err != nil {
-		return nil, err
-	}
-
-	return fromUnstructuredProgress(updated)
+func getProgress(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace, name string,
+) (*monov1alpha1.OSUpgradeProgress, error) {
+	return clients.MonoKS.
+		Monok8sV1alpha1().
+		OSUpgradeProgresses(namespace).
+		Get(ctx, name, metav1.GetOptions{})
 }
 
 func failProgress(
@@ -296,6 +209,7 @@ func failProgress(
 			cur.Status = &monov1alpha1.OSUpgradeProgressStatus{}
 		}
 
+		cur.Status.ObservedRetryNonce = cur.Spec.RetryNonce
 		cur.Status.LastUpdatedAt = &now
 		cur.Status.Message = fmt.Sprintf("%s: %v", action, cause)
 		cur.Status.Phase = monov1alpha1.OSUpgradeProgressPhaseFailed
@@ -324,8 +238,11 @@ func markProgressCompleted(
 			cur.Status = &monov1alpha1.OSUpgradeProgressStatus{}
 		}
 
+		cur.Status.ObservedRetryNonce = cur.Spec.RetryNonce
 		cur.Status.Phase = monov1alpha1.OSUpgradeProgressPhaseCompleted
 		cur.Status.Message = message
+		cur.Status.CurrentVersion = osup.Status.CurrentVersion
+		cur.Status.TargetVersion = osup.Status.TargetVersion
 		cur.Status.LastUpdatedAt = &now
 		cur.Status.CompletedAt = &now
 	})
@@ -335,19 +252,3 @@ func markProgressCompleted(
 
 	return nil
 }
-
-func toUnstructured(progress *monov1alpha1.OSUpgradeProgress) (*unstructured.Unstructured, error) {
-	m, err := unstructuredConverter.ToUnstructured(progress)
-	if err != nil {
-		return nil, fmt.Errorf("convert OSUpgradeProgress to unstructured: %w", err)
-	}
-	return &unstructured.Unstructured{Object: m}, nil
-}
-
-func fromUnstructuredProgress(obj *unstructured.Unstructured) (*monov1alpha1.OSUpgradeProgress, error) {
-	var progress monov1alpha1.OSUpgradeProgress
-	if err := unstructuredConverter.FromUnstructured(obj.Object, &progress); err != nil {
-		return nil, fmt.Errorf("convert unstructured to OSUpgradeProgress: %w", err)
-	}
-	return &progress, nil
-}

clitools/pkg/controller/osupgrade/watch.go

@@ -0,0 +1,345 @@
+package osupgrade
+
+import (
+	"context"
+	"fmt"
+	"time"
+
+	corev1 "k8s.io/api/core/v1"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/labels"
+	"k8s.io/apimachinery/pkg/watch"
+	"k8s.io/klog/v2"
+
+	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	"example.com/monok8s/pkg/kube"
+)
+
+func Watch(ctx context.Context, clients *kube.Clients, namespace string) error {
+	var resourceVersion string
+
+	for {
+		if ctx.Err() != nil {
+			return ctx.Err()
+		}
+
+		err := watchOnce(ctx, clients, namespace, &resourceVersion)
+		if err != nil {
+			if ctx.Err() != nil {
+				return ctx.Err()
+			}
+
+			// Expired RV is normal enough; clear it and relist.
+			if apierrors.IsResourceExpired(err) {
+				klog.InfoS("OSUpgrade watch resourceVersion expired; resetting",
+					"namespace", namespace,
+					"resourceVersion", resourceVersion,
+				)
+				resourceVersion = ""
+			} else {
+				klog.ErrorS(err, "OSUpgrade watch failed; retrying",
+					"namespace", namespace,
+					"resourceVersion", resourceVersion,
+				)
+			}
+
+			select {
+			case <-ctx.Done():
+				return ctx.Err()
+			case <-time.After(2 * time.Second):
+			}
+			continue
+		}
+	}
+}
+
+func watchOnce(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace string,
+	resourceVersion *string,
+) error {
+	// Cold start: list existing objects once, handle them, then watch from list RV.
+	if *resourceVersion == "" {
+		list, err := clients.MonoKS.
+			Monok8sV1alpha1().
+			OSUpgrades(namespace).
+			List(ctx, metav1.ListOptions{})
+		if err != nil {
+			return fmt.Errorf("list OSUpgrades: %w", err)
+		}
+
+		for i := range list.Items {
+			osu := &list.Items[i]
+			handled, err := handleOSUpgrade(ctx, clients, namespace, osu)
+			if err != nil {
+				klog.ErrorS(err, "reconcile existing OSUpgrade failed",
+					"name", osu.Name,
+					"resourceVersion", osu.ResourceVersion,
+				)
+				continue
+			}
+			if !handled {
+				klog.V(2).InfoS("skipping existing OSUpgrade",
+					"name", osu.Name,
+					"phase", osu.StatusPhase(),
+				)
+			}
+		}
+
+		*resourceVersion = list.ResourceVersion
+		klog.InfoS("initial OSUpgrade sync complete",
+			"namespace", namespace,
+			"resourceVersion", *resourceVersion,
+			"count", len(list.Items),
+		)
+	}
+
+	w, err := clients.MonoKS.
+		Monok8sV1alpha1().
+		OSUpgrades(namespace).
+		Watch(ctx, metav1.ListOptions{
+			ResourceVersion:     *resourceVersion,
+			AllowWatchBookmarks: true,
+		})
+	if err != nil {
+		return fmt.Errorf("watch OSUpgrades: %w", err)
+	}
+	defer w.Stop()
+
+	klog.InfoS("watching OSUpgrades",
+		"namespace", namespace,
+		"resourceVersion", *resourceVersion,
+	)
+
+	for {
+		select {
+		case <-ctx.Done():
+			return ctx.Err()
+
+		case evt, ok := <-w.ResultChan():
+			if !ok {
+				return fmt.Errorf("watch channel closed")
+			}
+
+			switch evt.Type {
+			case watch.Bookmark:
+				if rv := extractResourceVersion(evt.Object); rv != "" {
+					*resourceVersion = rv
+				}
+				continue
+
+			case watch.Error:
+				// Let outer loop retry / relist.
+				return fmt.Errorf("watch returned error event")
+
+			case watch.Deleted:
+				// Top-level delete does not require action here.
+				continue
+
+			case watch.Added, watch.Modified:
+				// handled below
+
+			default:
+				klog.V(1).InfoS("skipping unexpected watch event type",
+					"eventType", evt.Type,
+				)
+				continue
+			}
+
+			osu, ok := evt.Object.(*monov1alpha1.OSUpgrade)
+			if !ok {
+				klog.V(1).InfoS("skipping unexpected watch object type",
+					"type", fmt.Sprintf("%T", evt.Object),
+				)
+				continue
+			}
+
+			if osu.ResourceVersion != "" {
+				*resourceVersion = osu.ResourceVersion
+			}
+
+			handled, err := handleOSUpgrade(ctx, clients, namespace, osu)
+			if err != nil {
+				klog.ErrorS(err, "reconcile OSUpgrade failed",
+					"name", osu.Name,
+					"eventType", evt.Type,
+					"resourceVersion", osu.ResourceVersion,
+				)
+				continue
+			}
+			if !handled {
+				klog.V(2).InfoS("skipping OSUpgrade",
+					"name", osu.Name,
+					"eventType", evt.Type,
+					"phase", osu.StatusPhase(),
+				)
+			}
+		}
+	}
+}
+
+func handleOSUpgrade(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace string,
+	osu *monov1alpha1.OSUpgrade,
+) (bool, error) {
+	if !shouldHandle(osu) {
+		return false, nil
+	}
+
+	if osu.Status == nil || osu.Status.ObservedGeneration != osu.Generation {
+		return true, reconcileSpec(ctx, clients, namespace, osu)
+	}
+
+	if osu.Status.Phase == monov1alpha1.OSUpgradePhaseAccepted {
+		return true, reconcileFanout(ctx, clients, namespace, osu)
+	}
+
+	return false, nil
+}
+
+func reconcileSpec(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace string,
+	osu *monov1alpha1.OSUpgrade,
+) error {
+	osu = osu.DeepCopy()
+
+	osu.Status = &monov1alpha1.OSUpgradeStatus{
+		Phase:              monov1alpha1.OSUpgradePhaseAccepted,
+		ResolvedVersion:    osu.Spec.DesiredVersion,
+		ObservedGeneration: osu.Generation,
+	}
+
+	_, err := clients.MonoKS.
+		Monok8sV1alpha1().
+		OSUpgrades(namespace).
+		UpdateStatus(ctx, osu, metav1.UpdateOptions{})
+
+	return err
+}
+
+func reconcileFanout(
+	ctx context.Context,
+	clients *kube.Clients,
+	namespace string,
+	osu *monov1alpha1.OSUpgrade,
+) error {
+	nodeNames, err := listTargetNodeNames(ctx, clients, osu)
+	if err != nil {
+		return fmt.Errorf("list target nodes for %s: %w", osu.Name, err)
+	}
+
+	if len(nodeNames) == 0 {
+		klog.InfoS("no targets", "osupgrade", osu.Name)
+		return nil
+	}
+
+	klog.InfoS("ensuring OSUpgradeProgress for target nodes",
+		"osupgrade", osu.Name,
+		"targets", len(nodeNames),
+	)
+
+	for _, nodeName := range nodeNames {
+		if err := EnsureOSUpgradeProgressForNode(
+			ctx,
+			clients,
+			namespace,
+			nodeName,
+			osu,
+		); err != nil {
+			klog.ErrorS(err, "ensure OSUpgradeProgress for node failed",
+				"osupgrade", osu.Name,
+				"node", nodeName,
+			)
+		}
+	}
+
+	return nil
+}
+
+func listTargetNodeNames(
+	ctx context.Context,
+	clients *kube.Clients,
+	osu *monov1alpha1.OSUpgrade,
+) ([]string, error) {
+	selector := labels.SelectorFromSet(labels.Set{
+		monov1alpha1.NodeControlKey: "true",
+	})
+
+	if osu.Spec.NodeSelector != nil {
+		userSelector, err := metav1.LabelSelectorAsSelector(osu.Spec.NodeSelector)
+		if err != nil {
+			return nil, fmt.Errorf("invalid nodeSelector: %w", err)
+		}
+
+		reqs, selectable := userSelector.Requirements()
+		if !selectable {
+			selector = labels.Nothing()
+		} else {
+			selector = selector.Add(reqs...)
+		}
+	}
+
+	list, err := clients.Kubernetes.CoreV1().
+		Nodes().
+		List(ctx, metav1.ListOptions{
+			LabelSelector: selector.String(),
+		})
+	if err != nil {
+		return nil, fmt.Errorf("list nodes: %w", err)
+	}
+
+	out := make([]string, 0, len(list.Items))
+	for i := range list.Items {
+		node := &list.Items[i]
+		if shouldUseNode(node) {
+			out = append(out, node.Name)
+		}
+	}
+
+	return out, nil
+}
+
+func shouldUseNode(node *corev1.Node) bool {
+	return node != nil && node.Name != ""
+}
+
+func shouldHandle(osu *monov1alpha1.OSUpgrade) bool {
+	if osu == nil || osu.DeletionTimestamp != nil {
+		return false
+	}
+
+	if osu.Spec.DesiredVersion == "" {
+		return false
+	}
+
+	// NEW: initial processing stage
+	if osu.Status == nil {
+		return true
+	}
+
+	// Reconcile if spec changed
+	if osu.Status.ObservedGeneration != osu.Generation {
+		return true
+	}
+
+	// Fanout stage
+	return osu.Status.Phase == monov1alpha1.OSUpgradePhaseAccepted
+}
+
+func extractResourceVersion(obj interface{}) string {
+	type hasRV interface {
+		GetResourceVersion() string
+	}
+
+	if o, ok := obj.(hasRV); ok {
+		return o.GetResourceVersion()
+	}
+
+	return ""
+}

clitools/pkg/controller/server.go

@@ -0,0 +1,92 @@
+package controller
+
+import (
+	"context"
+	"net/http"
+	"time"
+
+	"example.com/monok8s/pkg/kube"
+
+	"github.com/emicklei/go-restful/v3"
+	"k8s.io/apiserver/pkg/server/httplog"
+)
+
+var statusesNoTracePred = httplog.StatusIsNot(
+	http.StatusOK,
+	http.StatusFound,
+	http.StatusMovedPermanently,
+	http.StatusTemporaryRedirect,
+	http.StatusBadRequest,
+	http.StatusNotFound,
+	http.StatusSwitchingProtocols,
+)
+
+type Server struct {
+	restfulCont *restful.Container
+
+	ctx       context.Context
+	clients   *kube.Clients
+	namespace string
+	nodeName  string
+	startedAt time.Time
+}
+
+type StatusResponse struct {
+	OK        bool   `json:"ok"`
+	Service   string `json:"service"`
+	Namespace string `json:"namespace,omitempty"`
+	NodeName  string `json:"nodeName,omitempty"`
+	UptimeSec int64  `json:"uptimeSec"`
+}
+
+func NewServer(ctx context.Context, clients *kube.Clients, namespace, nodeName string) *Server {
+	s := &Server{
+		ctx:       ctx,
+		clients:   clients,
+		namespace: namespace,
+		nodeName:  nodeName,
+		startedAt: time.Now(),
+	}
+	s.Initialize()
+	return s
+}
+
+func (s *Server) ServeHTTP(w http.ResponseWriter, req *http.Request) {
+	if s == nil {
+		http.Error(w, "server is nil", http.StatusInternalServerError)
+		return
+	}
+	if s.restfulCont == nil {
+		http.Error(w, "server not initialized", http.StatusInternalServerError)
+		return
+	}
+
+	handler := httplog.WithLogging(s.restfulCont, statusesNoTracePred)
+	handler.ServeHTTP(w, req)
+}
+
+func (s *Server) Initialize() {
+	s.restfulCont = restful.NewContainer()
+
+	ws := new(restful.WebService)
+	ws.Path("/")
+	ws.Consumes(restful.MIME_JSON)
+	ws.Produces(restful.MIME_JSON)
+
+	ws.Route(ws.GET("/status").To(s.queryStatus).
+		Doc("Return basic controller status"))
+
+	s.restfulCont.Add(ws)
+}
+
+func (s *Server) queryStatus(request *restful.Request, response *restful.Response) {
+	resp := StatusResponse{
+		OK:        true,
+		Service:   "monok8s-controller",
+		Namespace: s.namespace,
+		NodeName:  s.nodeName,
+		UptimeSec: int64(time.Since(s.startedAt).Seconds()),
+	}
+
+	_ = response.WriteHeaderAndEntity(http.StatusOK, resp)
+}

clitools/pkg/crds/crds.go

@@ -1,76 +0,0 @@
-package crds
-
-import (
-	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
-	apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-)
-
-func Definitions() []*apiextensionsv1.CustomResourceDefinition {
-	return []*apiextensionsv1.CustomResourceDefinition{
-		monoKSConfigCRD(),
-		osUpgradeCRD(),
-	}
-}
-
-func monoKSConfigCRD() *apiextensionsv1.CustomResourceDefinition {
-	return &apiextensionsv1.CustomResourceDefinition{
-		ObjectMeta: metav1.ObjectMeta{
-			Name: monov1alpha1.MonoKSConfigCRD,
-		},
-		Spec: apiextensionsv1.CustomResourceDefinitionSpec{
-			Group: monov1alpha1.Group,
-			Scope: apiextensionsv1.NamespaceScoped,
-			Names: apiextensionsv1.CustomResourceDefinitionNames{
-				Plural:     "monoksconfigs",
-				Singular:   "monoksconfig",
-				Kind:       "MonoKSConfig",
-				ShortNames: []string{"mkscfg"},
-			},
-			Versions: []apiextensionsv1.CustomResourceDefinitionVersion{{
-				Name:    "v1alpha1",
-				Served:  true,
-				Storage: true,
-				Schema: &apiextensionsv1.CustomResourceValidation{OpenAPIV3Schema: &apiextensionsv1.JSONSchemaProps{
-					Type: "object",
-					Properties: map[string]apiextensionsv1.JSONSchemaProps{
-						"spec":   {Type: "object", XPreserveUnknownFields: boolPtr(true)},
-						"status": {Type: "object", XPreserveUnknownFields: boolPtr(true)},
-					},
-				}},
-			}},
-		},
-	}
-}
-
-func osUpgradeCRD() *apiextensionsv1.CustomResourceDefinition {
-	return &apiextensionsv1.CustomResourceDefinition{
-		ObjectMeta: metav1.ObjectMeta{
-			Name: monov1alpha1.OSUpgradeCRD,
-		},
-		Spec: apiextensionsv1.CustomResourceDefinitionSpec{
-			Group: monov1alpha1.Group,
-			Scope: apiextensionsv1.NamespaceScoped,
-			Names: apiextensionsv1.CustomResourceDefinitionNames{
-				Plural:     "osupgrades",
-				Singular:   "osupgrade",
-				Kind:       "OSUpgrade",
-				ShortNames: []string{"osup"},
-			},
-			Versions: []apiextensionsv1.CustomResourceDefinitionVersion{{
-				Name:    "v1alpha1",
-				Served:  true,
-				Storage: true,
-				Schema: &apiextensionsv1.CustomResourceValidation{OpenAPIV3Schema: &apiextensionsv1.JSONSchemaProps{
-					Type: "object",
-					Properties: map[string]apiextensionsv1.JSONSchemaProps{
-						"spec":   {Type: "object", XPreserveUnknownFields: boolPtr(true)},
-						"status": {Type: "object", XPreserveUnknownFields: boolPtr(true)},
-					},
-				}},
-			}},
-		},
-	}
-}
-
-func boolPtr(v bool) *bool { return &v }

clitools/pkg/generated/clientset/versioned/clientset.go

@@ -0,0 +1,106 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package versioned
+
+import (
+	fmt "fmt"
+	http "net/http"
+
+	monok8sv1alpha1 "example.com/monok8s/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1"
+	discovery "k8s.io/client-go/discovery"
+	rest "k8s.io/client-go/rest"
+	flowcontrol "k8s.io/client-go/util/flowcontrol"
+)
+
+type Interface interface {
+	Discovery() discovery.DiscoveryInterface
+	Monok8sV1alpha1() monok8sv1alpha1.Monok8sV1alpha1Interface
+}
+
+// Clientset contains the clients for groups.
+type Clientset struct {
+	*discovery.DiscoveryClient
+	monok8sV1alpha1 *monok8sv1alpha1.Monok8sV1alpha1Client
+}
+
+// Monok8sV1alpha1 retrieves the Monok8sV1alpha1Client
+func (c *Clientset) Monok8sV1alpha1() monok8sv1alpha1.Monok8sV1alpha1Interface {
+	return c.monok8sV1alpha1
+}
+
+// Discovery retrieves the DiscoveryClient
+func (c *Clientset) Discovery() discovery.DiscoveryInterface {
+	if c == nil {
+		return nil
+	}
+	return c.DiscoveryClient
+}
+
+// NewForConfig creates a new Clientset for the given config.
+// If config's RateLimiter is not set and QPS and Burst are acceptable,
+// NewForConfig will generate a rate-limiter in configShallowCopy.
+// NewForConfig is equivalent to NewForConfigAndClient(c, httpClient),
+// where httpClient was generated with rest.HTTPClientFor(c).
+func NewForConfig(c *rest.Config) (*Clientset, error) {
+	configShallowCopy := *c
+
+	if configShallowCopy.UserAgent == "" {
+		configShallowCopy.UserAgent = rest.DefaultKubernetesUserAgent()
+	}
+
+	// share the transport between all clients
+	httpClient, err := rest.HTTPClientFor(&configShallowCopy)
+	if err != nil {
+		return nil, err
+	}
+
+	return NewForConfigAndClient(&configShallowCopy, httpClient)
+}
+
+// NewForConfigAndClient creates a new Clientset for the given config and http client.
+// Note the http client provided takes precedence over the configured transport values.
+// If config's RateLimiter is not set and QPS and Burst are acceptable,
+// NewForConfigAndClient will generate a rate-limiter in configShallowCopy.
+func NewForConfigAndClient(c *rest.Config, httpClient *http.Client) (*Clientset, error) {
+	configShallowCopy := *c
+	if configShallowCopy.RateLimiter == nil && configShallowCopy.QPS > 0 {
+		if configShallowCopy.Burst <= 0 {
+			return nil, fmt.Errorf("burst is required to be greater than 0 when RateLimiter is not set and QPS is set to greater than 0")
+		}
+		configShallowCopy.RateLimiter = flowcontrol.NewTokenBucketRateLimiter(configShallowCopy.QPS, configShallowCopy.Burst)
+	}
+
+	var cs Clientset
+	var err error
+	cs.monok8sV1alpha1, err = monok8sv1alpha1.NewForConfigAndClient(&configShallowCopy, httpClient)
+	if err != nil {
+		return nil, err
+	}
+
+	cs.DiscoveryClient, err = discovery.NewDiscoveryClientForConfigAndClient(&configShallowCopy, httpClient)
+	if err != nil {
+		return nil, err
+	}
+	return &cs, nil
+}
+
+// NewForConfigOrDie creates a new Clientset for the given config and
+// panics if there is an error in the config.
+func NewForConfigOrDie(c *rest.Config) *Clientset {
+	cs, err := NewForConfig(c)
+	if err != nil {
+		panic(err)
+	}
+	return cs
+}
+
+// New creates a new Clientset for the given RESTClient.
+func New(c rest.Interface) *Clientset {
+	var cs Clientset
+	cs.monok8sV1alpha1 = monok8sv1alpha1.New(c)
+
+	cs.DiscoveryClient = discovery.NewDiscoveryClient(c)
+	return &cs
+}

clitools/pkg/generated/clientset/versioned/fake/clientset_generated.go

@@ -0,0 +1,91 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package fake
+
+import (
+	clientset "example.com/monok8s/pkg/generated/clientset/versioned"
+	monok8sv1alpha1 "example.com/monok8s/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1"
+	fakemonok8sv1alpha1 "example.com/monok8s/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/fake"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/watch"
+	"k8s.io/client-go/discovery"
+	fakediscovery "k8s.io/client-go/discovery/fake"
+	"k8s.io/client-go/testing"
+)
+
+// NewSimpleClientset returns a clientset that will respond with the provided objects.
+// It's backed by a very simple object tracker that processes creates, updates and deletions as-is,
+// without applying any field management, validations and/or defaults. It shouldn't be considered a replacement
+// for a real clientset and is mostly useful in simple unit tests.
+//
+// Deprecated: NewClientset replaces this with support for field management, which significantly improves
+// server side apply testing. NewClientset is only available when apply configurations are generated (e.g.
+// via --with-applyconfig).
+func NewSimpleClientset(objects ...runtime.Object) *Clientset {
+	o := testing.NewObjectTracker(scheme, codecs.UniversalDecoder())
+	for _, obj := range objects {
+		if err := o.Add(obj); err != nil {
+			panic(err)
+		}
+	}
+
+	cs := &Clientset{tracker: o}
+	cs.discovery = &fakediscovery.FakeDiscovery{Fake: &cs.Fake}
+	cs.AddReactor("*", "*", testing.ObjectReaction(o))
+	cs.AddWatchReactor("*", func(action testing.Action) (handled bool, ret watch.Interface, err error) {
+		var opts metav1.ListOptions
+		if watchAction, ok := action.(testing.WatchActionImpl); ok {
+			opts = watchAction.ListOptions
+		}
+		gvr := action.GetResource()
+		ns := action.GetNamespace()
+		watch, err := o.Watch(gvr, ns, opts)
+		if err != nil {
+			return false, nil, err
+		}
+		return true, watch, nil
+	})
+
+	return cs
+}
+
+// Clientset implements clientset.Interface. Meant to be embedded into a
+// struct to get a default implementation. This makes faking out just the method
+// you want to test easier.
+type Clientset struct {
+	testing.Fake
+	discovery *fakediscovery.FakeDiscovery
+	tracker   testing.ObjectTracker
+}
+
+func (c *Clientset) Discovery() discovery.DiscoveryInterface {
+	return c.discovery
+}
+
+func (c *Clientset) Tracker() testing.ObjectTracker {
+	return c.tracker
+}
+
+// IsWatchListSemanticsSupported informs the reflector that this client
+// doesn't support WatchList semantics.
+//
+// This is a synthetic method whose sole purpose is to satisfy the optional
+// interface check performed by the reflector.
+// Returning true signals that WatchList can NOT be used.
+// No additional logic is implemented here.
+func (c *Clientset) IsWatchListSemanticsUnSupported() bool {
+	return true
+}
+
+var (
+	_ clientset.Interface = &Clientset{}
+	_ testing.FakeClient  = &Clientset{}
+)
+
+// Monok8sV1alpha1 retrieves the Monok8sV1alpha1Client
+func (c *Clientset) Monok8sV1alpha1() monok8sv1alpha1.Monok8sV1alpha1Interface {
+	return &fakemonok8sv1alpha1.FakeMonok8sV1alpha1{Fake: &c.Fake}
+}

clitools/pkg/generated/clientset/versioned/fake/doc.go

@@ -0,0 +1,6 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+// This package has the automatically generated fake clientset.
+package fake

clitools/pkg/generated/clientset/versioned/fake/register.go

@@ -0,0 +1,42 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package fake
+
+import (
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+	schema "k8s.io/apimachinery/pkg/runtime/schema"
+	serializer "k8s.io/apimachinery/pkg/runtime/serializer"
+	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
+)
+
+var scheme = runtime.NewScheme()
+var codecs = serializer.NewCodecFactory(scheme)
+
+var localSchemeBuilder = runtime.SchemeBuilder{
+	monok8sv1alpha1.AddToScheme,
+}
+
+// AddToScheme adds all types of this clientset into the given scheme. This allows composition
+// of clientsets, like in:
+//
+//	import (
+//	  "k8s.io/client-go/kubernetes"
+//	  clientsetscheme "k8s.io/client-go/kubernetes/scheme"
+//	  aggregatorclientsetscheme "k8s.io/kube-aggregator/pkg/client/clientset_generated/clientset/scheme"
+//	)
+//
+//	kclientset, _ := kubernetes.NewForConfig(c)
+//	_ = aggregatorclientsetscheme.AddToScheme(clientsetscheme.Scheme)
+//
+// After this, RawExtensions in Kubernetes types will serialize kube-aggregator types
+// correctly.
+var AddToScheme = localSchemeBuilder.AddToScheme
+
+func init() {
+	v1.AddToGroupVersion(scheme, schema.GroupVersion{Version: "v1"})
+	utilruntime.Must(AddToScheme(scheme))
+}

clitools/pkg/generated/clientset/versioned/scheme/doc.go

@@ -0,0 +1,6 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+// This package contains the scheme of the automatically generated clientset.
+package scheme

clitools/pkg/generated/clientset/versioned/scheme/register.go

@@ -0,0 +1,42 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package scheme
+
+import (
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+	schema "k8s.io/apimachinery/pkg/runtime/schema"
+	serializer "k8s.io/apimachinery/pkg/runtime/serializer"
+	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
+)
+
+var Scheme = runtime.NewScheme()
+var Codecs = serializer.NewCodecFactory(Scheme)
+var ParameterCodec = runtime.NewParameterCodec(Scheme)
+var localSchemeBuilder = runtime.SchemeBuilder{
+	monok8sv1alpha1.AddToScheme,
+}
+
+// AddToScheme adds all types of this clientset into the given scheme. This allows composition
+// of clientsets, like in:
+//
+//	import (
+//	  "k8s.io/client-go/kubernetes"
+//	  clientsetscheme "k8s.io/client-go/kubernetes/scheme"
+//	  aggregatorclientsetscheme "k8s.io/kube-aggregator/pkg/client/clientset_generated/clientset/scheme"
+//	)
+//
+//	kclientset, _ := kubernetes.NewForConfig(c)
+//	_ = aggregatorclientsetscheme.AddToScheme(clientsetscheme.Scheme)
+//
+// After this, RawExtensions in Kubernetes types will serialize kube-aggregator types
+// correctly.
+var AddToScheme = localSchemeBuilder.AddToScheme
+
+func init() {
+	v1.AddToGroupVersion(Scheme, schema.GroupVersion{Version: "v1"})
+	utilruntime.Must(AddToScheme(Scheme))
+}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/doc.go

@@ -0,0 +1,6 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+// This package has the automatically generated typed clients.
+package v1alpha1

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/fake/doc.go

@@ -0,0 +1,6 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+// Package fake has the automatically generated clients.
+package fake

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/fake/fake_monok8s_client.go

@@ -0,0 +1,30 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package fake
+
+import (
+	v1alpha1 "example.com/monok8s/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1"
+	rest "k8s.io/client-go/rest"
+	testing "k8s.io/client-go/testing"
+)
+
+type FakeMonok8sV1alpha1 struct {
+	*testing.Fake
+}
+
+func (c *FakeMonok8sV1alpha1) OSUpgrades(namespace string) v1alpha1.OSUpgradeInterface {
+	return newFakeOSUpgrades(c, namespace)
+}
+
+func (c *FakeMonok8sV1alpha1) OSUpgradeProgresses(namespace string) v1alpha1.OSUpgradeProgressInterface {
+	return newFakeOSUpgradeProgresses(c, namespace)
+}
+
+// RESTClient returns a RESTClient that is used to communicate
+// with API server by this client implementation.
+func (c *FakeMonok8sV1alpha1) RESTClient() rest.Interface {
+	var ret *rest.RESTClient
+	return ret
+}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/fake/fake_osupgrade.go

@@ -0,0 +1,36 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package fake
+
+import (
+	v1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	monok8sv1alpha1 "example.com/monok8s/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1"
+	gentype "k8s.io/client-go/gentype"
+)
+
+// fakeOSUpgrades implements OSUpgradeInterface
+type fakeOSUpgrades struct {
+	*gentype.FakeClientWithList[*v1alpha1.OSUpgrade, *v1alpha1.OSUpgradeList]
+	Fake *FakeMonok8sV1alpha1
+}
+
+func newFakeOSUpgrades(fake *FakeMonok8sV1alpha1, namespace string) monok8sv1alpha1.OSUpgradeInterface {
+	return &fakeOSUpgrades{
+		gentype.NewFakeClientWithList[*v1alpha1.OSUpgrade, *v1alpha1.OSUpgradeList](
+			fake.Fake,
+			namespace,
+			v1alpha1.SchemeGroupVersion.WithResource("osupgrades"),
+			v1alpha1.SchemeGroupVersion.WithKind("OSUpgrade"),
+			func() *v1alpha1.OSUpgrade { return &v1alpha1.OSUpgrade{} },
+			func() *v1alpha1.OSUpgradeList { return &v1alpha1.OSUpgradeList{} },
+			func(dst, src *v1alpha1.OSUpgradeList) { dst.ListMeta = src.ListMeta },
+			func(list *v1alpha1.OSUpgradeList) []*v1alpha1.OSUpgrade { return gentype.ToPointerSlice(list.Items) },
+			func(list *v1alpha1.OSUpgradeList, items []*v1alpha1.OSUpgrade) {
+				list.Items = gentype.FromPointerSlice(items)
+			},
+		),
+		fake,
+	}
+}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/fake/fake_osupgradeprogress.go

@@ -0,0 +1,38 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package fake
+
+import (
+	v1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	monok8sv1alpha1 "example.com/monok8s/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1"
+	gentype "k8s.io/client-go/gentype"
+)
+
+// fakeOSUpgradeProgresses implements OSUpgradeProgressInterface
+type fakeOSUpgradeProgresses struct {
+	*gentype.FakeClientWithList[*v1alpha1.OSUpgradeProgress, *v1alpha1.OSUpgradeProgressList]
+	Fake *FakeMonok8sV1alpha1
+}
+
+func newFakeOSUpgradeProgresses(fake *FakeMonok8sV1alpha1, namespace string) monok8sv1alpha1.OSUpgradeProgressInterface {
+	return &fakeOSUpgradeProgresses{
+		gentype.NewFakeClientWithList[*v1alpha1.OSUpgradeProgress, *v1alpha1.OSUpgradeProgressList](
+			fake.Fake,
+			namespace,
+			v1alpha1.SchemeGroupVersion.WithResource("osupgradeprogresses"),
+			v1alpha1.SchemeGroupVersion.WithKind("OSUpgradeProgress"),
+			func() *v1alpha1.OSUpgradeProgress { return &v1alpha1.OSUpgradeProgress{} },
+			func() *v1alpha1.OSUpgradeProgressList { return &v1alpha1.OSUpgradeProgressList{} },
+			func(dst, src *v1alpha1.OSUpgradeProgressList) { dst.ListMeta = src.ListMeta },
+			func(list *v1alpha1.OSUpgradeProgressList) []*v1alpha1.OSUpgradeProgress {
+				return gentype.ToPointerSlice(list.Items)
+			},
+			func(list *v1alpha1.OSUpgradeProgressList, items []*v1alpha1.OSUpgradeProgress) {
+				list.Items = gentype.FromPointerSlice(items)
+			},
+		),
+		fake,
+	}
+}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/generated_expansion.go

@@ -0,0 +1,9 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package v1alpha1
+
+type OSUpgradeExpansion interface{}
+
+type OSUpgradeProgressExpansion interface{}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/monok8s_client.go

@@ -0,0 +1,92 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	http "net/http"
+
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	scheme "example.com/monok8s/pkg/generated/clientset/versioned/scheme"
+	rest "k8s.io/client-go/rest"
+)
+
+type Monok8sV1alpha1Interface interface {
+	RESTClient() rest.Interface
+	OSUpgradesGetter
+	OSUpgradeProgressesGetter
+}
+
+// Monok8sV1alpha1Client is used to interact with features provided by the monok8s group.
+type Monok8sV1alpha1Client struct {
+	restClient rest.Interface
+}
+
+func (c *Monok8sV1alpha1Client) OSUpgrades(namespace string) OSUpgradeInterface {
+	return newOSUpgrades(c, namespace)
+}
+
+func (c *Monok8sV1alpha1Client) OSUpgradeProgresses(namespace string) OSUpgradeProgressInterface {
+	return newOSUpgradeProgresses(c, namespace)
+}
+
+// NewForConfig creates a new Monok8sV1alpha1Client for the given config.
+// NewForConfig is equivalent to NewForConfigAndClient(c, httpClient),
+// where httpClient was generated with rest.HTTPClientFor(c).
+func NewForConfig(c *rest.Config) (*Monok8sV1alpha1Client, error) {
+	config := *c
+	setConfigDefaults(&config)
+	httpClient, err := rest.HTTPClientFor(&config)
+	if err != nil {
+		return nil, err
+	}
+	return NewForConfigAndClient(&config, httpClient)
+}
+
+// NewForConfigAndClient creates a new Monok8sV1alpha1Client for the given config and http client.
+// Note the http client provided takes precedence over the configured transport values.
+func NewForConfigAndClient(c *rest.Config, h *http.Client) (*Monok8sV1alpha1Client, error) {
+	config := *c
+	setConfigDefaults(&config)
+	client, err := rest.RESTClientForConfigAndClient(&config, h)
+	if err != nil {
+		return nil, err
+	}
+	return &Monok8sV1alpha1Client{client}, nil
+}
+
+// NewForConfigOrDie creates a new Monok8sV1alpha1Client for the given config and
+// panics if there is an error in the config.
+func NewForConfigOrDie(c *rest.Config) *Monok8sV1alpha1Client {
+	client, err := NewForConfig(c)
+	if err != nil {
+		panic(err)
+	}
+	return client
+}
+
+// New creates a new Monok8sV1alpha1Client for the given RESTClient.
+func New(c rest.Interface) *Monok8sV1alpha1Client {
+	return &Monok8sV1alpha1Client{c}
+}
+
+func setConfigDefaults(config *rest.Config) {
+	gv := monok8sv1alpha1.SchemeGroupVersion
+	config.GroupVersion = &gv
+	config.APIPath = "/apis"
+	config.NegotiatedSerializer = rest.CodecFactoryForGeneratedClient(scheme.Scheme, scheme.Codecs).WithoutConversion()
+
+	if config.UserAgent == "" {
+		config.UserAgent = rest.DefaultKubernetesUserAgent()
+	}
+}
+
+// RESTClient returns a RESTClient that is used to communicate
+// with API server by this client implementation.
+func (c *Monok8sV1alpha1Client) RESTClient() rest.Interface {
+	if c == nil {
+		return nil
+	}
+	return c.restClient
+}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/osupgrade.go

@@ -0,0 +1,56 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	context "context"
+
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	scheme "example.com/monok8s/pkg/generated/clientset/versioned/scheme"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	types "k8s.io/apimachinery/pkg/types"
+	watch "k8s.io/apimachinery/pkg/watch"
+	gentype "k8s.io/client-go/gentype"
+)
+
+// OSUpgradesGetter has a method to return a OSUpgradeInterface.
+// A group's client should implement this interface.
+type OSUpgradesGetter interface {
+	OSUpgrades(namespace string) OSUpgradeInterface
+}
+
+// OSUpgradeInterface has methods to work with OSUpgrade resources.
+type OSUpgradeInterface interface {
+	Create(ctx context.Context, oSUpgrade *monok8sv1alpha1.OSUpgrade, opts v1.CreateOptions) (*monok8sv1alpha1.OSUpgrade, error)
+	Update(ctx context.Context, oSUpgrade *monok8sv1alpha1.OSUpgrade, opts v1.UpdateOptions) (*monok8sv1alpha1.OSUpgrade, error)
+	// Add a +genclient:noStatus comment above the type to avoid generating UpdateStatus().
+	UpdateStatus(ctx context.Context, oSUpgrade *monok8sv1alpha1.OSUpgrade, opts v1.UpdateOptions) (*monok8sv1alpha1.OSUpgrade, error)
+	Delete(ctx context.Context, name string, opts v1.DeleteOptions) error
+	DeleteCollection(ctx context.Context, opts v1.DeleteOptions, listOpts v1.ListOptions) error
+	Get(ctx context.Context, name string, opts v1.GetOptions) (*monok8sv1alpha1.OSUpgrade, error)
+	List(ctx context.Context, opts v1.ListOptions) (*monok8sv1alpha1.OSUpgradeList, error)
+	Watch(ctx context.Context, opts v1.ListOptions) (watch.Interface, error)
+	Patch(ctx context.Context, name string, pt types.PatchType, data []byte, opts v1.PatchOptions, subresources ...string) (result *monok8sv1alpha1.OSUpgrade, err error)
+	OSUpgradeExpansion
+}
+
+// oSUpgrades implements OSUpgradeInterface
+type oSUpgrades struct {
+	*gentype.ClientWithList[*monok8sv1alpha1.OSUpgrade, *monok8sv1alpha1.OSUpgradeList]
+}
+
+// newOSUpgrades returns a OSUpgrades
+func newOSUpgrades(c *Monok8sV1alpha1Client, namespace string) *oSUpgrades {
+	return &oSUpgrades{
+		gentype.NewClientWithList[*monok8sv1alpha1.OSUpgrade, *monok8sv1alpha1.OSUpgradeList](
+			"osupgrades",
+			c.RESTClient(),
+			scheme.ParameterCodec,
+			namespace,
+			func() *monok8sv1alpha1.OSUpgrade { return &monok8sv1alpha1.OSUpgrade{} },
+			func() *monok8sv1alpha1.OSUpgradeList { return &monok8sv1alpha1.OSUpgradeList{} },
+		),
+	}
+}

clitools/pkg/generated/clientset/versioned/typed/monok8s/v1alpha1/osupgradeprogress.go

@@ -0,0 +1,56 @@
+/* MIT License */
+
+// Code generated by client-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	context "context"
+
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	scheme "example.com/monok8s/pkg/generated/clientset/versioned/scheme"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	types "k8s.io/apimachinery/pkg/types"
+	watch "k8s.io/apimachinery/pkg/watch"
+	gentype "k8s.io/client-go/gentype"
+)
+
+// OSUpgradeProgressesGetter has a method to return a OSUpgradeProgressInterface.
+// A group's client should implement this interface.
+type OSUpgradeProgressesGetter interface {
+	OSUpgradeProgresses(namespace string) OSUpgradeProgressInterface
+}
+
+// OSUpgradeProgressInterface has methods to work with OSUpgradeProgress resources.
+type OSUpgradeProgressInterface interface {
+	Create(ctx context.Context, oSUpgradeProgress *monok8sv1alpha1.OSUpgradeProgress, opts v1.CreateOptions) (*monok8sv1alpha1.OSUpgradeProgress, error)
+	Update(ctx context.Context, oSUpgradeProgress *monok8sv1alpha1.OSUpgradeProgress, opts v1.UpdateOptions) (*monok8sv1alpha1.OSUpgradeProgress, error)
+	// Add a +genclient:noStatus comment above the type to avoid generating UpdateStatus().
+	UpdateStatus(ctx context.Context, oSUpgradeProgress *monok8sv1alpha1.OSUpgradeProgress, opts v1.UpdateOptions) (*monok8sv1alpha1.OSUpgradeProgress, error)
+	Delete(ctx context.Context, name string, opts v1.DeleteOptions) error
+	DeleteCollection(ctx context.Context, opts v1.DeleteOptions, listOpts v1.ListOptions) error
+	Get(ctx context.Context, name string, opts v1.GetOptions) (*monok8sv1alpha1.OSUpgradeProgress, error)
+	List(ctx context.Context, opts v1.ListOptions) (*monok8sv1alpha1.OSUpgradeProgressList, error)
+	Watch(ctx context.Context, opts v1.ListOptions) (watch.Interface, error)
+	Patch(ctx context.Context, name string, pt types.PatchType, data []byte, opts v1.PatchOptions, subresources ...string) (result *monok8sv1alpha1.OSUpgradeProgress, err error)
+	OSUpgradeProgressExpansion
+}
+
+// oSUpgradeProgresses implements OSUpgradeProgressInterface
+type oSUpgradeProgresses struct {
+	*gentype.ClientWithList[*monok8sv1alpha1.OSUpgradeProgress, *monok8sv1alpha1.OSUpgradeProgressList]
+}
+
+// newOSUpgradeProgresses returns a OSUpgradeProgresses
+func newOSUpgradeProgresses(c *Monok8sV1alpha1Client, namespace string) *oSUpgradeProgresses {
+	return &oSUpgradeProgresses{
+		gentype.NewClientWithList[*monok8sv1alpha1.OSUpgradeProgress, *monok8sv1alpha1.OSUpgradeProgressList](
+			"osupgradeprogresses",
+			c.RESTClient(),
+			scheme.ParameterCodec,
+			namespace,
+			func() *monok8sv1alpha1.OSUpgradeProgress { return &monok8sv1alpha1.OSUpgradeProgress{} },
+			func() *monok8sv1alpha1.OSUpgradeProgressList { return &monok8sv1alpha1.OSUpgradeProgressList{} },
+		),
+	}
+}

clitools/pkg/generated/informers/externalversions/factory.go

@@ -0,0 +1,249 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package externalversions
+
+import (
+	reflect "reflect"
+	sync "sync"
+	time "time"
+
+	versioned "example.com/monok8s/pkg/generated/clientset/versioned"
+	internalinterfaces "example.com/monok8s/pkg/generated/informers/externalversions/internalinterfaces"
+	monok8s "example.com/monok8s/pkg/generated/informers/externalversions/monok8s"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+	schema "k8s.io/apimachinery/pkg/runtime/schema"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// SharedInformerOption defines the functional option type for SharedInformerFactory.
+type SharedInformerOption func(*sharedInformerFactory) *sharedInformerFactory
+
+type sharedInformerFactory struct {
+	client           versioned.Interface
+	namespace        string
+	tweakListOptions internalinterfaces.TweakListOptionsFunc
+	lock             sync.Mutex
+	defaultResync    time.Duration
+	customResync     map[reflect.Type]time.Duration
+	transform        cache.TransformFunc
+
+	informers map[reflect.Type]cache.SharedIndexInformer
+	// startedInformers is used for tracking which informers have been started.
+	// This allows Start() to be called multiple times safely.
+	startedInformers map[reflect.Type]bool
+	// wg tracks how many goroutines were started.
+	wg sync.WaitGroup
+	// shuttingDown is true when Shutdown has been called. It may still be running
+	// because it needs to wait for goroutines.
+	shuttingDown bool
+}
+
+// WithCustomResyncConfig sets a custom resync period for the specified informer types.
+func WithCustomResyncConfig(resyncConfig map[v1.Object]time.Duration) SharedInformerOption {
+	return func(factory *sharedInformerFactory) *sharedInformerFactory {
+		for k, v := range resyncConfig {
+			factory.customResync[reflect.TypeOf(k)] = v
+		}
+		return factory
+	}
+}
+
+// WithTweakListOptions sets a custom filter on all listers of the configured SharedInformerFactory.
+func WithTweakListOptions(tweakListOptions internalinterfaces.TweakListOptionsFunc) SharedInformerOption {
+	return func(factory *sharedInformerFactory) *sharedInformerFactory {
+		factory.tweakListOptions = tweakListOptions
+		return factory
+	}
+}
+
+// WithNamespace limits the SharedInformerFactory to the specified namespace.
+func WithNamespace(namespace string) SharedInformerOption {
+	return func(factory *sharedInformerFactory) *sharedInformerFactory {
+		factory.namespace = namespace
+		return factory
+	}
+}
+
+// WithTransform sets a transform on all informers.
+func WithTransform(transform cache.TransformFunc) SharedInformerOption {
+	return func(factory *sharedInformerFactory) *sharedInformerFactory {
+		factory.transform = transform
+		return factory
+	}
+}
+
+// NewSharedInformerFactory constructs a new instance of sharedInformerFactory for all namespaces.
+func NewSharedInformerFactory(client versioned.Interface, defaultResync time.Duration) SharedInformerFactory {
+	return NewSharedInformerFactoryWithOptions(client, defaultResync)
+}
+
+// NewFilteredSharedInformerFactory constructs a new instance of sharedInformerFactory.
+// Listers obtained via this SharedInformerFactory will be subject to the same filters
+// as specified here.
+//
+// Deprecated: Please use NewSharedInformerFactoryWithOptions instead
+func NewFilteredSharedInformerFactory(client versioned.Interface, defaultResync time.Duration, namespace string, tweakListOptions internalinterfaces.TweakListOptionsFunc) SharedInformerFactory {
+	return NewSharedInformerFactoryWithOptions(client, defaultResync, WithNamespace(namespace), WithTweakListOptions(tweakListOptions))
+}
+
+// NewSharedInformerFactoryWithOptions constructs a new instance of a SharedInformerFactory with additional options.
+func NewSharedInformerFactoryWithOptions(client versioned.Interface, defaultResync time.Duration, options ...SharedInformerOption) SharedInformerFactory {
+	factory := &sharedInformerFactory{
+		client:           client,
+		namespace:        v1.NamespaceAll,
+		defaultResync:    defaultResync,
+		informers:        make(map[reflect.Type]cache.SharedIndexInformer),
+		startedInformers: make(map[reflect.Type]bool),
+		customResync:     make(map[reflect.Type]time.Duration),
+	}
+
+	// Apply all options
+	for _, opt := range options {
+		factory = opt(factory)
+	}
+
+	return factory
+}
+
+func (f *sharedInformerFactory) Start(stopCh <-chan struct{}) {
+	f.lock.Lock()
+	defer f.lock.Unlock()
+
+	if f.shuttingDown {
+		return
+	}
+
+	for informerType, informer := range f.informers {
+		if !f.startedInformers[informerType] {
+			f.wg.Add(1)
+			// We need a new variable in each loop iteration,
+			// otherwise the goroutine would use the loop variable
+			// and that keeps changing.
+			informer := informer
+			go func() {
+				defer f.wg.Done()
+				informer.Run(stopCh)
+			}()
+			f.startedInformers[informerType] = true
+		}
+	}
+}
+
+func (f *sharedInformerFactory) Shutdown() {
+	f.lock.Lock()
+	f.shuttingDown = true
+	f.lock.Unlock()
+
+	// Will return immediately if there is nothing to wait for.
+	f.wg.Wait()
+}
+
+func (f *sharedInformerFactory) WaitForCacheSync(stopCh <-chan struct{}) map[reflect.Type]bool {
+	informers := func() map[reflect.Type]cache.SharedIndexInformer {
+		f.lock.Lock()
+		defer f.lock.Unlock()
+
+		informers := map[reflect.Type]cache.SharedIndexInformer{}
+		for informerType, informer := range f.informers {
+			if f.startedInformers[informerType] {
+				informers[informerType] = informer
+			}
+		}
+		return informers
+	}()
+
+	res := map[reflect.Type]bool{}
+	for informType, informer := range informers {
+		res[informType] = cache.WaitForCacheSync(stopCh, informer.HasSynced)
+	}
+	return res
+}
+
+// InformerFor returns the SharedIndexInformer for obj using an internal
+// client.
+func (f *sharedInformerFactory) InformerFor(obj runtime.Object, newFunc internalinterfaces.NewInformerFunc) cache.SharedIndexInformer {
+	f.lock.Lock()
+	defer f.lock.Unlock()
+
+	informerType := reflect.TypeOf(obj)
+	informer, exists := f.informers[informerType]
+	if exists {
+		return informer
+	}
+
+	resyncPeriod, exists := f.customResync[informerType]
+	if !exists {
+		resyncPeriod = f.defaultResync
+	}
+
+	informer = newFunc(f.client, resyncPeriod)
+	informer.SetTransform(f.transform)
+	f.informers[informerType] = informer
+
+	return informer
+}
+
+// SharedInformerFactory provides shared informers for resources in all known
+// API group versions.
+//
+// It is typically used like this:
+//
+//	ctx, cancel := context.WithCancel(context.Background())
+//	defer cancel()
+//	factory := NewSharedInformerFactory(client, resyncPeriod)
+//	defer factory.WaitForStop()    // Returns immediately if nothing was started.
+//	genericInformer := factory.ForResource(resource)
+//	typedInformer := factory.SomeAPIGroup().V1().SomeType()
+//	factory.Start(ctx.Done())          // Start processing these informers.
+//	synced := factory.WaitForCacheSync(ctx.Done())
+//	for v, ok := range synced {
+//	    if !ok {
+//	        fmt.Fprintf(os.Stderr, "caches failed to sync: %v", v)
+//	        return
+//	    }
+//	}
+//
+//	// Creating informers can also be created after Start, but then
+//	// Start must be called again:
+//	anotherGenericInformer := factory.ForResource(resource)
+//	factory.Start(ctx.Done())
+type SharedInformerFactory interface {
+	internalinterfaces.SharedInformerFactory
+
+	// Start initializes all requested informers. They are handled in goroutines
+	// which run until the stop channel gets closed.
+	// Warning: Start does not block. When run in a go-routine, it will race with a later WaitForCacheSync.
+	Start(stopCh <-chan struct{})
+
+	// Shutdown marks a factory as shutting down. At that point no new
+	// informers can be started anymore and Start will return without
+	// doing anything.
+	//
+	// In addition, Shutdown blocks until all goroutines have terminated. For that
+	// to happen, the close channel(s) that they were started with must be closed,
+	// either before Shutdown gets called or while it is waiting.
+	//
+	// Shutdown may be called multiple times, even concurrently. All such calls will
+	// block until all goroutines have terminated.
+	Shutdown()
+
+	// WaitForCacheSync blocks until all started informers' caches were synced
+	// or the stop channel gets closed.
+	WaitForCacheSync(stopCh <-chan struct{}) map[reflect.Type]bool
+
+	// ForResource gives generic access to a shared informer of the matching type.
+	ForResource(resource schema.GroupVersionResource) (GenericInformer, error)
+
+	// InformerFor returns the SharedIndexInformer for obj using an internal
+	// client.
+	InformerFor(obj runtime.Object, newFunc internalinterfaces.NewInformerFunc) cache.SharedIndexInformer
+
+	Monok8s() monok8s.Interface
+}
+
+func (f *sharedInformerFactory) Monok8s() monok8s.Interface {
+	return monok8s.New(f, f.namespace, f.tweakListOptions)
+}

clitools/pkg/generated/informers/externalversions/generic.go

@@ -0,0 +1,50 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package externalversions
+
+import (
+	fmt "fmt"
+
+	v1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	schema "k8s.io/apimachinery/pkg/runtime/schema"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// GenericInformer is type of SharedIndexInformer which will locate and delegate to other
+// sharedInformers based on type
+type GenericInformer interface {
+	Informer() cache.SharedIndexInformer
+	Lister() cache.GenericLister
+}
+
+type genericInformer struct {
+	informer cache.SharedIndexInformer
+	resource schema.GroupResource
+}
+
+// Informer returns the SharedIndexInformer.
+func (f *genericInformer) Informer() cache.SharedIndexInformer {
+	return f.informer
+}
+
+// Lister returns the GenericLister.
+func (f *genericInformer) Lister() cache.GenericLister {
+	return cache.NewGenericLister(f.Informer().GetIndexer(), f.resource)
+}
+
+// ForResource gives generic access to a shared informer of the matching type
+// TODO extend this to unknown resources with a client pool
+func (f *sharedInformerFactory) ForResource(resource schema.GroupVersionResource) (GenericInformer, error) {
+	switch resource {
+	// Group=monok8s, Version=v1alpha1
+	case v1alpha1.SchemeGroupVersion.WithResource("osupgrades"):
+		return &genericInformer{resource: resource.GroupResource(), informer: f.Monok8s().V1alpha1().OSUpgrades().Informer()}, nil
+	case v1alpha1.SchemeGroupVersion.WithResource("osupgradeprogresses"):
+		return &genericInformer{resource: resource.GroupResource(), informer: f.Monok8s().V1alpha1().OSUpgradeProgresses().Informer()}, nil
+
+	}
+
+	return nil, fmt.Errorf("no informer found for %v", resource)
+}

clitools/pkg/generated/informers/externalversions/internalinterfaces/factory_interfaces.go

@@ -0,0 +1,26 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package internalinterfaces
+
+import (
+	time "time"
+
+	versioned "example.com/monok8s/pkg/generated/clientset/versioned"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// NewInformerFunc takes versioned.Interface and time.Duration to return a SharedIndexInformer.
+type NewInformerFunc func(versioned.Interface, time.Duration) cache.SharedIndexInformer
+
+// SharedInformerFactory a small interface to allow for adding an informer without an import cycle
+type SharedInformerFactory interface {
+	Start(stopCh <-chan struct{})
+	InformerFor(obj runtime.Object, newFunc NewInformerFunc) cache.SharedIndexInformer
+}
+
+// TweakListOptionsFunc is a function that transforms a v1.ListOptions.
+type TweakListOptionsFunc func(*v1.ListOptions)

clitools/pkg/generated/informers/externalversions/monok8s/interface.go

@@ -0,0 +1,32 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package monok8s
+
+import (
+	internalinterfaces "example.com/monok8s/pkg/generated/informers/externalversions/internalinterfaces"
+	v1alpha1 "example.com/monok8s/pkg/generated/informers/externalversions/monok8s/v1alpha1"
+)
+
+// Interface provides access to each of this group's versions.
+type Interface interface {
+	// V1alpha1 provides access to shared informers for resources in V1alpha1.
+	V1alpha1() v1alpha1.Interface
+}
+
+type group struct {
+	factory          internalinterfaces.SharedInformerFactory
+	namespace        string
+	tweakListOptions internalinterfaces.TweakListOptionsFunc
+}
+
+// New returns a new Interface.
+func New(f internalinterfaces.SharedInformerFactory, namespace string, tweakListOptions internalinterfaces.TweakListOptionsFunc) Interface {
+	return &group{factory: f, namespace: namespace, tweakListOptions: tweakListOptions}
+}
+
+// V1alpha1 returns a new v1alpha1.Interface.
+func (g *group) V1alpha1() v1alpha1.Interface {
+	return v1alpha1.New(g.factory, g.namespace, g.tweakListOptions)
+}

clitools/pkg/generated/informers/externalversions/monok8s/v1alpha1/interface.go

@@ -0,0 +1,38 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	internalinterfaces "example.com/monok8s/pkg/generated/informers/externalversions/internalinterfaces"
+)
+
+// Interface provides access to all the informers in this group version.
+type Interface interface {
+	// OSUpgrades returns a OSUpgradeInformer.
+	OSUpgrades() OSUpgradeInformer
+	// OSUpgradeProgresses returns a OSUpgradeProgressInformer.
+	OSUpgradeProgresses() OSUpgradeProgressInformer
+}
+
+type version struct {
+	factory          internalinterfaces.SharedInformerFactory
+	namespace        string
+	tweakListOptions internalinterfaces.TweakListOptionsFunc
+}
+
+// New returns a new Interface.
+func New(f internalinterfaces.SharedInformerFactory, namespace string, tweakListOptions internalinterfaces.TweakListOptionsFunc) Interface {
+	return &version{factory: f, namespace: namespace, tweakListOptions: tweakListOptions}
+}
+
+// OSUpgrades returns a OSUpgradeInformer.
+func (v *version) OSUpgrades() OSUpgradeInformer {
+	return &oSUpgradeInformer{factory: v.factory, namespace: v.namespace, tweakListOptions: v.tweakListOptions}
+}
+
+// OSUpgradeProgresses returns a OSUpgradeProgressInformer.
+func (v *version) OSUpgradeProgresses() OSUpgradeProgressInformer {
+	return &oSUpgradeProgressInformer{factory: v.factory, namespace: v.namespace, tweakListOptions: v.tweakListOptions}
+}

clitools/pkg/generated/informers/externalversions/monok8s/v1alpha1/osupgrade.go

@@ -0,0 +1,88 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	context "context"
+	time "time"
+
+	apismonok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	versioned "example.com/monok8s/pkg/generated/clientset/versioned"
+	internalinterfaces "example.com/monok8s/pkg/generated/informers/externalversions/internalinterfaces"
+	monok8sv1alpha1 "example.com/monok8s/pkg/generated/listers/monok8s/v1alpha1"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+	watch "k8s.io/apimachinery/pkg/watch"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// OSUpgradeInformer provides access to a shared informer and lister for
+// OSUpgrades.
+type OSUpgradeInformer interface {
+	Informer() cache.SharedIndexInformer
+	Lister() monok8sv1alpha1.OSUpgradeLister
+}
+
+type oSUpgradeInformer struct {
+	factory          internalinterfaces.SharedInformerFactory
+	tweakListOptions internalinterfaces.TweakListOptionsFunc
+	namespace        string
+}
+
+// NewOSUpgradeInformer constructs a new informer for OSUpgrade type.
+// Always prefer using an informer factory to get a shared informer instead of getting an independent
+// one. This reduces memory footprint and number of connections to the server.
+func NewOSUpgradeInformer(client versioned.Interface, namespace string, resyncPeriod time.Duration, indexers cache.Indexers) cache.SharedIndexInformer {
+	return NewFilteredOSUpgradeInformer(client, namespace, resyncPeriod, indexers, nil)
+}
+
+// NewFilteredOSUpgradeInformer constructs a new informer for OSUpgrade type.
+// Always prefer using an informer factory to get a shared informer instead of getting an independent
+// one. This reduces memory footprint and number of connections to the server.
+func NewFilteredOSUpgradeInformer(client versioned.Interface, namespace string, resyncPeriod time.Duration, indexers cache.Indexers, tweakListOptions internalinterfaces.TweakListOptionsFunc) cache.SharedIndexInformer {
+	return cache.NewSharedIndexInformer(
+		cache.ToListWatcherWithWatchListSemantics(&cache.ListWatch{
+			ListFunc: func(options v1.ListOptions) (runtime.Object, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgrades(namespace).List(context.Background(), options)
+			},
+			WatchFunc: func(options v1.ListOptions) (watch.Interface, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgrades(namespace).Watch(context.Background(), options)
+			},
+			ListWithContextFunc: func(ctx context.Context, options v1.ListOptions) (runtime.Object, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgrades(namespace).List(ctx, options)
+			},
+			WatchFuncWithContext: func(ctx context.Context, options v1.ListOptions) (watch.Interface, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgrades(namespace).Watch(ctx, options)
+			},
+		}, client),
+		&apismonok8sv1alpha1.OSUpgrade{},
+		resyncPeriod,
+		indexers,
+	)
+}
+
+func (f *oSUpgradeInformer) defaultInformer(client versioned.Interface, resyncPeriod time.Duration) cache.SharedIndexInformer {
+	return NewFilteredOSUpgradeInformer(client, f.namespace, resyncPeriod, cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc}, f.tweakListOptions)
+}
+
+func (f *oSUpgradeInformer) Informer() cache.SharedIndexInformer {
+	return f.factory.InformerFor(&apismonok8sv1alpha1.OSUpgrade{}, f.defaultInformer)
+}
+
+func (f *oSUpgradeInformer) Lister() monok8sv1alpha1.OSUpgradeLister {
+	return monok8sv1alpha1.NewOSUpgradeLister(f.Informer().GetIndexer())
+}

clitools/pkg/generated/informers/externalversions/monok8s/v1alpha1/osupgradeprogress.go

@@ -0,0 +1,88 @@
+/* MIT License */
+
+// Code generated by informer-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	context "context"
+	time "time"
+
+	apismonok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	versioned "example.com/monok8s/pkg/generated/clientset/versioned"
+	internalinterfaces "example.com/monok8s/pkg/generated/informers/externalversions/internalinterfaces"
+	monok8sv1alpha1 "example.com/monok8s/pkg/generated/listers/monok8s/v1alpha1"
+	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	runtime "k8s.io/apimachinery/pkg/runtime"
+	watch "k8s.io/apimachinery/pkg/watch"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// OSUpgradeProgressInformer provides access to a shared informer and lister for
+// OSUpgradeProgresses.
+type OSUpgradeProgressInformer interface {
+	Informer() cache.SharedIndexInformer
+	Lister() monok8sv1alpha1.OSUpgradeProgressLister
+}
+
+type oSUpgradeProgressInformer struct {
+	factory          internalinterfaces.SharedInformerFactory
+	tweakListOptions internalinterfaces.TweakListOptionsFunc
+	namespace        string
+}
+
+// NewOSUpgradeProgressInformer constructs a new informer for OSUpgradeProgress type.
+// Always prefer using an informer factory to get a shared informer instead of getting an independent
+// one. This reduces memory footprint and number of connections to the server.
+func NewOSUpgradeProgressInformer(client versioned.Interface, namespace string, resyncPeriod time.Duration, indexers cache.Indexers) cache.SharedIndexInformer {
+	return NewFilteredOSUpgradeProgressInformer(client, namespace, resyncPeriod, indexers, nil)
+}
+
+// NewFilteredOSUpgradeProgressInformer constructs a new informer for OSUpgradeProgress type.
+// Always prefer using an informer factory to get a shared informer instead of getting an independent
+// one. This reduces memory footprint and number of connections to the server.
+func NewFilteredOSUpgradeProgressInformer(client versioned.Interface, namespace string, resyncPeriod time.Duration, indexers cache.Indexers, tweakListOptions internalinterfaces.TweakListOptionsFunc) cache.SharedIndexInformer {
+	return cache.NewSharedIndexInformer(
+		cache.ToListWatcherWithWatchListSemantics(&cache.ListWatch{
+			ListFunc: func(options v1.ListOptions) (runtime.Object, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgradeProgresses(namespace).List(context.Background(), options)
+			},
+			WatchFunc: func(options v1.ListOptions) (watch.Interface, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgradeProgresses(namespace).Watch(context.Background(), options)
+			},
+			ListWithContextFunc: func(ctx context.Context, options v1.ListOptions) (runtime.Object, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgradeProgresses(namespace).List(ctx, options)
+			},
+			WatchFuncWithContext: func(ctx context.Context, options v1.ListOptions) (watch.Interface, error) {
+				if tweakListOptions != nil {
+					tweakListOptions(&options)
+				}
+				return client.Monok8sV1alpha1().OSUpgradeProgresses(namespace).Watch(ctx, options)
+			},
+		}, client),
+		&apismonok8sv1alpha1.OSUpgradeProgress{},
+		resyncPeriod,
+		indexers,
+	)
+}
+
+func (f *oSUpgradeProgressInformer) defaultInformer(client versioned.Interface, resyncPeriod time.Duration) cache.SharedIndexInformer {
+	return NewFilteredOSUpgradeProgressInformer(client, f.namespace, resyncPeriod, cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc}, f.tweakListOptions)
+}
+
+func (f *oSUpgradeProgressInformer) Informer() cache.SharedIndexInformer {
+	return f.factory.InformerFor(&apismonok8sv1alpha1.OSUpgradeProgress{}, f.defaultInformer)
+}
+
+func (f *oSUpgradeProgressInformer) Lister() monok8sv1alpha1.OSUpgradeProgressLister {
+	return monok8sv1alpha1.NewOSUpgradeProgressLister(f.Informer().GetIndexer())
+}

clitools/pkg/generated/listers/monok8s/v1alpha1/expansion_generated.go

@@ -0,0 +1,21 @@
+/* MIT License */
+
+// Code generated by lister-gen. DO NOT EDIT.
+
+package v1alpha1
+
+// OSUpgradeListerExpansion allows custom methods to be added to
+// OSUpgradeLister.
+type OSUpgradeListerExpansion interface{}
+
+// OSUpgradeNamespaceListerExpansion allows custom methods to be added to
+// OSUpgradeNamespaceLister.
+type OSUpgradeNamespaceListerExpansion interface{}
+
+// OSUpgradeProgressListerExpansion allows custom methods to be added to
+// OSUpgradeProgressLister.
+type OSUpgradeProgressListerExpansion interface{}
+
+// OSUpgradeProgressNamespaceListerExpansion allows custom methods to be added to
+// OSUpgradeProgressNamespaceLister.
+type OSUpgradeProgressNamespaceListerExpansion interface{}

clitools/pkg/generated/listers/monok8s/v1alpha1/osupgrade.go

@@ -0,0 +1,56 @@
+/* MIT License */
+
+// Code generated by lister-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	labels "k8s.io/apimachinery/pkg/labels"
+	listers "k8s.io/client-go/listers"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// OSUpgradeLister helps list OSUpgrades.
+// All objects returned here must be treated as read-only.
+type OSUpgradeLister interface {
+	// List lists all OSUpgrades in the indexer.
+	// Objects returned here must be treated as read-only.
+	List(selector labels.Selector) (ret []*monok8sv1alpha1.OSUpgrade, err error)
+	// OSUpgrades returns an object that can list and get OSUpgrades.
+	OSUpgrades(namespace string) OSUpgradeNamespaceLister
+	OSUpgradeListerExpansion
+}
+
+// oSUpgradeLister implements the OSUpgradeLister interface.
+type oSUpgradeLister struct {
+	listers.ResourceIndexer[*monok8sv1alpha1.OSUpgrade]
+}
+
+// NewOSUpgradeLister returns a new OSUpgradeLister.
+func NewOSUpgradeLister(indexer cache.Indexer) OSUpgradeLister {
+	return &oSUpgradeLister{listers.New[*monok8sv1alpha1.OSUpgrade](indexer, monok8sv1alpha1.Resource("osupgrade"))}
+}
+
+// OSUpgrades returns an object that can list and get OSUpgrades.
+func (s *oSUpgradeLister) OSUpgrades(namespace string) OSUpgradeNamespaceLister {
+	return oSUpgradeNamespaceLister{listers.NewNamespaced[*monok8sv1alpha1.OSUpgrade](s.ResourceIndexer, namespace)}
+}
+
+// OSUpgradeNamespaceLister helps list and get OSUpgrades.
+// All objects returned here must be treated as read-only.
+type OSUpgradeNamespaceLister interface {
+	// List lists all OSUpgrades in the indexer for a given namespace.
+	// Objects returned here must be treated as read-only.
+	List(selector labels.Selector) (ret []*monok8sv1alpha1.OSUpgrade, err error)
+	// Get retrieves the OSUpgrade from the indexer for a given namespace and name.
+	// Objects returned here must be treated as read-only.
+	Get(name string) (*monok8sv1alpha1.OSUpgrade, error)
+	OSUpgradeNamespaceListerExpansion
+}
+
+// oSUpgradeNamespaceLister implements the OSUpgradeNamespaceLister
+// interface.
+type oSUpgradeNamespaceLister struct {
+	listers.ResourceIndexer[*monok8sv1alpha1.OSUpgrade]
+}

clitools/pkg/generated/listers/monok8s/v1alpha1/osupgradeprogress.go

@@ -0,0 +1,56 @@
+/* MIT License */
+
+// Code generated by lister-gen. DO NOT EDIT.
+
+package v1alpha1
+
+import (
+	monok8sv1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	labels "k8s.io/apimachinery/pkg/labels"
+	listers "k8s.io/client-go/listers"
+	cache "k8s.io/client-go/tools/cache"
+)
+
+// OSUpgradeProgressLister helps list OSUpgradeProgresses.
+// All objects returned here must be treated as read-only.
+type OSUpgradeProgressLister interface {
+	// List lists all OSUpgradeProgresses in the indexer.
+	// Objects returned here must be treated as read-only.
+	List(selector labels.Selector) (ret []*monok8sv1alpha1.OSUpgradeProgress, err error)
+	// OSUpgradeProgresses returns an object that can list and get OSUpgradeProgresses.
+	OSUpgradeProgresses(namespace string) OSUpgradeProgressNamespaceLister
+	OSUpgradeProgressListerExpansion
+}
+
+// oSUpgradeProgressLister implements the OSUpgradeProgressLister interface.
+type oSUpgradeProgressLister struct {
+	listers.ResourceIndexer[*monok8sv1alpha1.OSUpgradeProgress]
+}
+
+// NewOSUpgradeProgressLister returns a new OSUpgradeProgressLister.
+func NewOSUpgradeProgressLister(indexer cache.Indexer) OSUpgradeProgressLister {
+	return &oSUpgradeProgressLister{listers.New[*monok8sv1alpha1.OSUpgradeProgress](indexer, monok8sv1alpha1.Resource("osupgradeprogress"))}
+}
+
+// OSUpgradeProgresses returns an object that can list and get OSUpgradeProgresses.
+func (s *oSUpgradeProgressLister) OSUpgradeProgresses(namespace string) OSUpgradeProgressNamespaceLister {
+	return oSUpgradeProgressNamespaceLister{listers.NewNamespaced[*monok8sv1alpha1.OSUpgradeProgress](s.ResourceIndexer, namespace)}
+}
+
+// OSUpgradeProgressNamespaceLister helps list and get OSUpgradeProgresses.
+// All objects returned here must be treated as read-only.
+type OSUpgradeProgressNamespaceLister interface {
+	// List lists all OSUpgradeProgresses in the indexer for a given namespace.
+	// Objects returned here must be treated as read-only.
+	List(selector labels.Selector) (ret []*monok8sv1alpha1.OSUpgradeProgress, err error)
+	// Get retrieves the OSUpgradeProgress from the indexer for a given namespace and name.
+	// Objects returned here must be treated as read-only.
+	Get(name string) (*monok8sv1alpha1.OSUpgradeProgress, error)
+	OSUpgradeProgressNamespaceListerExpansion
+}
+
+// oSUpgradeProgressNamespaceLister implements the OSUpgradeProgressNamespaceLister
+// interface.
+type oSUpgradeProgressNamespaceLister struct {
+	listers.ResourceIndexer[*monok8sv1alpha1.OSUpgradeProgress]
+}

clitools/pkg/kube/clients.go

@@ -3,11 +3,12 @@ package kube
 import (
 	"fmt"
 
+	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	monoclientset "example.com/monok8s/pkg/generated/clientset/versioned"
 	apiextensionsclientset "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset"
 	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
 	kubernetes "k8s.io/client-go/kubernetes"
 	clientgoscheme "k8s.io/client-go/kubernetes/scheme"
-	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
 
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/cli-runtime/pkg/genericclioptions"
@@ -21,6 +22,7 @@ type Clients struct {
 	Dynamic          dynamic.Interface
 	APIExtensions    apiextensionsclientset.Interface
 	RESTClientGetter genericclioptions.RESTClientGetter
+	MonoKS           monoclientset.Interface
 }
 
 func NewClients(flags *genericclioptions.ConfigFlags) (*Clients, error) {
@@ -40,7 +42,19 @@ func NewClients(flags *genericclioptions.ConfigFlags) (*Clients, error) {
 	if err != nil {
 		return nil, fmt.Errorf("build apiextensions client: %w", err)
 	}
-	return &Clients{Config: cfg, Kubernetes: kubeClient, Dynamic: dyn, APIExtensions: ext, RESTClientGetter: flags}, nil
+	mono, err := monoclientset.NewForConfig(cfg)
+	if err != nil {
+		return nil, fmt.Errorf("build monok8s client: %w", err)
+	}
+
+	return &Clients{
+		Config:           cfg,
+		Kubernetes:       kubeClient,
+		Dynamic:          dyn,
+		APIExtensions:    ext,
+		RESTClientGetter: flags,
+		MonoKS:           mono,
+	}, nil
 }
 
 func NewClientsFromKubeconfig(kubeconfigPath string) (*Clients, error) {

clitools/pkg/node/agent.go

@@ -3,39 +3,27 @@ package node
 import (
 	"context"
 	"fmt"
-	"reflect"
 	"strings"
 
-	appsv1 "k8s.io/api/apps/v1"
-	corev1 "k8s.io/api/core/v1"
-	rbacv1 "k8s.io/api/rbac/v1"
-	apierrors "k8s.io/apimachinery/pkg/api/errors"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/client-go/kubernetes"
 	"k8s.io/klog/v2"
 
-	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
 	"example.com/monok8s/pkg/kube"
+	"example.com/monok8s/pkg/render"
 	templates "example.com/monok8s/pkg/templates"
 )
 
-const (
-	controlAgentName              = "control-agent"
-	controlAgentNodeSelectorValue = "true"
-	controlAgentImage             = "localhost/monok8s/control-agent:dev"
-	kubeconfig                    = "/etc/kubernetes/admin.conf"
-)
+const kubeconfig = "/etc/kubernetes/admin.conf"
 
-func ApplyControlAgentDaemonSetResources(ctx context.Context, n *NodeContext) error {
-	// Only the control-plane should bootstrap this DaemonSet definition.
-	// And only when the feature is enabled.
-	if strings.TrimSpace(n.Config.Spec.ClusterRole) != "control-plane" || !n.Config.Spec.EnableControlAgent {
-		klog.InfoS("skipped for", "clusterRole", n.Config.Spec.ClusterRole, "enableControlAgent", n.Config.Spec.EnableControlAgent)
+func ApplyNodeControlDaemonSetResources(ctx context.Context, n *NodeContext) error {
+	if strings.TrimSpace(n.Config.Spec.ClusterRole) != "control-plane" || !n.Config.Spec.EnableNodeControl {
+		klog.InfoS("skipped for",
+			"clusterRole", n.Config.Spec.ClusterRole,
+			"enableNodeAgent", n.Config.Spec.EnableNodeControl,
+		)
 		return nil
 	}
 
-	err := ApplyCRDs(ctx, n)
-	if err != nil {
+	if err := ApplyCRDs(ctx, n); err != nil {
 		return err
 	}
 
@@ -49,330 +37,13 @@ func ApplyControlAgentDaemonSetResources(ctx context.Context, n *NodeContext) er
 		return fmt.Errorf("build kube clients from %s: %w", kubeconfig, err)
 	}
 
-	labels := map[string]string{
-		"app.kubernetes.io/name":       controlAgentName,
-		"app.kubernetes.io/component":  "agent",
-		"app.kubernetes.io/part-of":    "monok8s",
-		"app.kubernetes.io/managed-by": "ctl",
+	conf := render.AgentConf{
+		Namespace: namespace,
 	}
 
-	kubeClient := clients.Kubernetes
-
-	if err := applyControlAgentServiceAccount(ctx, kubeClient, namespace, labels); err != nil {
-		return fmt.Errorf("apply serviceaccount: %w", err)
-	}
-	if err := applyControlAgentClusterRole(ctx, kubeClient, labels); err != nil {
-		return fmt.Errorf("apply clusterrole: %w", err)
-	}
-	if err := applyControlAgentClusterRoleBinding(ctx, kubeClient, namespace, labels); err != nil {
-		return fmt.Errorf("apply clusterrolebinding: %w", err)
-	}
-	if err := applyControlAgentDaemonSet(ctx, kubeClient, namespace, labels); err != nil {
-		return fmt.Errorf("apply daemonset: %w", err)
+	if err := render.ApplyAgentDaemonSets(ctx, clients.Kubernetes, conf); err != nil {
+		return fmt.Errorf("apply node agent daemonset resources: %w", err)
 	}
 
 	return nil
 }
-
-func applyControlAgentServiceAccount(ctx context.Context, kubeClient kubernetes.Interface, namespace string, labels map[string]string) error {
-	want := &corev1.ServiceAccount{
-		ObjectMeta: metav1.ObjectMeta{
-			Name:      controlAgentName,
-			Namespace: namespace,
-			Labels:    labels,
-		},
-	}
-
-	existing, err := kubeClient.CoreV1().ServiceAccounts(namespace).Get(ctx, controlAgentName, metav1.GetOptions{})
-	if apierrors.IsNotFound(err) {
-		_, err = kubeClient.CoreV1().ServiceAccounts(namespace).Create(ctx, want, metav1.CreateOptions{})
-		return err
-	}
-	if err != nil {
-		return err
-	}
-
-	changed := false
-	if !reflect.DeepEqual(existing.Labels, want.Labels) {
-		existing.Labels = want.Labels
-		changed = true
-	}
-
-	if !changed {
-		return nil
-	}
-
-	_, err = kubeClient.CoreV1().ServiceAccounts(namespace).Update(ctx, existing, metav1.UpdateOptions{})
-	return err
-}
-
-func applyControlAgentClusterRole(ctx context.Context, kubeClient kubernetes.Interface, labels map[string]string) error {
-	wantRules := []rbacv1.PolicyRule{
-		{
-			APIGroups: []string{monov1alpha1.Group},
-			Resources: []string{"osupgrades"},
-			Verbs:     []string{"get", "list", "watch"},
-		},
-		{
-			APIGroups: []string{monov1alpha1.Group},
-			Resources: []string{"osupgrades/status"},
-			Verbs:     []string{"get", "patch", "update"},
-		},
-		{
-			APIGroups: []string{monov1alpha1.Group},
-			Resources: []string{"osupgradeprogresses"},
-			Verbs:     []string{"get", "list", "watch", "create", "patch", "update"},
-		},
-		{
-			APIGroups: []string{monov1alpha1.Group},
-			Resources: []string{"osupgradeprogresses/status"},
-			Verbs:     []string{"get", "list", "watch", "create", "patch", "update"},
-		},
-		{
-			APIGroups: []string{""},
-			Resources: []string{"nodes"},
-			Verbs:     []string{"get", "list", "watch"},
-		},
-	}
-
-	want := &rbacv1.ClusterRole{
-		ObjectMeta: metav1.ObjectMeta{
-			Name:   controlAgentName,
-			Labels: labels,
-		},
-		Rules: wantRules,
-	}
-
-	existing, err := kubeClient.RbacV1().ClusterRoles().Get(ctx, controlAgentName, metav1.GetOptions{})
-	if apierrors.IsNotFound(err) {
-		_, err = kubeClient.RbacV1().ClusterRoles().Create(ctx, want, metav1.CreateOptions{})
-		return err
-	}
-	if err != nil {
-		return err
-	}
-
-	changed := false
-	if !reflect.DeepEqual(existing.Labels, want.Labels) {
-		existing.Labels = want.Labels
-		changed = true
-	}
-	if !reflect.DeepEqual(existing.Rules, want.Rules) {
-		existing.Rules = want.Rules
-		changed = true
-	}
-
-	if !changed {
-		return nil
-	}
-
-	_, err = kubeClient.RbacV1().ClusterRoles().Update(ctx, existing, metav1.UpdateOptions{})
-	return err
-}
-
-func applyControlAgentClusterRoleBinding(ctx context.Context, kubeClient kubernetes.Interface, namespace string, labels map[string]string) error {
-	wantRoleRef := rbacv1.RoleRef{
-		APIGroup: rbacv1.GroupName,
-		Kind:     "ClusterRole",
-		Name:     controlAgentName,
-	}
-	wantSubjects := []rbacv1.Subject{
-		{
-			Kind:      "ServiceAccount",
-			Name:      controlAgentName,
-			Namespace: namespace,
-		},
-	}
-
-	want := &rbacv1.ClusterRoleBinding{
-		ObjectMeta: metav1.ObjectMeta{
-			Name:   controlAgentName,
-			Labels: labels,
-		},
-		RoleRef:  wantRoleRef,
-		Subjects: wantSubjects,
-	}
-
-	existing, err := kubeClient.RbacV1().ClusterRoleBindings().Get(ctx, controlAgentName, metav1.GetOptions{})
-	if apierrors.IsNotFound(err) {
-		_, err = kubeClient.RbacV1().ClusterRoleBindings().Create(ctx, want, metav1.CreateOptions{})
-		return err
-	}
-	if err != nil {
-		return err
-	}
-
-	// roleRef is immutable. If it differs, fail loudly instead of pretending we can patch it.
-	if !reflect.DeepEqual(existing.RoleRef, want.RoleRef) {
-		return fmt.Errorf("existing ClusterRoleBinding %q has different roleRef and must be recreated", controlAgentName)
-	}
-
-	changed := false
-	if !reflect.DeepEqual(existing.Labels, want.Labels) {
-		existing.Labels = want.Labels
-		changed = true
-	}
-	if !reflect.DeepEqual(existing.Subjects, want.Subjects) {
-		existing.Subjects = want.Subjects
-		changed = true
-	}
-
-	if !changed {
-		return nil
-	}
-
-	_, err = kubeClient.RbacV1().ClusterRoleBindings().Update(ctx, existing, metav1.UpdateOptions{})
-	return err
-}
-
-func applyControlAgentDaemonSet(ctx context.Context, kubeClient kubernetes.Interface, namespace string, labels map[string]string) error {
-	privileged := true
-
-	dsLabels := map[string]string{
-		"app.kubernetes.io/name":       controlAgentName,
-		"app.kubernetes.io/component":  "agent",
-		"app.kubernetes.io/part-of":    "monok8s",
-		"app.kubernetes.io/managed-by": "ctl",
-	}
-
-	want := &appsv1.DaemonSet{
-		ObjectMeta: metav1.ObjectMeta{
-			Name:      controlAgentName,
-			Namespace: namespace,
-			Labels:    labels,
-		},
-		Spec: appsv1.DaemonSetSpec{
-			Selector: &metav1.LabelSelector{
-				MatchLabels: map[string]string{
-					"app.kubernetes.io/name": controlAgentName,
-				},
-			},
-			Template: corev1.PodTemplateSpec{
-				ObjectMeta: metav1.ObjectMeta{
-					Labels: dsLabels,
-				},
-				Spec: corev1.PodSpec{
-					ServiceAccountName: controlAgentName,
-					HostNetwork:        true,
-					HostPID:            true,
-					DNSPolicy:          corev1.DNSClusterFirstWithHostNet,
-					NodeSelector: map[string]string{
-						monov1alpha1.ControlAgentKey: controlAgentNodeSelectorValue,
-					},
-					Tolerations: []corev1.Toleration{
-						{Operator: corev1.TolerationOpExists},
-					},
-					Containers: []corev1.Container{
-						{
-							Name:            "agent",
-							Image:           controlAgentImage,
-							ImagePullPolicy: corev1.PullNever,
-							Args:            []string{"agent", "--env-file", "$(CLUSTER_ENV_FILE)"},
-							Env: []corev1.EnvVar{
-								{
-									Name: "NODE_NAME",
-									ValueFrom: &corev1.EnvVarSource{
-										FieldRef: &corev1.ObjectFieldSelector{
-											APIVersion: "v1",
-											FieldPath:  "spec.nodeName",
-										},
-									},
-								},
-								{
-									Name:  "CLUSTER_ENV_FILE",
-									Value: "/host/opt/monok8s/config/cluster.env",
-								},
-								{
-									Name:  "FW_ENV_CONFIG_FILE",
-									Value: "/host/etc/fw_env.config",
-								},
-							},
-							SecurityContext: &corev1.SecurityContext{
-								Privileged: &privileged,
-							},
-							VolumeMounts: []corev1.VolumeMount{
-								{
-									Name:      "host-dev",
-									MountPath: "/dev",
-								},
-								{
-									Name:      "host-etc",
-									MountPath: "/host/etc",
-									ReadOnly:  true,
-								},
-								{
-									Name:      "host-config",
-									MountPath: "/host/opt/monok8s/config",
-									ReadOnly:  true,
-								},
-							},
-						},
-					},
-					Volumes: []corev1.Volume{
-						{
-							Name: "host-dev",
-							VolumeSource: corev1.VolumeSource{
-								HostPath: &corev1.HostPathVolumeSource{
-									Path: "/dev",
-									Type: hostPathType(corev1.HostPathDirectory),
-								},
-							},
-						},
-						{
-							Name: "host-etc",
-							VolumeSource: corev1.VolumeSource{
-								HostPath: &corev1.HostPathVolumeSource{
-									Path: "/etc",
-									Type: hostPathType(corev1.HostPathDirectory),
-								},
-							},
-						},
-						{
-							Name: "host-config",
-							VolumeSource: corev1.VolumeSource{
-								HostPath: &corev1.HostPathVolumeSource{
-									Path: "/opt/monok8s/config",
-									Type: hostPathType(corev1.HostPathDirectory),
-								},
-							},
-						},
-					},
-				},
-			},
-		},
-	}
-
-	existing, err := kubeClient.AppsV1().DaemonSets(namespace).Get(ctx, controlAgentName, metav1.GetOptions{})
-	if apierrors.IsNotFound(err) {
-		_, err = kubeClient.AppsV1().DaemonSets(namespace).Create(ctx, want, metav1.CreateOptions{})
-		return err
-	}
-	if err != nil {
-		return err
-	}
-
-	changed := false
-	if !reflect.DeepEqual(existing.Labels, want.Labels) {
-		existing.Labels = want.Labels
-		changed = true
-	}
-	if !reflect.DeepEqual(existing.Spec, want.Spec) {
-		existing.Spec = want.Spec
-		changed = true
-	}
-
-	if !changed {
-		return nil
-	}
-
-	_, err = kubeClient.AppsV1().DaemonSets(namespace).Update(ctx, existing, metav1.UpdateOptions{})
-	return err
-}
-
-func hostPathType(t corev1.HostPathType) *corev1.HostPathType {
-	return &t
-}
-
-func mountPropagationMode(m corev1.MountPropagationMode) *corev1.MountPropagationMode {
-	return &m
-}

clitools/pkg/node/fs.go

@@ -6,6 +6,8 @@ import (
 	"os"
 	"path/filepath"
 
+	"k8s.io/klog/v2"
+
 	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
 	"example.com/monok8s/pkg/controller/osimage"
 )
@@ -26,7 +28,34 @@ func ReleaseControlGate(ctx context.Context, nctx *NodeContext) error {
 	gateFile := filepath.Join(monov1alpha1.EnvConfigDir, ".control-gate")
 
 	if err := os.Remove(gateFile); err != nil {
-		return fmt.Errorf("relate control gate: %w", err)
+		return fmt.Errorf("release control gate: %w", err)
+	}
+
+	return WriteLastState(ctx, nctx)
+}
+
+// Required for detecting bootslot changes
+func WriteLastState(ctx context.Context, nctx *NodeContext) error {
+	stBootPart := filepath.Join(monov1alpha1.EnvConfigDir, ".bootpart")
+
+	state, err := osimage.ReadBootState()
+	if err != nil {
+		return fmt.Errorf("read boot state: %w", err)
+	}
+
+	bootPart := state["BOOT_PART"]
+	if bootPart == "" {
+		return fmt.Errorf("BOOT_PART missing")
+	}
+
+	klog.Infof("Writing last state: %+v", bootPart)
+
+	tmp := stBootPart + ".tmp"
+	if err := os.WriteFile(tmp, []byte(bootPart+"\n"), 0o644); err != nil {
+		return err
+	}
+	if err := os.Rename(tmp, stBootPart); err != nil {
+		return err
 	}
 
 	return nil

clitools/pkg/node/kubeadm.go

@@ -7,13 +7,11 @@ import (
 	"fmt"
 	"net"
 	"os"
+	"sort"
 	"strings"
 	"time"
 
 	"gopkg.in/yaml.v3"
-	"k8s.io/client-go/discovery"
-	"k8s.io/client-go/kubernetes"
-	"k8s.io/client-go/tools/clientcmd"
 	"k8s.io/klog/v2"
 
 	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
@@ -26,6 +24,16 @@ const (
 	tmpKubeadmInitConf    = "/tmp/kubeadm-init.yaml"
 )
 
+func chooseVersionKubeconfig(state *LocalClusterState) string {
+	if state.HasAdminKubeconfig {
+		return adminKubeconfigPath
+	}
+	if state.HasKubeletKubeconfig {
+		return kubeletKubeconfigPath
+	}
+	return ""
+}
+
 func DetectLocalClusterState(ctx context.Context, nctx *NodeContext) error {
 	_ = ctx
 
@@ -258,110 +266,6 @@ func waitForAPIViaKubeconfig(ctx context.Context, kubeconfigPath string, timeout
 	}
 }
 
-func getServerVersion(ctx context.Context, kubeconfigPath string) (string, error) {
-	restCfg, err := clientcmd.BuildConfigFromFlags("", kubeconfigPath)
-	if err != nil {
-		return "", fmt.Errorf("build kubeconfig %s: %w", kubeconfigPath, err)
-	}
-
-	// Keep this short. This is a probe, not a long-running client.
-	restCfg.Timeout = 5 * time.Second
-
-	clientset, err := kubernetes.NewForConfig(restCfg)
-	if err != nil {
-		return "", fmt.Errorf("create clientset: %w", err)
-	}
-
-	disc := clientset.Discovery()
-	return discoverServerVersion(ctx, disc)
-}
-
-func discoverServerVersion(ctx context.Context, disc discovery.DiscoveryInterface) (string, error) {
-	info, err := disc.ServerVersion()
-	if err != nil {
-		return "", err
-	}
-	if info == nil || strings.TrimSpace(info.GitVersion) == "" {
-		return "", errors.New("server version is empty")
-	}
-	return normalizeKubeVersion(info.GitVersion), nil
-}
-
-type kubeVersion struct {
-	Major int
-	Minor int
-	Patch int
-}
-
-func parseKubeVersion(s string) (kubeVersion, error) {
-	s = strings.TrimSpace(s)
-	s = strings.TrimPrefix(s, "v")
-
-	var v kubeVersion
-	n, err := fmt.Sscanf(s, "%d.%d.%d", &v.Major, &v.Minor, &v.Patch)
-	// Accepts "1.29" or "1.29.3"
-	if err != nil || n < 2 {
-		return kubeVersion{}, fmt.Errorf("invalid kubernetes version %q", s)
-	}
-	return v, nil
-}
-
-// Control-plane: keep this strict.
-// Accept same version, or a one-minor step where the node binary is newer than the current cluster.
-// That covers normal control-plane upgrade flow but blocks nonsense.
-func isSupportedControlPlaneSkew(clusterVersion, nodeVersion string) bool {
-	cv, err := parseKubeVersion(clusterVersion)
-	if err != nil {
-		return false
-	}
-	nv, err := parseKubeVersion(nodeVersion)
-	if err != nil {
-		return false
-	}
-
-	if cv.Major != nv.Major {
-		return false
-	}
-	if cv.Minor == nv.Minor {
-		return true
-	}
-	if nv.Minor == cv.Minor+1 {
-		return true
-	}
-	return false
-}
-
-// Worker: kubelet generally must not be newer than the apiserver.
-// Older kubelets are allowed within supported skew range.
-// Your requirement says unsupported worker skew should still proceed, so this
-// only classifies support status and must NOT be used to block this function.
-func isSupportedWorkerSkew(clusterVersion, nodeVersion string) bool {
-	cv, err := parseKubeVersion(clusterVersion)
-	if err != nil {
-		return false
-	}
-	nv, err := parseKubeVersion(nodeVersion)
-	if err != nil {
-		return false
-	}
-
-	if cv.Major != nv.Major {
-		return false
-	}
-
-	// kubelet newer than apiserver => unsupported
-	if nv.Minor > cv.Minor {
-		return false
-	}
-
-	// kubelet up to 3 minors older than apiserver => supported
-	if cv.Minor-nv.Minor <= 3 {
-		return true
-	}
-
-	return false
-}
-
 func ValidateRequiredImagesPresent(ctx context.Context, n *NodeContext) error {
 	if n.Config.Spec.SkipImageCheck {
 		klog.Infof("skipping image check (skipImageCheck=true)")
@@ -409,8 +313,8 @@ func checkImagePresent(ctx context.Context, n *NodeContext, image string) error
 
 	// crictl inspecti exits non-zero when the image is absent.
 	_, err := n.SystemRunner.RunRetry(ctx, system.RetryOptions{
-		Attempts: 3,
-		Delay:    1 * system.DefaultSecond,
+		Attempts: 6,
+		Delay:    2 * system.DefaultSecond,
 	}, "crictl", "inspecti", image)
 	if err != nil {
 		return fmt.Errorf("image %q not present: %w", image, err)
@@ -418,31 +322,6 @@ func checkImagePresent(ctx context.Context, n *NodeContext, image string) error
 	return nil
 }
 
-func chooseVersionKubeconfig(state *LocalClusterState) string {
-	if state.HasAdminKubeconfig {
-		return adminKubeconfigPath
-	}
-	if state.HasKubeletKubeconfig {
-		return kubeletKubeconfigPath
-	}
-	return ""
-}
-
-func versionEq(a, b string) bool {
-	return normalizeKubeVersion(a) == normalizeKubeVersion(b)
-}
-
-func normalizeKubeVersion(v string) string {
-	v = strings.TrimSpace(v)
-	if v == "" {
-		return ""
-	}
-	if !strings.HasPrefix(v, "v") {
-		v = "v" + v
-	}
-	return v
-}
-
 func buildNodeRegistration(spec monov1alpha1.MonoKSConfigSpec) NodeRegistrationOptions {
 	nodeName := strings.TrimSpace(spec.NodeName)
 	criSocket := strings.TrimSpace(spec.ContainerRuntimeEndpoint)
@@ -464,9 +343,57 @@ func buildNodeRegistration(spec monov1alpha1.MonoKSConfigSpec) NodeRegistrationO
 		)
 	}
 
+	labels := effectiveNodeLabels(spec)
+	if nodeLabels := buildNodeLabelsArg(labels); nodeLabels != "" {
+		nr.KubeletExtraArgs = append(nr.KubeletExtraArgs,
+			KubeadmArg{Name: "node-labels", Value: nodeLabels},
+		)
+	}
+
 	return nr
 }
 
+func effectiveNodeLabels(spec monov1alpha1.MonoKSConfigSpec) map[string]string {
+	if len(spec.NodeLabels) == 0 && !spec.EnableNodeControl {
+		return nil
+	}
+
+	labels := make(map[string]string, len(spec.NodeLabels)+1)
+	for k, v := range spec.NodeLabels {
+		labels[k] = v
+	}
+
+	if spec.EnableNodeControl {
+		labels[monov1alpha1.NodeControlKey] = "true"
+	}
+
+	return labels
+}
+
+func buildNodeLabelsArg(labels map[string]string) string {
+	if len(labels) == 0 {
+		return ""
+	}
+
+	keys := make([]string, 0, len(labels))
+	for k := range labels {
+		k = strings.TrimSpace(k)
+		if k == "" {
+			continue
+		}
+		keys = append(keys, k)
+	}
+	sort.Strings(keys)
+
+	parts := make([]string, 0, len(keys))
+	for _, k := range keys {
+		v := strings.TrimSpace(labels[k])
+		parts = append(parts, k+"="+v)
+	}
+
+	return strings.Join(parts, ",")
+}
+
 func maybeAddBootstrapTaint(nr *NodeRegistrationOptions, role string) {
 	if strings.TrimSpace(role) != "worker" {
 		return
@@ -732,11 +659,6 @@ func RunKubeadmJoin(ctx context.Context, nctx *NodeContext) error {
 	return nil
 }
 
-func RunKubeadmUpgradeNode(context.Context, *NodeContext) error {
-	klog.Info("run_kubeadm_upgrade_node: TODO implement kubeadm upgrade node")
-	return nil
-}
-
 func ReconcileControlPlane(ctx context.Context, nctx *NodeContext) error {
 	if nctx.BootstrapState == nil {
 		return errors.New("BootstrapState is nil, call ClassifyBootstrapAction() first")

clitools/pkg/node/kubeadm_compat.go

@@ -0,0 +1,108 @@
+package node
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"os"
+	"strings"
+
+	"k8s.io/klog/v2"
+
+	"example.com/monok8s/pkg/system"
+)
+
+const kubeadmUpgradeNodeHostnameBugFixedIn = "v1.35.0"
+
+// COMPAT(kubeadm-upgrade-node-hostname)
+// Affects: Kubernetes/kubeadm < v1.35.0
+// Upstream: kubernetes/kubeadm#3244, kubernetes/kubernetes#134319
+// RemoveWhen: minimum supported Kubernetes version >= v1.35.0
+//
+// Affected kubeadm versions can derive the target Node name for
+// `kubeadm upgrade node` from the local OS hostname instead of the existing
+// kubeadm NodeRegistration / kubelet --hostname-override state.
+func needsKubeadmUpgradeNodeHostnameWorkaround(kubeadmVersion string) bool {
+	lt, err := versionLt(kubeadmVersion, kubeadmUpgradeNodeHostnameBugFixedIn)
+	if err != nil {
+		klog.Warningf(
+			"could not parse kubeadm version %q; enabling kubeadm upgrade node hostname workaround: %v",
+			kubeadmVersion,
+			err,
+		)
+		return true
+	}
+	return lt
+}
+
+// runWithTemporaryHostname works around kubernetes/kubeadm#3244, fixed by
+// kubernetes/kubernetes#134319 in Kubernetes v1.35.0.
+//
+// Affected kubeadm versions can derive the target Node name for
+// `kubeadm upgrade node` from the local OS hostname instead of the existing
+// kubeadm NodeRegistration / kubelet --hostname-override state. That breaks
+// valid setups where the machine hostname differs from the Kubernetes Node
+// name: kubeadm may authenticate as one node but try to get/patch another Node,
+// and the Node authorizer correctly rejects it.
+//
+// Keep this workaround scoped to affected kubeadm versions only. Set the
+// temporary hostname to the Kubernetes Node name, run kubeadm, then restore the
+// configured machine hostname immediately afterward.
+func runWithTemporaryHostname(ctx context.Context, nctx *NodeContext, fn func(context.Context) error) error {
+	if nctx == nil {
+		return errors.New("node context is nil")
+	}
+
+	temporaryHostname := strings.TrimSpace(nctx.Config.Spec.NodeName)
+	if temporaryHostname == "" {
+		return errors.New("temporary hostname is required")
+	}
+
+	originalHostname, err := os.Hostname()
+	if err != nil {
+		return fmt.Errorf("get current hostname: %w", err)
+	}
+
+	if originalHostname == temporaryHostname {
+		return fn(ctx)
+	}
+
+	restoreHostname := strings.TrimSpace(nctx.Config.Spec.Network.Hostname)
+	if restoreHostname == "" {
+		restoreHostname = originalHostname
+	}
+
+	klog.Warningf(
+		"temporarily changing hostname for kubeadm upgrade node: current=%q temporary=%q restore=%q",
+		originalHostname,
+		temporaryHostname,
+		restoreHostname,
+	)
+
+	if err := system.SetHostname(temporaryHostname); err != nil {
+		return fmt.Errorf("set temporary hostname to %q: %w", temporaryHostname, err)
+	}
+
+	defer func() {
+		if err := system.SetHostname(restoreHostname); err != nil {
+			klog.Errorf("failed to restore hostname to %q: %v", restoreHostname, err)
+		}
+	}()
+
+	return fn(ctx)
+}
+
+// COMPAT(kubeadm-upgrade-node-hostname)
+// RemoveWhen: minimum supported Kubernetes version >= v1.35.0
+func runKubeadmUpgradeNodeWithCompat(
+	ctx context.Context,
+	nctx *NodeContext,
+	kubeadmVersion string,
+	fn func(context.Context) error,
+) error {
+	if needsKubeadmUpgradeNodeHostnameWorkaround(kubeadmVersion) {
+		return runWithTemporaryHostname(ctx, nctx, fn)
+	}
+
+	return fn(ctx)
+}

clitools/pkg/node/kubeadm_upgrade.go

@@ -14,6 +14,7 @@ import (
 	"k8s.io/client-go/kubernetes"
 	"k8s.io/klog/v2"
 
+	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
 	"example.com/monok8s/pkg/kube"
 	"example.com/monok8s/pkg/system"
 )
@@ -113,7 +114,7 @@ func runUpgradeSelfHealthCheck(ctx context.Context, kubeClient kubernetes.Interf
 			Namespace: healthCheckNamespace,
 			Labels: map[string]string{
 				"app.kubernetes.io/name":       "preupgrade-health-check",
-				"app.kubernetes.io/managed-by": "monok8s",
+				"app.kubernetes.io/managed-by": monov1alpha1.NodeControlName,
 			},
 		},
 		Spec: corev1.PodSpec{
@@ -256,3 +257,102 @@ func describeHealthCheckFailure(ctx context.Context, kubeClient kubernetes.Inter
 
 	return nil
 }
+
+func RunKubeadmUpgradeNode(ctx context.Context, nctx *NodeContext) error {
+	if nctx == nil {
+		return errors.New("node context is nil")
+	}
+	if nctx.Config == nil {
+		return errors.New("node config is nil")
+	}
+	if nctx.LocalClusterState == nil {
+		return errors.New("LocalClusterState is nil. Please run earlier steps first")
+	}
+	if nctx.BootstrapState == nil {
+		return errors.New("BootstrapState is nil. Please run earlier steps first")
+	}
+
+	switch nctx.BootstrapState.Action {
+	case BootstrapActionUpgradeWorker:
+		// continue
+	default:
+		klog.V(4).Infof("RunKubeadmUpgradeNode skipped for action %q", nctx.BootstrapState.Action)
+		return nil
+	}
+
+	wantVersion := normalizeKubeVersion(strings.TrimSpace(nctx.Config.Spec.KubernetesVersion))
+	if wantVersion == "" {
+		return errors.New("spec.kubernetesVersion is required")
+	}
+
+	kubeconfigPath := chooseVersionKubeconfig(nctx.LocalClusterState)
+	if kubeconfigPath == "" {
+		return errors.New("no kubeconfig available for detecting cluster version")
+	}
+
+	clusterVersion := strings.TrimSpace(nctx.BootstrapState.DetectedClusterVersion)
+	if clusterVersion == "" {
+		var err error
+		clusterVersion, err = getServerVersion(ctx, kubeconfigPath)
+		if err != nil {
+			if nctx.BootstrapState.UnsupportedWorkerVersionSkew {
+				klog.Warningf(
+					"cluster version unavailable but worker skew was marked unsupported/permissive, continuing: reason=%s",
+					nctx.BootstrapState.VersionSkewReason,
+				)
+			} else {
+				return fmt.Errorf("get cluster version via %s: %w", kubeconfigPath, err)
+			}
+		}
+	}
+
+	if clusterVersion != "" && !isSupportedWorkerSkew(clusterVersion, wantVersion) {
+		klog.Warningf(
+			"unsupported worker version skew detected, continuing anyway: cluster=%s node=%s",
+			clusterVersion,
+			wantVersion,
+		)
+	}
+
+	klog.Infof(
+		"running kubeadm upgrade node: role=%s clusterVersion=%s nodeVersion=%s kubeconfig=%s",
+		strings.TrimSpace(nctx.Config.Spec.ClusterRole),
+		clusterVersion,
+		wantVersion,
+		kubeconfigPath,
+	)
+
+	args := []string{
+		"upgrade",
+		"node",
+		"--kubeconfig",
+		kubeconfigPath,
+	}
+
+	runKubeadm := func(ctx context.Context) error {
+		_, err := nctx.SystemRunner.RunWithOptions(
+			ctx,
+			"kubeadm",
+			args,
+			system.RunOptions{
+				Timeout: 10 * time.Minute,
+				OnStdoutLine: func(line string) {
+					klog.Infof("[kubeadm] %s", line)
+				},
+				OnStderrLine: func(line string) {
+					klog.Infof("[kubeadm] %s", line)
+				},
+			},
+		)
+		return err
+	}
+
+	// COMPAT(kubeadm-upgrade-node-hostname)
+	// RemoveWhen: minimum supported Kubernetes version >= v1.35.0
+	// Replace this wrapper with direct runKubeadm(ctx).
+	if err := runKubeadmUpgradeNodeWithCompat(ctx, nctx, wantVersion, runKubeadm); err != nil {
+		return fmt.Errorf("run kubeadm upgrade node: %w", err)
+	}
+
+	return nil
+}

clitools/pkg/node/metadata.go

@@ -17,8 +17,8 @@ import (
 func ApplyLocalNodeMetadataIfPossible(ctx context.Context, nctx *NodeContext) error {
 	spec := nctx.Config.Spec
 
-	if len(spec.NodeAnnotations) == 0 && len(spec.NodeLabels) == 0 {
-		klog.V(4).Infof("No annotations or labels was defined")
+	if len(spec.NodeLabels) == 0 {
+		klog.V(4).Infof("No labels was defined")
 		return nil // nothing to do
 	}
 
@@ -53,9 +53,6 @@ func ApplyLocalNodeMetadataIfPossible(ctx context.Context, nctx *NodeContext) er
 	if node.Labels == nil {
 		node.Labels = make(map[string]string)
 	}
-	if node.Annotations == nil {
-		node.Annotations = make(map[string]string)
-	}
 
 	// Apply labels
 	for k, v := range spec.NodeLabels {
@@ -63,13 +60,8 @@ func ApplyLocalNodeMetadataIfPossible(ctx context.Context, nctx *NodeContext) er
 	}
 
 	// Additional Labels
-	if spec.EnableControlAgent {
-		node.Labels[monov1alpah1.ControlAgentKey] = controlAgentNodeSelectorValue
-	}
-
-	// Apply annotations
-	for k, v := range spec.NodeAnnotations {
-		node.Annotations[k] = v
+	if spec.EnableNodeControl {
+		node.Labels[monov1alpah1.NodeControlKey] = "true"
 	}
 
 	_, err = client.CoreV1().Nodes().Update(ctx, node, metav1.UpdateOptions{})
@@ -77,6 +69,6 @@ func ApplyLocalNodeMetadataIfPossible(ctx context.Context, nctx *NodeContext) er
 		return fmt.Errorf("update node metadata: %w", err)
 	}
 
-	klog.Infof("applied labels/annotations to node %q", nodeName)
+	klog.Infof("applied labels to node %q", nodeName)
 	return nil
 }

clitools/pkg/node/prereqs.go

@@ -8,9 +8,18 @@ import (
 	"strings"
 	"time"
 
+	"k8s.io/client-go/discovery"
+	"k8s.io/client-go/kubernetes"
+	"k8s.io/client-go/tools/clientcmd"
 	"k8s.io/klog/v2"
 )
 
+type kubeVersion struct {
+	Major int
+	Minor int
+	Patch int
+}
+
 func ValidateNodeIPAndAPIServerReachability(ctx context.Context, nct *NodeContext) error {
 	requireLocalIP := func(wantedIP string) error {
 		wantedIP = strings.TrimSpace(wantedIP)
@@ -189,3 +198,136 @@ func CheckForVersionSkew(ctx context.Context, nctx *NodeContext) error {
 
 	return nil
 }
+
+func versionEq(a, b string) bool {
+	return normalizeKubeVersion(a) == normalizeKubeVersion(b)
+}
+
+func versionLt(a, b string) (bool, error) {
+	av, err := parseKubeVersion(a)
+	if err != nil {
+		return false, err
+	}
+
+	bv, err := parseKubeVersion(b)
+	if err != nil {
+		return false, err
+	}
+
+	if av.Major != bv.Major {
+		return av.Major < bv.Major, nil
+	}
+	if av.Minor != bv.Minor {
+		return av.Minor < bv.Minor, nil
+	}
+	return av.Patch < bv.Patch, nil
+}
+
+func normalizeKubeVersion(v string) string {
+	v = strings.TrimSpace(v)
+	if v == "" {
+		return ""
+	}
+	if !strings.HasPrefix(v, "v") {
+		v = "v" + v
+	}
+	return v
+}
+
+func parseKubeVersion(s string) (kubeVersion, error) {
+	s = strings.TrimSpace(s)
+	s = strings.TrimPrefix(s, "v")
+
+	var v kubeVersion
+	n, err := fmt.Sscanf(s, "%d.%d.%d", &v.Major, &v.Minor, &v.Patch)
+	// Accepts "1.29" or "1.29.3"
+	if err != nil || n < 2 {
+		return kubeVersion{}, fmt.Errorf("invalid kubernetes version %q", s)
+	}
+	return v, nil
+}
+
+// Control-plane: keep this strict.
+// Accept same version, or a one-minor step where the node binary is newer than the current cluster.
+// That covers normal control-plane upgrade flow but blocks nonsense.
+func isSupportedControlPlaneSkew(clusterVersion, nodeVersion string) bool {
+	cv, err := parseKubeVersion(clusterVersion)
+	if err != nil {
+		return false
+	}
+	nv, err := parseKubeVersion(nodeVersion)
+	if err != nil {
+		return false
+	}
+
+	if cv.Major != nv.Major {
+		return false
+	}
+	if cv.Minor == nv.Minor {
+		return true
+	}
+	if nv.Minor == cv.Minor+1 {
+		return true
+	}
+	return false
+}
+
+// Worker: kubelet generally must not be newer than the apiserver.
+// Older kubelets are allowed within supported skew range.
+// Your requirement says unsupported worker skew should still proceed, so this
+// only classifies support status and must NOT be used to block this function.
+func isSupportedWorkerSkew(clusterVersion, nodeVersion string) bool {
+	cv, err := parseKubeVersion(clusterVersion)
+	if err != nil {
+		return false
+	}
+	nv, err := parseKubeVersion(nodeVersion)
+	if err != nil {
+		return false
+	}
+
+	if cv.Major != nv.Major {
+		return false
+	}
+
+	// kubelet newer than apiserver => unsupported
+	if nv.Minor > cv.Minor {
+		return false
+	}
+
+	// kubelet up to 3 minors older than apiserver => supported
+	if cv.Minor-nv.Minor <= 3 {
+		return true
+	}
+
+	return false
+}
+
+func getServerVersion(ctx context.Context, kubeconfigPath string) (string, error) {
+	restCfg, err := clientcmd.BuildConfigFromFlags("", kubeconfigPath)
+	if err != nil {
+		return "", fmt.Errorf("build kubeconfig %s: %w", kubeconfigPath, err)
+	}
+
+	// Keep this short. This is a probe, not a long-running client.
+	restCfg.Timeout = 5 * time.Second
+
+	clientset, err := kubernetes.NewForConfig(restCfg)
+	if err != nil {
+		return "", fmt.Errorf("create clientset: %w", err)
+	}
+
+	disc := clientset.Discovery()
+	return discoverServerVersion(ctx, disc)
+}
+
+func discoverServerVersion(ctx context.Context, disc discovery.DiscoveryInterface) (string, error) {
+	info, err := disc.ServerVersion()
+	if err != nil {
+		return "", err
+	}
+	if info == nil || strings.TrimSpace(info.GitVersion) == "" {
+		return "", errors.New("server version is empty")
+	}
+	return normalizeKubeVersion(info.GitVersion), nil
+}

clitools/pkg/render/agent.go

@@ -0,0 +1,284 @@
+package render
+
+import (
+	"fmt"
+	"strings"
+
+	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
+	rbacv1 "k8s.io/api/rbac/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+
+	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	buildinfo "example.com/monok8s/pkg/buildinfo"
+)
+
+type AgentConf struct {
+	Namespace        string
+	Image            string
+	ImagePullSecrets []string
+	Labels           map[string]string
+}
+
+func RenderAgentDaemonSets(conf AgentConf) (string, error) {
+	objs, err := buildAgentDaemonSetObjects(conf)
+	if err != nil {
+		return "", err
+	}
+
+	return renderObjects(objs)
+}
+
+func buildAgentDaemonSetObjects(conf AgentConf) ([]runtime.Object, error) {
+	if strings.TrimSpace(conf.Namespace) == "" {
+		return nil, fmt.Errorf("namespace is required")
+	}
+
+	conf.Labels = map[string]string{
+		"app.kubernetes.io/name":       monov1alpha1.NodeAgentName,
+		"app.kubernetes.io/component":  "agent",
+		"app.kubernetes.io/part-of":    "monok8s",
+		"app.kubernetes.io/managed-by": monov1alpha1.NodeControlName,
+	}
+
+	return []runtime.Object{
+		buildAgentServiceAccount(conf),
+		buildAgentClusterRole(conf),
+		buildAgentClusterRoleBinding(conf),
+		buildAgentDaemonSet(conf),
+	}, nil
+}
+
+func buildAgentNamespace(conf AgentConf) *corev1.Namespace {
+	return &corev1.Namespace{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "v1",
+			Kind:       "Namespace",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:   conf.Namespace,
+			Labels: copyStringMap(conf.Labels),
+		},
+	}
+}
+
+func buildAgentServiceAccount(conf AgentConf) *corev1.ServiceAccount {
+	return &corev1.ServiceAccount{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "v1",
+			Kind:       "ServiceAccount",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      monov1alpha1.NodeAgentName,
+			Namespace: conf.Namespace,
+			Labels:    copyStringMap(conf.Labels),
+		},
+	}
+}
+
+func buildAgentClusterRole(conf AgentConf) *rbacv1.ClusterRole {
+	wantRules := []rbacv1.PolicyRule{
+		{
+			APIGroups: []string{monov1alpha1.Group},
+			Resources: []string{"osupgrades"},
+			Verbs:     []string{"get"},
+		},
+		{
+			APIGroups: []string{monov1alpha1.Group},
+			Resources: []string{"osupgradeprogresses"},
+			Verbs:     []string{"get", "list", "watch", "create", "patch", "update"},
+		},
+		{
+			APIGroups: []string{monov1alpha1.Group},
+			Resources: []string{"osupgradeprogresses/status"},
+			Verbs:     []string{"get", "list", "watch", "create", "patch", "update"},
+		},
+		{
+			APIGroups: []string{""},
+			Resources: []string{"nodes"},
+			Verbs:     []string{"get", "list", "watch"},
+		},
+	}
+
+	return &rbacv1.ClusterRole{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "rbac.authorization.k8s.io/v1",
+			Kind:       "ClusterRole",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:   monov1alpha1.NodeAgentName,
+			Labels: copyStringMap(conf.Labels),
+		},
+		Rules: wantRules,
+	}
+}
+
+func buildAgentClusterRoleBinding(conf AgentConf) *rbacv1.ClusterRoleBinding {
+	return &rbacv1.ClusterRoleBinding{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "rbac.authorization.k8s.io/v1",
+			Kind:       "ClusterRoleBinding",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:   monov1alpha1.NodeAgentName,
+			Labels: copyStringMap(conf.Labels),
+		},
+		RoleRef: rbacv1.RoleRef{
+			APIGroup: rbacv1.GroupName,
+			Kind:     "ClusterRole",
+			Name:     monov1alpha1.NodeAgentName,
+		},
+		Subjects: []rbacv1.Subject{
+			{
+				Kind:      "ServiceAccount",
+				Name:      monov1alpha1.NodeAgentName,
+				Namespace: conf.Namespace,
+			},
+		},
+	}
+}
+
+func buildAgentDaemonSet(conf AgentConf) *appsv1.DaemonSet {
+	privileged := true
+	dsLabels := monov1alpha1.NodeAgentLabels()
+
+	image, pullPolicy := agentImage(conf)
+
+	return &appsv1.DaemonSet{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "apps/v1",
+			Kind:       "DaemonSet",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      monov1alpha1.NodeAgentName,
+			Namespace: conf.Namespace,
+			Labels:    copyStringMap(conf.Labels),
+		},
+		Spec: appsv1.DaemonSetSpec{
+			Selector: &metav1.LabelSelector{
+				MatchLabels: map[string]string{
+					"app.kubernetes.io/name": monov1alpha1.NodeAgentName,
+				},
+			},
+			Template: corev1.PodTemplateSpec{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: dsLabels,
+				},
+				Spec: corev1.PodSpec{
+					ServiceAccountName: monov1alpha1.NodeAgentName,
+					HostNetwork:        true,
+					HostPID:            true,
+					DNSPolicy:          corev1.DNSClusterFirstWithHostNet,
+					ImagePullSecrets:   imagePullSecrets(conf.ImagePullSecrets),
+					NodeSelector: map[string]string{
+						monov1alpha1.NodeControlKey: "true",
+					},
+					Tolerations: []corev1.Toleration{
+						{Operator: corev1.TolerationOpExists},
+					},
+					Containers: []corev1.Container{
+						{
+							Name:            "agent",
+							Image:           image,
+							ImagePullPolicy: pullPolicy,
+							Args:            []string{"agent", "--env-file", "$(CLUSTER_ENV_FILE)"},
+							Env: []corev1.EnvVar{
+								{
+									Name: "NODE_NAME",
+									ValueFrom: &corev1.EnvVarSource{
+										FieldRef: &corev1.ObjectFieldSelector{
+											APIVersion: "v1",
+											FieldPath:  "spec.nodeName",
+										},
+									},
+								},
+								{
+									Name:  "CLUSTER_ENV_FILE",
+									Value: "/host/opt/monok8s/config/cluster.env",
+								},
+								{
+									Name:  "FW_ENV_CONFIG_FILE",
+									Value: "/host/etc/fw_env.config",
+								},
+							},
+							SecurityContext: &corev1.SecurityContext{
+								Privileged: &privileged,
+							},
+							VolumeMounts: []corev1.VolumeMount{
+								{
+									Name:      "host-dev",
+									MountPath: "/dev",
+								},
+								{
+									Name:      "host-etc",
+									MountPath: "/host/etc",
+									ReadOnly:  true,
+								},
+								{
+									Name:      "host-config",
+									MountPath: "/host/opt/monok8s/config",
+									ReadOnly:  true,
+								},
+							},
+						},
+					},
+					Volumes: []corev1.Volume{
+						{
+							Name: "host-dev",
+							VolumeSource: corev1.VolumeSource{
+								HostPath: &corev1.HostPathVolumeSource{
+									Path: "/dev",
+									Type: hostPathType(corev1.HostPathDirectory),
+								},
+							},
+						},
+						{
+							Name: "host-etc",
+							VolumeSource: corev1.VolumeSource{
+								HostPath: &corev1.HostPathVolumeSource{
+									Path: "/etc",
+									Type: hostPathType(corev1.HostPathDirectory),
+								},
+							},
+						},
+						{
+							Name: "host-config",
+							VolumeSource: corev1.VolumeSource{
+								HostPath: &corev1.HostPathVolumeSource{
+									Path: "/opt/monok8s/config",
+									Type: hostPathType(corev1.HostPathDirectory),
+								},
+							},
+						},
+					},
+				},
+			},
+		},
+	}
+}
+
+func agentImage(conf AgentConf) (string, corev1.PullPolicy) {
+	if conf.Image != "" {
+		return conf.Image, corev1.PullIfNotPresent
+	}
+
+	return fmt.Sprintf("localhost/monok8s/node-control:%s", buildinfo.Version), corev1.PullNever
+}
+
+func copyStringMap(in map[string]string) map[string]string {
+	if len(in) == 0 {
+		return nil
+	}
+
+	out := make(map[string]string, len(in))
+	for k, v := range in {
+		out[k] = v
+	}
+
+	return out
+}
+
+func hostPathType(t corev1.HostPathType) *corev1.HostPathType {
+	return &t
+}

clitools/pkg/render/agent_apply.go

@@ -0,0 +1,203 @@
+package render
+
+import (
+	"context"
+	"fmt"
+	"reflect"
+
+	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
+	rbacv1 "k8s.io/api/rbac/v1"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/client-go/kubernetes"
+)
+
+func ApplyAgentDaemonSets(ctx context.Context, kubeClient kubernetes.Interface, conf AgentConf) error {
+	objs, err := buildAgentDaemonSetObjects(conf)
+	if err != nil {
+		return err
+	}
+
+	if err := applyAgentNamespace(ctx, kubeClient, buildAgentNamespace(conf)); err != nil {
+		return fmt.Errorf("apply namespace: %w", err)
+	}
+
+	for _, obj := range objs {
+		if err := applyAgentObject(ctx, kubeClient, obj); err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func applyAgentObject(ctx context.Context, kubeClient kubernetes.Interface, obj runtime.Object) error {
+	switch want := obj.(type) {
+	case *corev1.ServiceAccount:
+		return applyAgentServiceAccount(ctx, kubeClient, want)
+	case *rbacv1.ClusterRole:
+		return applyAgentClusterRole(ctx, kubeClient, want)
+	case *rbacv1.ClusterRoleBinding:
+		return applyAgentClusterRoleBinding(ctx, kubeClient, want)
+	case *appsv1.DaemonSet:
+		return applyAgentDaemonSet(ctx, kubeClient, want)
+	default:
+		return fmt.Errorf("unsupported agent object type %T", obj)
+	}
+}
+
+func applyAgentNamespace(ctx context.Context, kubeClient kubernetes.Interface, want *corev1.Namespace) error {
+	existing, err := kubeClient.CoreV1().Namespaces().Get(ctx, want.Name, metav1.GetOptions{})
+	if apierrors.IsNotFound(err) {
+		_, err = kubeClient.CoreV1().Namespaces().Create(ctx, want, metav1.CreateOptions{})
+		return err
+	}
+	if err != nil {
+		return err
+	}
+
+	labels, changed := mergeStringMapsInto(existing.Labels, want.Labels)
+	if !changed {
+		return nil
+	}
+
+	existing.Labels = labels
+	_, err = kubeClient.CoreV1().Namespaces().Update(ctx, existing, metav1.UpdateOptions{})
+	return err
+}
+
+func applyAgentServiceAccount(ctx context.Context, kubeClient kubernetes.Interface, want *corev1.ServiceAccount) error {
+	existing, err := kubeClient.CoreV1().ServiceAccounts(want.Namespace).Get(ctx, want.Name, metav1.GetOptions{})
+	if apierrors.IsNotFound(err) {
+		_, err = kubeClient.CoreV1().ServiceAccounts(want.Namespace).Create(ctx, want, metav1.CreateOptions{})
+		return err
+	}
+	if err != nil {
+		return err
+	}
+
+	changed := false
+	if !reflect.DeepEqual(existing.Labels, want.Labels) {
+		existing.Labels = want.Labels
+		changed = true
+	}
+
+	if !changed {
+		return nil
+	}
+
+	_, err = kubeClient.CoreV1().ServiceAccounts(want.Namespace).Update(ctx, existing, metav1.UpdateOptions{})
+	return err
+}
+
+func applyAgentClusterRole(ctx context.Context, kubeClient kubernetes.Interface, want *rbacv1.ClusterRole) error {
+	existing, err := kubeClient.RbacV1().ClusterRoles().Get(ctx, want.Name, metav1.GetOptions{})
+	if apierrors.IsNotFound(err) {
+		_, err = kubeClient.RbacV1().ClusterRoles().Create(ctx, want, metav1.CreateOptions{})
+		return err
+	}
+	if err != nil {
+		return err
+	}
+
+	changed := false
+	if !reflect.DeepEqual(existing.Labels, want.Labels) {
+		existing.Labels = want.Labels
+		changed = true
+	}
+	if !reflect.DeepEqual(existing.Rules, want.Rules) {
+		existing.Rules = want.Rules
+		changed = true
+	}
+
+	if !changed {
+		return nil
+	}
+
+	_, err = kubeClient.RbacV1().ClusterRoles().Update(ctx, existing, metav1.UpdateOptions{})
+	return err
+}
+
+func applyAgentClusterRoleBinding(ctx context.Context, kubeClient kubernetes.Interface, want *rbacv1.ClusterRoleBinding) error {
+	existing, err := kubeClient.RbacV1().ClusterRoleBindings().Get(ctx, want.Name, metav1.GetOptions{})
+	if apierrors.IsNotFound(err) {
+		_, err = kubeClient.RbacV1().ClusterRoleBindings().Create(ctx, want, metav1.CreateOptions{})
+		return err
+	}
+	if err != nil {
+		return err
+	}
+
+	// roleRef is immutable. If it differs, fail loudly instead of pretending we can patch it.
+	if !reflect.DeepEqual(existing.RoleRef, want.RoleRef) {
+		return fmt.Errorf("existing ClusterRoleBinding %q has different roleRef and must be recreated", want.Name)
+	}
+
+	changed := false
+	if !reflect.DeepEqual(existing.Labels, want.Labels) {
+		existing.Labels = want.Labels
+		changed = true
+	}
+	if !reflect.DeepEqual(existing.Subjects, want.Subjects) {
+		existing.Subjects = want.Subjects
+		changed = true
+	}
+
+	if !changed {
+		return nil
+	}
+
+	_, err = kubeClient.RbacV1().ClusterRoleBindings().Update(ctx, existing, metav1.UpdateOptions{})
+	return err
+}
+
+func applyAgentDaemonSet(ctx context.Context, kubeClient kubernetes.Interface, want *appsv1.DaemonSet) error {
+	existing, err := kubeClient.AppsV1().DaemonSets(want.Namespace).Get(ctx, want.Name, metav1.GetOptions{})
+	if apierrors.IsNotFound(err) {
+		_, err = kubeClient.AppsV1().DaemonSets(want.Namespace).Create(ctx, want, metav1.CreateOptions{})
+		return err
+	}
+	if err != nil {
+		return err
+	}
+
+	changed := false
+	if !reflect.DeepEqual(existing.Labels, want.Labels) {
+		existing.Labels = want.Labels
+		changed = true
+	}
+	if !reflect.DeepEqual(existing.Spec, want.Spec) {
+		existing.Spec = want.Spec
+		changed = true
+	}
+
+	if !changed {
+		return nil
+	}
+
+	_, err = kubeClient.AppsV1().DaemonSets(want.Namespace).Update(ctx, existing, metav1.UpdateOptions{})
+	return err
+}
+
+func mergeStringMapsInto(dst map[string]string, src map[string]string) (map[string]string, bool) {
+	if len(src) == 0 {
+		return dst, false
+	}
+
+	changed := false
+	if dst == nil {
+		dst = map[string]string{}
+		changed = true
+	}
+
+	for k, v := range src {
+		if dst[k] != v {
+			dst[k] = v
+			changed = true
+		}
+	}
+
+	return dst, changed
+}

clitools/pkg/render/controller.go

@@ -0,0 +1,324 @@
+package render
+
+import (
+	"fmt"
+
+	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
+	rbacv1 "k8s.io/api/rbac/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/util/intstr"
+
+	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	buildinfo "example.com/monok8s/pkg/buildinfo"
+)
+
+type ControllerConf struct {
+	Namespace        string
+	Image            string
+	ImagePullSecrets []string
+	Labels           map[string]string
+}
+
+func RenderControllerDeployments(conf ControllerConf) (string, error) {
+	if conf.Namespace == "" {
+		return "", fmt.Errorf("namespace is required")
+	}
+
+	conf.Labels = map[string]string{
+		"app.kubernetes.io/name":       monov1alpha1.ControllerName,
+		"app.kubernetes.io/component":  "controller",
+		"app.kubernetes.io/part-of":    "monok8s",
+		"app.kubernetes.io/managed-by": monov1alpha1.NodeControlName,
+	}
+
+	objs := []runtime.Object{
+		buildControllerServiceAccount(conf),
+		buildControllerClusterRole(conf),
+		buildControllerClusterRoleBinding(conf),
+		buildControllerDeployment(conf),
+	}
+
+	return renderObjects(objs)
+}
+
+func buildControllerServiceAccount(conf ControllerConf) *corev1.ServiceAccount {
+
+	automount := true
+
+	return &corev1.ServiceAccount{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "v1",
+			Kind:       "ServiceAccount",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      monov1alpha1.ControllerName,
+			Namespace: conf.Namespace,
+			Labels:    conf.Labels,
+		},
+		AutomountServiceAccountToken: &automount,
+	}
+}
+
+func buildControllerClusterRole(conf ControllerConf) *rbacv1.ClusterRole {
+	wantRules := []rbacv1.PolicyRule{
+		{
+			APIGroups: []string{monov1alpha1.Group},
+			Resources: []string{"osupgrades"},
+			Verbs:     []string{"get", "list", "watch"},
+		},
+		{
+			APIGroups: []string{monov1alpha1.Group},
+			Resources: []string{"osupgrades/status"},
+			Verbs:     []string{"get", "patch", "update"},
+		},
+		{
+			APIGroups: []string{monov1alpha1.Group},
+			Resources: []string{"osupgradeprogresses"},
+			Verbs:     []string{"get", "create"},
+		},
+		{
+			APIGroups: []string{monov1alpha1.Group},
+			Resources: []string{"osupgradeprogresses/status"},
+			Verbs:     []string{"update"},
+		},
+		{
+			APIGroups: []string{""},
+			Resources: []string{"nodes"},
+			Verbs:     []string{"get", "list"},
+		},
+	}
+
+	return &rbacv1.ClusterRole{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "rbac.authorization.k8s.io/v1",
+			Kind:       "ClusterRole",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:   monov1alpha1.ControllerName,
+			Labels: conf.Labels,
+		},
+		Rules: wantRules,
+	}
+}
+
+func buildControllerClusterRoleBinding(conf ControllerConf) *rbacv1.ClusterRoleBinding {
+
+	wantSubjects := []rbacv1.Subject{
+		{
+			Kind:      "ServiceAccount",
+			Name:      monov1alpha1.ControllerName,
+			Namespace: conf.Namespace,
+		},
+	}
+
+	wantRoleRef := rbacv1.RoleRef{
+		APIGroup: rbacv1.GroupName,
+		Kind:     "ClusterRole",
+		Name:     monov1alpha1.ControllerName,
+	}
+
+	return &rbacv1.ClusterRoleBinding{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "rbac.authorization.k8s.io/v1",
+			Kind:       "ClusterRoleBinding",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:   monov1alpha1.ControllerName,
+			Labels: conf.Labels,
+		},
+		Subjects: wantSubjects,
+		RoleRef:  wantRoleRef,
+	}
+}
+
+func buildControllerDeployment(conf ControllerConf) *appsv1.Deployment {
+	replicas := int32(1)
+
+	selectorLabels := map[string]string{
+		"app.kubernetes.io/name":      monov1alpha1.ControllerName,
+		"app.kubernetes.io/component": "controller",
+	}
+
+	podLabels := mergeStringMaps(conf.Labels, selectorLabels)
+
+	runAsNonRoot := true
+	allowPrivilegeEscalation := false
+	userGroup := int64(65532)
+
+	image, pullPolicy := controllerImage(conf)
+
+	return &appsv1.Deployment{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "apps/v1",
+			Kind:       "Deployment",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      monov1alpha1.ControllerName,
+			Namespace: conf.Namespace,
+			Labels:    conf.Labels,
+		},
+		Spec: appsv1.DeploymentSpec{
+			Replicas: &replicas,
+			Selector: &metav1.LabelSelector{
+				MatchLabels: selectorLabels,
+			},
+			Template: corev1.PodTemplateSpec{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: podLabels,
+				},
+				Spec: corev1.PodSpec{
+					ServiceAccountName: monov1alpha1.ControllerName,
+					ImagePullSecrets:   imagePullSecrets(conf.ImagePullSecrets),
+					Containers: []corev1.Container{
+						{
+							Name:            "controller",
+							Image:           image,
+							ImagePullPolicy: pullPolicy,
+							Args: []string{
+								"controller",
+								"--namespace",
+								conf.Namespace,
+							},
+							Env: []corev1.EnvVar{
+								{
+									Name: "POD_NAME",
+									ValueFrom: &corev1.EnvVarSource{
+										FieldRef: &corev1.ObjectFieldSelector{
+											APIVersion: "v1",
+											FieldPath:  "metadata.name",
+										},
+									},
+								},
+								{
+									Name: "POD_NAMESPACE",
+									ValueFrom: &corev1.EnvVarSource{
+										FieldRef: &corev1.ObjectFieldSelector{
+											APIVersion: "v1",
+											FieldPath:  "metadata.namespace",
+										},
+									},
+								},
+								{
+									Name: "NODE_NAME",
+									ValueFrom: &corev1.EnvVarSource{
+										FieldRef: &corev1.ObjectFieldSelector{
+											APIVersion: "v1",
+											FieldPath:  "spec.nodeName",
+										},
+									},
+								},
+							},
+							Ports: []corev1.ContainerPort{
+								{
+									Name:          "http",
+									ContainerPort: 8080,
+									Protocol:      corev1.ProtocolTCP,
+								},
+								{
+									Name:          "https",
+									ContainerPort: 8443,
+									Protocol:      corev1.ProtocolTCP,
+								},
+							},
+							LivenessProbe: &corev1.Probe{
+								ProbeHandler: corev1.ProbeHandler{
+									HTTPGet: &corev1.HTTPGetAction{
+										Path: "/healthz",
+										Port: intstr.FromString("http"),
+									},
+								},
+								InitialDelaySeconds: 5,
+								PeriodSeconds:       60,
+								TimeoutSeconds:      2,
+								FailureThreshold:    3,
+							},
+							ReadinessProbe: &corev1.Probe{
+								ProbeHandler: corev1.ProbeHandler{
+									HTTPGet: &corev1.HTTPGetAction{
+										Path: "/readyz",
+										Port: intstr.FromString("http"),
+									},
+								},
+								InitialDelaySeconds: 2,
+								PeriodSeconds:       5,
+								TimeoutSeconds:      2,
+								FailureThreshold:    3,
+							},
+							SecurityContext: &corev1.SecurityContext{
+								RunAsNonRoot:             &runAsNonRoot,
+								RunAsUser:                &userGroup,
+								RunAsGroup:               &userGroup,
+								AllowPrivilegeEscalation: &allowPrivilegeEscalation,
+							},
+						},
+					},
+					NodeSelector: controllerNodeSelector(conf),
+					Affinity:     controllerAffinity(conf),
+				},
+			},
+		},
+	}
+}
+
+func controllerImage(conf ControllerConf) (string, corev1.PullPolicy) {
+	if conf.Image != "" {
+		return conf.Image, corev1.PullIfNotPresent
+	}
+
+	return fmt.Sprintf("localhost/monok8s/node-control:%s", buildinfo.Version), corev1.PullNever
+}
+
+func controllerNodeSelector(conf ControllerConf) map[string]string {
+	if conf.Image != "" {
+		return nil
+	}
+
+	// Local image exists on managed nodes only.
+	return map[string]string{
+		monov1alpha1.NodeControlKey: "true",
+	}
+}
+
+func controllerAffinity(conf ControllerConf) *corev1.Affinity {
+	// Local image exists only on managed nodes, so in that mode we already use
+	// NodeSelector and should not fight placement with anti-affinity.
+	if conf.Image == "" {
+		return nil
+	}
+
+	return &corev1.Affinity{
+		PodAntiAffinity: &corev1.PodAntiAffinity{
+			PreferredDuringSchedulingIgnoredDuringExecution: []corev1.WeightedPodAffinityTerm{
+				{
+					Weight: 100,
+					PodAffinityTerm: corev1.PodAffinityTerm{
+						TopologyKey: corev1.LabelHostname,
+						LabelSelector: &metav1.LabelSelector{
+							MatchLabels: monov1alpha1.NodeAgentLabels(),
+						},
+					},
+				},
+			},
+		},
+	}
+}
+
+func mergeStringMaps(maps ...map[string]string) map[string]string {
+	var total int
+	for _, m := range maps {
+		total += len(m)
+	}
+	if total == 0 {
+		return nil
+	}
+
+	out := make(map[string]string, total)
+	for _, m := range maps {
+		for k, v := range m {
+			out[k] = v
+		}
+	}
+	return out
+}

clitools/pkg/render/helpers.go

@@ -0,0 +1,74 @@
+package render
+
+import (
+	"bytes"
+	"fmt"
+	"strings"
+
+	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"sigs.k8s.io/yaml"
+)
+
+func renderObjects(objs []runtime.Object) (string, error) {
+	var buf bytes.Buffer
+
+	for i, obj := range objs {
+		if i > 0 {
+			if _, err := fmt.Fprintln(&buf, "---"); err != nil {
+				return "", err
+			}
+		}
+
+		b, err := renderObjectYAML(obj)
+		if err != nil {
+			return "", err
+		}
+
+		if _, err := buf.Write(b); err != nil {
+			return "", err
+		}
+	}
+
+	return buf.String(), nil
+}
+
+func renderObjectYAML(obj runtime.Object) ([]byte, error) {
+	b, err := yaml.Marshal(obj)
+	if err != nil {
+		return nil, err
+	}
+
+	var m map[string]any
+	if err := yaml.Unmarshal(b, &m); err != nil {
+		return nil, err
+	}
+
+	delete(m, "status")
+
+	return yaml.Marshal(m)
+}
+
+func imagePullSecrets(names []string) []corev1.LocalObjectReference {
+	if len(names) == 0 {
+		return nil
+	}
+
+	refs := make([]corev1.LocalObjectReference, 0, len(names))
+	for _, name := range names {
+		name = strings.TrimSpace(name)
+		if name == "" {
+			continue
+		}
+
+		refs = append(refs, corev1.LocalObjectReference{
+			Name: name,
+		})
+	}
+
+	if len(refs) == 0 {
+		return nil
+	}
+
+	return refs
+}

clitools/pkg/render/monoks.go

@@ -1,4 +1,4 @@
-package templates
+package render
 
 import (
 	"bytes"
@@ -31,10 +31,12 @@ func RenderMonoKSConfig() (string, error) {
 	return buf.String(), nil
 }
 
-func RenderOSUpgrade() (string, error) {
+func RenderOSUpgrade(namespace string) (string, error) {
 	vals := templates.LoadTemplateValuesFromEnv()
 	cfg := templates.DefaultOSUpgrade(vals)
 
+	cfg.Namespace = namespace
+
 	s := runtime.NewScheme()
 	if err := monov1alpha1.AddToScheme(s); err != nil {
 		return "", err

clitools/pkg/render/sshd.go

@@ -0,0 +1,304 @@
+package render
+
+import (
+	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/resource"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+	"k8s.io/apimachinery/pkg/util/intstr"
+
+	monov1alpha1 "example.com/monok8s/pkg/apis/monok8s/v1alpha1"
+	"example.com/monok8s/pkg/templates"
+)
+
+const (
+	sshdName       = "sshd"
+	sshdConfigName = "sshd-authorized-keys"
+	sshdNodePort   = int32(30022)
+)
+
+func RenderSSHDDeployments(namespace, authKeys string) (string, error) {
+	vals := templates.LoadTemplateValuesFromEnv()
+
+	labels := map[string]string{
+		"app.kubernetes.io/name":       sshdName,
+		"app.kubernetes.io/component":  "host-access",
+		"app.kubernetes.io/part-of":    "monok8s",
+		"app.kubernetes.io/managed-by": monov1alpha1.NodeControlName,
+	}
+
+	objs := []runtime.Object{
+		buildSSHDConfigMap(authKeys, namespace, labels),
+		buildSSHDService(vals, namespace, labels),
+		buildSSHDDeployment(vals, namespace, labels),
+	}
+
+	return renderObjects(objs)
+}
+
+func buildSSHDConfigMap(
+	authorizedKeys string,
+	namespace string,
+	labels map[string]string,
+) *corev1.ConfigMap {
+
+	return &corev1.ConfigMap{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "v1",
+			Kind:       "ConfigMap",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      sshdConfigName,
+			Namespace: namespace,
+			Labels:    labels,
+		},
+		Data: map[string]string{
+			"authorized_keys": authorizedKeys,
+		},
+	}
+}
+
+func buildSSHDService(
+	tVals templates.TemplateValues,
+	namespace string,
+	labels map[string]string,
+) *corev1.Service {
+	selectorLabels := map[string]string{
+		monov1alpha1.NodeControlKey: "true",
+		"kubernetes.io/hostname":    tVals.NodeName,
+	}
+
+	return &corev1.Service{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "v1",
+			Kind:       "Service",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      sshdName,
+			Namespace: namespace,
+			Labels:    labels,
+		},
+		Spec: corev1.ServiceSpec{
+			Type:     corev1.ServiceTypeNodePort,
+			Selector: selectorLabels,
+			Ports: []corev1.ServicePort{
+				{
+					Name:       "ssh",
+					Protocol:   corev1.ProtocolTCP,
+					Port:       22,
+					TargetPort: intstr.FromInt32(22),
+					NodePort:   sshdNodePort,
+				},
+			},
+		},
+	}
+}
+
+func buildSSHDDeployment(
+	tVals templates.TemplateValues,
+	namespace string,
+	labels map[string]string,
+) *appsv1.Deployment {
+	replicas := int32(1)
+
+	selectorLabels := map[string]string{
+		monov1alpha1.NodeControlKey: "true",
+		"kubernetes.io/hostname":    tVals.NodeName,
+	}
+
+	podLabels := mergeStringMaps(labels, selectorLabels)
+
+	runAsUser := int64(0)
+	runAsNonRoot := false
+	privileged := true
+	allowPrivilegeEscalation := true
+	readOnlyRootFilesystem := false
+
+	return &appsv1.Deployment{
+		TypeMeta: metav1.TypeMeta{
+			APIVersion: "apps/v1",
+			Kind:       "Deployment",
+		},
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      sshdName,
+			Namespace: namespace,
+			Labels:    labels,
+		},
+		Spec: appsv1.DeploymentSpec{
+			Replicas: &replicas,
+			Selector: &metav1.LabelSelector{
+				MatchLabels: selectorLabels,
+			},
+			Template: corev1.PodTemplateSpec{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: podLabels,
+				},
+				Spec: corev1.PodSpec{
+					HostPID:      true,
+					NodeSelector: selectorLabels,
+					Containers: []corev1.Container{
+						{
+							Name:  sshdName,
+							Image: "alpine:latest",
+							Command: []string{
+								"/bin/sh",
+								"-ceu",
+								`
+apk add --no-cache openssh-server
+
+mkdir -p /run/sshd
+mkdir -p /root/.ssh
+
+cp /authorized-keys/authorized_keys /root/.ssh/authorized_keys
+chmod 700 /root/.ssh
+chmod 600 /root/.ssh/authorized_keys
+
+ssh-keygen -A
+
+exec /usr/sbin/sshd \
+  -D \
+  -e \
+  -p 22 \
+  -o PermitRootLogin=prohibit-password \
+  -o PasswordAuthentication=no \
+  -o KbdInteractiveAuthentication=no \
+  -o PubkeyAuthentication=yes \
+  -o AuthorizedKeysFile=/root/.ssh/authorized_keys
+`,
+							},
+							Ports: []corev1.ContainerPort{
+								{
+									Name:          "ssh",
+									ContainerPort: 22,
+									Protocol:      corev1.ProtocolTCP,
+								},
+							},
+							SecurityContext: &corev1.SecurityContext{
+								RunAsUser:                &runAsUser,
+								RunAsNonRoot:             &runAsNonRoot,
+								Privileged:               &privileged,
+								AllowPrivilegeEscalation: &allowPrivilegeEscalation,
+								ReadOnlyRootFilesystem:   &readOnlyRootFilesystem,
+							},
+							Resources: corev1.ResourceRequirements{
+								Requests: corev1.ResourceList{
+									corev1.ResourceCPU:    resource.MustParse("10m"),
+									corev1.ResourceMemory: resource.MustParse("32Mi"),
+								},
+								Limits: corev1.ResourceList{
+									corev1.ResourceCPU:    resource.MustParse("200m"),
+									corev1.ResourceMemory: resource.MustParse("128Mi"),
+								},
+							},
+							VolumeMounts: append(
+								[]corev1.VolumeMount{
+									{
+										Name:      "authorized-keys",
+										MountPath: "/authorized-keys",
+										ReadOnly:  true,
+									},
+								},
+								buildHostRootVolumeMounts()...,
+							),
+						},
+					},
+					Volumes: append(
+						[]corev1.Volume{
+							{
+								Name: "authorized-keys",
+								VolumeSource: corev1.VolumeSource{
+									ConfigMap: &corev1.ConfigMapVolumeSource{
+										LocalObjectReference: corev1.LocalObjectReference{
+											Name: sshdConfigName,
+										},
+										DefaultMode: ptrInt32(0600),
+									},
+								},
+							},
+						},
+						buildHostRootVolumes()...,
+					),
+				},
+			},
+		},
+	}
+}
+
+func buildHostRootVolumeMounts() []corev1.VolumeMount {
+	paths := []struct {
+		name      string
+		mountPath string
+		readOnly  bool
+	}{
+		{"host-bin", "/host/bin", true},
+		{"host-sbin", "/host/sbin", true},
+		{"host-lib", "/host/lib", true},
+		{"host-usr", "/host/usr", true},
+		{"host-etc", "/host/etc", false},
+		{"host-run", "/host/run", false},
+		{"host-proc", "/host/proc", false},
+		{"host-sys", "/host/sys", false},
+		{"host-dev", "/host/dev", false},
+		{"host-var", "/host/var", false},
+	}
+
+	mounts := make([]corev1.VolumeMount, 0, len(paths))
+
+	for _, p := range paths {
+		mounts = append(mounts, corev1.VolumeMount{
+			Name:      p.name,
+			MountPath: p.mountPath,
+			ReadOnly:  p.readOnly,
+		})
+	}
+
+	return mounts
+}
+
+func buildHostRootVolumes() []corev1.Volume {
+	hostPathDir := corev1.HostPathDirectory
+
+	paths := []struct {
+		name string
+		path string
+	}{
+		{"host-bin", "/bin"},
+		{"host-sbin", "/sbin"},
+		{"host-lib", "/lib"},
+		{"host-usr", "/usr"},
+		{"host-etc", "/etc"},
+		{"host-run", "/run"},
+		{"host-proc", "/proc"},
+		{"host-sys", "/sys"},
+		{"host-dev", "/dev"},
+
+		// /var is an rbind mount in monok8s and may be private.
+		// Mount the real backing path instead.
+		{"host-var", "/data/var"},
+	}
+
+	volumes := make([]corev1.Volume, 0, len(paths))
+
+	for _, p := range paths {
+		volumes = append(volumes, corev1.Volume{
+			Name: p.name,
+			VolumeSource: corev1.VolumeSource{
+				HostPath: &corev1.HostPathVolumeSource{
+					Path: p.path,
+					Type: &hostPathDir,
+				},
+			},
+		})
+	}
+
+	return volumes
+}
+
+func ptrInt32(v int32) *int32 {
+	return &v
+}
+
+func ptrHostPathType(v corev1.HostPathType) *corev1.HostPathType {
+	return &v
+}

clitools/pkg/templates/templates.go

@@ -7,7 +7,7 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
 
-const DefaultNamespace = "kube-system"
+const DefaultNamespace = "mono-system"
 
 func DefaultMonoKSConfig(v TemplateValues) monov1alpha1.MonoKSConfig {
 	return monov1alpha1.MonoKSConfig{
@@ -25,7 +25,7 @@ func DefaultMonoKSConfig(v TemplateValues) monov1alpha1.MonoKSConfig {
 
 			ClusterRole:        v.ClusterRole,
 			InitControlPlane:   v.InitControlPlane,
-			EnableControlAgent: v.EnableControlAgent,
+			EnableNodeControl: v.EnableNodeControl,
 
 			ClusterName:   v.ClusterName,
 			ClusterDomain: v.ClusterDomain,
@@ -51,7 +51,6 @@ func DefaultMonoKSConfig(v TemplateValues) monov1alpha1.MonoKSConfig {
 
 			SubjectAltNames: copyStringSlice(v.SubjectAltNames),
 			NodeLabels:      copyStringMap(v.NodeLabels),
-			NodeAnnotations: copyStringMap(v.NodeAnnotations),
 
 			Network: monov1alpha1.NetworkSpec{
 				Hostname:         firstNonEmpty(v.Hostname, v.NodeName),

clitools/pkg/templates/values.go

@@ -24,9 +24,9 @@ type TemplateValues struct {
 	ContainerRuntimeEndpoint  string
 	CNIPlugin                 string
 
-	ClusterRole        string // worker, control-plane
-	InitControlPlane   bool
-	EnableControlAgent bool
+	ClusterRole       string // worker, control-plane
+	InitControlPlane  bool
+	EnableNodeControl bool
 
 	AllowSchedulingOnControlPlane bool
 	SkipImageCheck                bool
@@ -39,7 +39,6 @@ type TemplateValues struct {
 
 	SubjectAltNames []string
 	NodeLabels      map[string]string
-	NodeAnnotations map[string]string
 }
 
 func defaultTemplateValues() TemplateValues {
@@ -59,9 +58,9 @@ func defaultTemplateValues() TemplateValues {
 		ContainerRuntimeEndpoint:  "unix:///var/run/crio/crio.sock",
 		CNIPlugin:                 "default",
 
-		ClusterRole:        "control-plane",
-		InitControlPlane:   true,
-		EnableControlAgent: true,
+		ClusterRole:       "control-plane",
+		InitControlPlane:  true,
+		EnableNodeControl: true,
 
 		AllowSchedulingOnControlPlane: true,
 		SkipImageCheck:                false,
@@ -78,9 +77,6 @@ func defaultTemplateValues() TemplateValues {
 		NodeLabels: map[string]string{
 			monov1alpha1.Label: "value",
 		},
-		NodeAnnotations: map[string]string{
-			monov1alpha1.Annotation: "value",
-		},
 	}
 }
 
@@ -88,7 +84,7 @@ func LoadTemplateValuesFromEnv() TemplateValues {
 	v := defaultTemplateValues()
 
 	v.Hostname = getenvDefault("MKS_HOSTNAME", v.Hostname)
-	v.NodeName = getenvDefault("MKS_NODE_NAME", v.Hostname)
+	v.NodeName = getenvDefault("MKS_NODE_NAME", getenvDefault("NODE_NAME", v.Hostname))
 
 	v.KubernetesVersion = getenvDefault("MKS_KUBERNETES_VERSION", v.KubernetesVersion)
 	v.ClusterName = getenvDefault("MKS_CLUSTER_NAME", v.ClusterName)
@@ -108,7 +104,7 @@ func LoadTemplateValuesFromEnv() TemplateValues {
 
 	v.ClusterRole = getenvDefault("MKS_CLUSTER_ROLE", v.ClusterRole)
 	v.InitControlPlane = getenvBoolDefault("MKS_INIT_CONTROL_PLANE", v.InitControlPlane)
-	v.EnableControlAgent = getenvBoolDefault("MKS_ENABLE_CONTROL_AGENT", v.EnableControlAgent)
+	v.EnableNodeControl = getenvBoolDefault("MKS_ENABLE_NODE_CONTROL", v.EnableNodeControl)
 
 	v.AllowSchedulingOnControlPlane = getenvBoolDefault("MKS_ALLOW_SCHEDULING_ON_CONTROL_PLANE", v.AllowSchedulingOnControlPlane)
 	v.SkipImageCheck = getenvBoolDefault("MKS_SKIP_IMAGE_CHECK", v.SkipImageCheck)
@@ -129,9 +125,6 @@ func LoadTemplateValuesFromEnv() TemplateValues {
 	if m := parseKeyValueMap(os.Getenv("MKS_NODE_LABELS")); len(m) > 0 {
 		v.NodeLabels = m
 	}
-	if m := parseKeyValueMap(os.Getenv("MKS_NODE_ANNOTATIONS")); len(m) > 0 {
-		v.NodeAnnotations = m
-	}
 
 	return v
 }

configs/cluster.env.default

@@ -30,8 +30,8 @@ MKS_CLUSTER_DOMAIN=cluster.local
 MKS_CLUSTER_ROLE=control-plane
 MKS_INIT_CONTROL_PLANE=yes
 
-# OSUpgrade agent
-MKS_ENABLE_CONTROL_AGENT=yes
+# Enable if you want OTA OSUpgrade
+MKS_ENABLE_NODE_CONTROL=yes
 
 # Boot configs
 # usb, emmc
@@ -56,8 +56,7 @@ MKS_CNI_PLUGIN=default
 
 # Node registration metadata
 # Comma-separated key=value pairs
-MKS_NODE_LABELS=topology.kubernetes.io/zone=lab,node.kubernetes.io/instance-type=mono-gateway
-MKS_NODE_ANNOTATIONS=mono.si/board=ls1046a,monok8s.io/image-version=dev
+MKS_NODE_LABELS=topology.kubernetes.io/zone=lab,node.kubernetes.io/instance-type=mono-gateway,mono.si/board=ls1046a
 
 # Optional
 # Extra API server SANs, comma-separated

devtools/build-all.sh

@@ -0,0 +1,248 @@
+#!/bin/bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+OUT_DIR="$(realpath "$SCRIPT_DIR/../out/")"
+
+cd "$SCRIPT_DIR/../"
+
+# ===== CONFIG TABLE =====
+# Add supported concrete builds here.
+# format: kube=crio
+CONFIGS=(
+  "v1.33.3=cri-o.arm64.v1.33.3"
+  "v1.34.1=cri-o.arm64.v1.34.1"
+  "v1.35.3=cri-o.arm64.v1.35.2"
+)
+
+# ===== HELPERS =====
+
+strip_v() {
+  echo "${1#v}"
+}
+
+minor_of_version() {
+  # input: 1.35.3 or v1.35.3
+  local v="${1#v}"
+  IFS='.' read -r major minor patch <<< "$v"
+  echo "$major.$minor"
+}
+
+version_sort() {
+  sort -t. -k1,1n -k2,2n -k3,3n
+}
+
+list_configs() {
+  echo "Available build targets:"
+  for c in "${CONFIGS[@]}"; do
+    local kube="${c%%=*}"
+    local crio="${c##*=}"
+    echo "  ${kube#v}  (CRI-O: $crio)"
+  done
+}
+
+list_minors() {
+  echo "Latest supported target per minor:"
+  latest_per_minor | while read -r kube crio; do
+    echo "  ${kube#v}  (CRI-O: $crio)"
+  done
+}
+
+latest_per_minor() {
+  # Output lines: <kube> <crio>
+  local minors
+  minors="$(
+    for c in "${CONFIGS[@]}"; do
+      kube="${c%%=*}"
+      minor_of_version "$kube"
+    done | sort -uV
+  )"
+
+  while read -r minor; do
+    [ -n "$minor" ] || continue
+    resolve_minor "$minor"
+  done <<< "$minors"
+}
+
+resolve_exact() {
+  local target="$1"
+  local target_v="v${target#v}"
+
+  for c in "${CONFIGS[@]}"; do
+    local kube="${c%%=*}"
+    local crio="${c##*=}"
+    if [[ "$kube" == "$target_v" ]]; then
+      echo "$kube $crio"
+      return 0
+    fi
+  done
+
+  return 1
+}
+
+resolve_minor() {
+  # input: 1.35 or v1.35
+  local want_minor
+  want_minor="$(minor_of_version "${1#v}.0")"
+
+  local matches=()
+  local c kube crio
+  for c in "${CONFIGS[@]}"; do
+    kube="${c%%=*}"
+    crio="${c##*=}"
+    if [[ "$(minor_of_version "$kube")" == "$want_minor" ]]; then
+      matches+=("${kube}=${crio}")
+    fi
+  done
+
+  if [[ ${#matches[@]} -eq 0 ]]; then
+    return 1
+  fi
+
+  local latest_kube
+  latest_kube="$(
+    for c in "${matches[@]}"; do
+      echo "${c%%=*#}" | sed 's/^v//'
+    done | version_sort | tail -n1
+  )"
+
+  resolve_exact "$latest_kube"
+}
+
+build_exact() {
+  local target="$1"
+  local resolved
+  if ! resolved="$(resolve_exact "$target")"; then
+    echo "❌ Unknown exact target: $target" >&2
+    echo >&2
+    list_configs >&2
+    exit 1
+  fi
+
+  local kube crio
+  read -r kube crio <<< "$resolved"
+
+  echo ">>> Building Kubernetes $kube with $crio"
+  make release CRIO_VERSION="$crio" KUBE_VERSION="$kube" 
+}
+
+build_minor() {
+  local minor="$1"
+  local resolved
+  if ! resolved="$(resolve_minor "$minor")"; then
+    echo "❌ No supported target found for minor: $minor" >&2
+    echo >&2
+    list_minors >&2
+    exit 1
+  fi
+
+  local kube crio
+  read -r kube crio <<< "$resolved"
+
+  echo ">>> Minor $minor resolved to latest supported target ${kube#v}"
+  echo ">>> Building Kubernetes $kube with $crio"
+  make release CRIO_VERSION="$crio" KUBE_VERSION="$kube" 
+}
+
+build_range() {
+  # input like 1.33-1.35
+  local range="$1"
+  local start="${range%-*}"
+  local end="${range#*-}"
+
+  local start_minor="${start#v}"
+  local end_minor="${end#v}"
+
+  local any=0
+  while read -r minor; do
+    [ -n "$minor" ] || continue
+
+    # simple lexical-safe because format is N.N and sort -V was used
+    if [[ "$(printf '%s\n%s\n' "$start_minor" "$minor" | sort -V | head -n1)" != "$start_minor" ]]; then
+      continue
+    fi
+    if [[ "$(printf '%s\n%s\n' "$minor" "$end_minor" | sort -V | head -n1)" != "$minor" ]]; then
+      continue
+    fi
+
+    any=1
+    build_minor "$minor"
+  done < <(
+    for c in "${CONFIGS[@]}"; do
+      kube="${c%%=*}"
+      minor_of_version "$kube"
+    done | sort -uV
+  )
+
+  if [[ $any -eq 0 ]]; then
+    echo "❌ No supported minors found in range: $range" >&2
+    echo >&2
+    list_minors >&2
+    exit 1
+  fi
+}
+
+build_all() {
+  local c kube crio
+  for c in "${CONFIGS[@]}"; do
+    kube="${c%%=*}"
+    crio="${c##*=}"
+    echo ">>> Building Kubernetes $kube with $crio"
+    make release CRIO_VERSION="$crio" KUBE_VERSION="$kube" 
+  done
+}
+
+usage() {
+  cat <<'EOF'
+Usage:
+  ./devtools/build-all.sh
+  ./devtools/build-all.sh list
+  ./devtools/build-all.sh list-minors
+  ./devtools/build-all.sh 1.35.3
+  ./devtools/build-all.sh 1.35
+  ./devtools/build-all.sh 1.33-1.35
+
+Behavior:
+  no args       Build all configured targets
+  list          List all exact configured targets
+  list-minors   List latest supported target per minor
+  X.Y.Z         Build exact version
+  X.Y           Build latest supported patch in that minor
+  X.Y-A.B       Build latest supported patch for each minor in range
+EOF
+}
+
+# ===== ENTRY =====
+
+if [[ $# -eq 0 ]]; then
+  echo "No target specified -> building all configured targets"
+  build_all
+  exit 0
+fi
+
+case "$1" in
+  list)
+    list_configs
+    ;;
+  list-minors)
+    list_minors
+    ;;
+  -*|--help|help)
+    usage
+    ;;
+  *-*)
+    build_range "$1"
+    ;;
+  *.*.*)
+    build_exact "$1"
+    ;;
+  *.*)
+    build_minor "$1"
+    ;;
+  *)
+    echo "❌ Unrecognized target: $1" >&2
+    echo >&2
+    usage >&2
+    exit 1
+    ;;
+esac

devtools/create-join-token.sh

@@ -0,0 +1,196 @@
+#!/bin/sh
+set -eu
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+ROOT_DIR="$(realpath "$SCRIPT_DIR/..")"
+LIB_DIR="$ROOT_DIR/scripts"
+CLUSTER_ENV_WORK="${CLUSTER_ENV_WORK:-$ROOT_DIR/configs/cluster.env.work}"
+
+KUBECTL="${KUBECTL:-kubectl}"
+TTL_HOURS="${TTL_HOURS:-24}"
+WAIT_SECONDS="${WAIT_SECONDS:-30}"
+
+need() {
+    command -v "$1" >/dev/null 2>&1 || {
+        echo "missing required command: $1" >&2
+        exit 1
+    }
+}
+
+rfc3339_after_hours() {
+    hours="$1"
+
+    # GNU date
+    if date -u -d "+${hours} hours" '+%Y-%m-%dT%H:%M:%SZ' >/dev/null 2>&1; then
+        date -u -d "+${hours} hours" '+%Y-%m-%dT%H:%M:%SZ'
+        return
+    fi
+
+    # BSD/macOS date
+    if date -u -v+"${hours}"H '+%Y-%m-%dT%H:%M:%SZ' >/dev/null 2>&1; then
+        date -u -v+"${hours}"H '+%Y-%m-%dT%H:%M:%SZ'
+        return
+    fi
+
+    echo "cannot compute expiration time with this date(1). Set EXPIRATION manually." >&2
+    exit 1
+}
+
+decode_base64_to_file() {
+    input="$1"
+    output="$2"
+
+    if printf '%s' "$input" | base64 -d >"$output" 2>/dev/null; then
+        return
+    fi
+
+    if printf '%s' "$input" | base64 -D >"$output" 2>/dev/null; then
+        return
+    fi
+
+    if printf '%s' "$input" | openssl base64 -d -A >"$output" 2>/dev/null; then
+        return
+    fi
+
+    echo "failed to decode certificate-authority-data" >&2
+    exit 1
+}
+
+need "$KUBECTL"
+need openssl
+need awk
+need sed
+
+TOKEN_ID="${TOKEN_ID:-$(openssl rand -hex 3)}"
+TOKEN_SECRET="${TOKEN_SECRET:-$(openssl rand -hex 8)}"
+TOKEN="${TOKEN_ID}.${TOKEN_SECRET}"
+SECRET_NAME="bootstrap-token-${TOKEN_ID}"
+
+if [ "${TTL_HOURS}" = "0" ]; then
+    EXPIRATION=""
+else
+    EXPIRATION="${EXPIRATION:-$(rfc3339_after_hours "$TTL_HOURS")}"
+fi
+
+echo "Creating bootstrap token Secret: ${SECRET_NAME}" >&2
+
+{
+    cat <<EOF
+apiVersion: v1
+kind: Secret
+metadata:
+  name: ${SECRET_NAME}
+  namespace: kube-system
+type: bootstrap.kubernetes.io/token
+stringData:
+  description: "Join token created with kubectl"
+  token-id: "${TOKEN_ID}"
+  token-secret: "${TOKEN_SECRET}"
+  usage-bootstrap-authentication: "true"
+  usage-bootstrap-signing: "true"
+  auth-extra-groups: "system:bootstrappers:kubeadm:default-node-token"
+EOF
+
+    if [ -n "$EXPIRATION" ]; then
+        printf '  expiration: "%s"\n' "$EXPIRATION"
+    fi
+} | "$KUBECTL" apply -f -
+
+TMPDIR="$(mktemp -d)"
+trap 'rm -rf "$TMPDIR"' EXIT INT TERM
+
+CA_FILE="$TMPDIR/ca.crt"
+
+CA_DATA="$("$KUBECTL" config view --raw --minify --flatten -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')"
+
+if [ -z "$CA_DATA" ]; then
+    echo "could not find certificate-authority-data in current kubeconfig" >&2
+    echo "token was created, but cannot print a safe kubeadm join command" >&2
+    echo "token: ${TOKEN}"
+    exit 0
+fi
+
+decode_base64_to_file "$CA_DATA" "$CA_FILE"
+
+CA_HASH="$(
+    openssl x509 -in "$CA_FILE" -pubkey -noout |
+        openssl pkey -pubin -outform der 2>/dev/null |
+        openssl dgst -sha256 -hex |
+        awk '{print $2}'
+)"
+
+SERVER="$("$KUBECTL" config view --raw --minify -o jsonpath='{.clusters[0].cluster.server}')"
+JOIN_ENDPOINT="$(printf '%s\n' "$SERVER" | sed -E 's#^https?://##')"
+
+echo "Waiting for cluster-info signature for token ${TOKEN_ID}..." >&2
+
+i=0
+signed="false"
+while [ "$i" -lt "$WAIT_SECONDS" ]; do
+    template="{{ index .data \"jws-kubeconfig-${TOKEN_ID}\" }}"
+    sig="$("$KUBECTL" -n kube-public get configmap cluster-info -o "go-template=${template}" 2>/dev/null || true)"
+
+    if [ -n "$sig" ]; then
+        signed="true"
+        break
+    fi
+
+    i=$((i + 1))
+    sleep 1
+done
+
+echo
+echo "Token:"
+echo "  ${TOKEN}"
+
+if [ -n "$EXPIRATION" ]; then
+    echo
+    echo "Expires:"
+    echo "  ${EXPIRATION}"
+fi
+
+echo
+echo "Join command:"
+echo "  kubeadm join ${JOIN_ENDPOINT} --token ${TOKEN} --discovery-token-ca-cert-hash sha256:${CA_HASH}"
+
+TMP_ENV="$(mktemp)"
+trap 'rm -f "$TMP_ENV"; rm -rf "$TMPDIR"' EXIT INT TERM
+
+cat >"$TMP_ENV" <<EOF
+MKS_API_SERVER_ENDPOINT=${JOIN_ENDPOINT}
+MKS_BOOTSTRAP_TOKEN=${TOKEN}
+MKS_DISCOVERY_TOKEN_CA_CERT_HASH=sha256:${CA_HASH}
+EOF
+
+echo
+echo "cluster-config:"
+cat "$TMP_ENV"
+
+if [ ! -x "$LIB_DIR/merge-env.sh" ]; then
+    echo "merge-env.sh not found or not executable: $LIB_DIR/merge-env.sh" >&2
+    exit 1
+fi
+
+"$LIB_DIR/merge-env.sh" "$TMP_ENV" "$CLUSTER_ENV_WORK"
+
+echo
+echo "Merged into:"
+echo "  $CLUSTER_ENV_WORK"
+echo
+echo "Try"
+cat <<EOF
+make cluster-config \\
+    MKS_HOSTNAME=monok8s-worker \\
+    MKS_CLUSTER_ROLE=worker \\
+    MKS_INIT_CONTROL_PLANE=no \\
+    MKS_MGMT_ADDRESS=10.0.0.10/24 \\
+    MKS_APISERVER_ADVERTISE_ADDRESS=10.0.0.10 \\
+    MKS_CNI_PLUGIN=none
+EOF
+
+
+if [ "$signed" != "true" ]; then
+    echo >&2
+    echo "warning: cluster-info was not signed within ${WAIT_SECONDS}s." >&2
+    echo "If kubeadm join fails discovery, check that kube-controller-manager enables bootstrapsigner." >&2
+fi

devtools/serve-images.sh

@@ -0,0 +1,6 @@
+#/bin/bash
+
+SCRIPT_DIR="$(dirname "${BASH_SOURCE[0]}")"
+OUT_DIR="$(realpath "$SCRIPT_DIR"/../out/)"
+
+python3 -m http.server 8000 --bind 0.0.0.0 --directory "$OUT_DIR"

devtools/setup-bulid-host.sh

@@ -0,0 +1,73 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+if [ "$(id -u)" -ne 0 ]; then
+    echo "Run as root, e.g. sudo $0" >&2
+    exit 1
+fi
+
+. /etc/os-release
+
+if [ "${ID:-}" != "debian" ]; then
+    echo "This script is intended for Debian. Detected ID=${ID:-unknown}" >&2
+    exit 1
+fi
+
+echo "==> Removing conflicting Docker packages, if present"
+apt-get remove -y \
+    docker.io \
+    docker-compose \
+    docker-doc \
+    podman-docker \
+    containerd \
+    runc || true
+
+echo "==> Installing minimal repo setup tools"
+apt-get update
+apt-get install -y --no-install-recommends \
+    ca-certificates \
+    curl
+
+echo "==> Adding Docker official APT repo"
+install -m 0755 -d /etc/apt/keyrings
+
+curl -fsSL https://download.docker.com/linux/debian/gpg \
+    -o /etc/apt/keyrings/docker.asc
+
+chmod a+r /etc/apt/keyrings/docker.asc
+
+cat > /etc/apt/sources.list.d/docker.sources <<EOF
+Types: deb
+URIs: https://download.docker.com/linux/debian
+Suites: ${VERSION_CODENAME}
+Components: stable
+Architectures: $(dpkg --print-architecture)
+Signed-By: /etc/apt/keyrings/docker.asc
+EOF
+
+echo "==> Installing build/test packages"
+apt-get update
+apt-get install -y --no-install-recommends \
+    docker-ce \
+    docker-buildx-plugin \
+    qemu-user-static \
+    binfmt-support \
+    make
+
+echo "==> Enabling Docker"
+systemctl enable --now docker
+
+echo "==> Registering binfmt handlers"
+systemctl restart binfmt-support || true
+
+echo "==> Docker version"
+docker --version
+
+echo "==> Buildx version"
+docker buildx version || true
+
+echo "==> Done"
+echo
+echo "Optional: allow your normal user to run docker without sudo:"
+echo "  sudo usermod -aG docker \$USER"
+echo "Then log out and back in."

devtools/test-upgrade.sh

@@ -0,0 +1,54 @@
+#!/bin/bash
+
+SCRIPT_DIR="$(dirname "${BASH_SOURCE[0]}")"
+OUT_DIR="$(realpath "$SCRIPT_DIR"/../out/)"
+
+set -e
+
+DEFAULT_BASE_URL="http://localhost:8000"
+DEFAULT_TARGET_VERSION="v1.34.1"
+STABLE_VERSION="v1.34.1"
+NAME="my-upgrade-1"
+
+if [ -r /dev/tty ]; then
+    printf "Enter the base url (%s): " "$DEFAULT_BASE_URL" > /dev/tty
+    read -r BASE_URL < /dev/tty
+
+    printf "Enter the target version (%s): " "$DEFAULT_TARGET_VERSION" > /dev/tty
+    read -r TARGET_VERSION < /dev/tty
+else
+    echo "No TTY available for interactive input" >&2
+    exit 1
+fi
+
+BASE_URL="${BASE_URL:-$DEFAULT_BASE_URL}"
+TARGET_VERSION="${TARGET_VERSION:-$DEFAULT_TARGET_VERSION}"
+
+echo "apiVersion: monok8s.io/v1alpha1"
+echo "kind: OSUpgrade"
+echo "metadata:"
+echo "  name: \"$NAME\""
+echo "spec:"
+echo "  desiredVersion: \"$TARGET_VERSION\""
+echo "  nodeSelector: {}"
+echo "  catalog:"
+echo "    inline: |"
+echo "      stable: $STABLE_VERSION"
+echo "      images:"
+
+for c in "$OUT_DIR"/catalog-*.txt; do
+    version=$(grep 'version:' "$c" | awk '{print $3}')
+    url=$(grep 'url:' "$c" | sed 's/.*"\(.*\)"/\1/')
+    checksum=$(grep 'checksum:' "$c" | awk '{print $2}')
+    size=$(grep 'size:' "$c" | awk '{print $2}')
+
+    filename=$(basename "$url")
+
+    echo "        - version: $version"
+    echo "          url: $BASE_URL/$filename"
+    echo "          checksum: sha256:$checksum"
+    echo "          size: $size"
+done
+
+echo "      blocked:"
+echo "        - v1.34.0"

docker/alpine.Dockerfile

@@ -27,8 +27,12 @@ RUN mkdir -p /out/rootfs/usr/local/bin/
 COPY packages/kubernetes/kubelet-${KUBE_VERSION} /out/rootfs/usr/local/bin/kubelet
 COPY packages/kubernetes/kubeadm-${KUBE_VERSION} /out/rootfs/usr/local/bin/kubeadm
 COPY packages/kubernetes/kubectl-${KUBE_VERSION} /out/rootfs/usr/local/bin/kubectl
-RUN chmod +x /out/rootfs/usr/local/bin/*
+# COPY clitools/out/dpdk/bin/dpdk-testpmd /out/rootfs/usr/local/bin/dpdk-testpmd
+# COPY clitools/out/dpdk/usr/local/lib/*.so* /out/rootfs/usr/local/lib/
+# COPY clitools/out/dpdk/usr/local/lib/dpdk/pmds-23.0/*.so* /out/rootfs/usr/local/lib/dpdk/pmds-23.0/
 
 COPY alpine/rootfs-extra ./rootfs-extra
+COPY alpine/migrations ./migrations
 COPY out/build-info ./rootfs-extra/etc/profile.d/build-info.sh
 COPY alpine/*.sh /
+RUN chmod +x /out/rootfs/usr/local/bin/*

docker/ask.Dockerfile

@@ -0,0 +1,148 @@
+ARG BUILD_BASE_TAG=dev
+ARG DOCKER_IMAGE_ROOT=monok8s
+
+FROM --platform=$BUILDPLATFORM ${DOCKER_IMAGE_ROOT}/build-base:${BUILD_BASE_TAG} AS build
+
+# Install glibc cross-compiler for kernel and standard build dependencies
+RUN apt-get update && apt-get install -y pkg-config
+
+WORKDIR /src
+
+ARG AARCH64_MUSL_CC_TAR
+ARG NXP_TAR
+ARG MONO_ASK_TAR
+ARG LIBNFNETLINK_TAR
+ARG LIBMNL_TAR
+ARG LIBNFCT_TAR
+ARG FMLIB_TAR
+ARG FMC_TAR
+ARG LIBXML2_TAR
+ARG LIBPCAP_TAR
+ARG LIBCLI_TAR
+ARG TCLAP_TAR
+
+# ASK's version pins (hardcoded wget)
+ARG LIBNFNETLINK_VERSION
+ARG LIBNFCT_VERSION
+
+# MUSL Cross Compiler
+COPY "${AARCH64_MUSL_CC_TAR}"  ./aarch64_musl_cc.tar.gz
+
+# Linux kernel
+COPY "${NXP_TAR}"          ./kernel.tar.gz
+
+# Copy the ASK deps
+COPY "${MONO_ASK_TAR}"     ./mono-ask.tar.gz
+COPY "${FMC_TAR}"          ./fmc.tar.gz
+COPY "${FMLIB_TAR}"        ./fmlib.tar.gz
+COPY "${LIBXML2_TAR}"      ./libxml2.tar.xz
+COPY "${LIBPCAP_TAR}"      ./libpcap.tar.xz
+COPY "${TCLAP_TAR}"        ./tclap.tar.gz
+COPY "${LIBMNL_TAR}"       ./libmnl.tar.bz2
+COPY "${LIBCLI_TAR}"       ./libcli.tar.gz
+# Pinned version should keep version names
+COPY "${LIBNFNETLINK_TAR}" ./libnfnetlink-${LIBNFNETLINK_VERSION}.tar.bz2
+COPY "${LIBNFCT_TAR}"      ./libnetfilter_conntrack-${LIBNFCT_VERSION}.tar.xz
+
+# Provision the musl cross-compiler from musl.cc
+RUN tar zxf "aarch64_musl_cc.tar.gz" -C /opt
+
+# Expose the musl compiler to the PATH
+ENV PATH="/opt/aarch64-linux-musl-cross/bin:${PATH}"
+
+# Extract and build the dependency libraries
+RUN mkdir -p ASK/sources/tarballs && \
+    tar zxf "mono-ask.tar.gz" -C "ASK" --strip-components=1 && \
+    mv libnfnetlink-${LIBNFNETLINK_VERSION}.tar.bz2 ASK/sources/tarballs/ && \
+    mv libnetfilter_conntrack-${LIBNFCT_VERSION}.tar.xz ASK/sources/tarballs/
+
+RUN mkdir linux && tar zxf "kernel.tar.gz" -C "linux" --strip-components=1
+
+# fmc & fmlib
+RUN mkdir -p ASK/sources/fmc && \
+    mkdir -p ASK/sources/fmlib && \
+    tar zxf "fmc.tar.gz" -C "ASK/sources/fmc" --strip-components=1 && \
+    tar zxf "fmlib.tar.gz" -C "ASK/sources/fmlib" --strip-components=1
+
+# tclap
+RUN mkdir -p tclap && tar zxf "tclap.tar.gz" -C "tclap" --strip-components=1 && \
+    cp -r tclap/include/tclap /opt/aarch64-linux-musl-cross/aarch64-linux-musl/include/ && \
+    rm -rf tclap
+
+# libxml2
+RUN mkdir -p libxml2 && tar xf "libxml2.tar.xz" -C "libxml2" --strip-components=1 && \
+    cd libxml2 && \
+    CC=aarch64-linux-musl-gcc ./configure --host=aarch64-linux-musl \
+        --prefix=/opt/aarch64-linux-musl-cross/aarch64-linux-musl \
+        --enable-static --disable-shared --without-python --without-zlib --without-lzma && \
+    make -j$(nproc) && make install && \
+    cd .. && rm -rf libxml2
+
+# libmnl
+RUN mkdir -p libmnl && tar xjf "libmnl.tar.bz2" -C "libmnl" --strip-components=1 && \
+    cd libmnl && \
+    CC=aarch64-linux-musl-gcc ./configure --host=aarch64-linux-musl \
+        --prefix=/opt/aarch64-linux-musl-cross/aarch64-linux-musl \
+        --enable-static --disable-shared && \
+    make -j$(nproc) && make install && \
+    cd .. && rm -rf libmnl
+
+# libcli
+RUN mkdir -p libcli && tar zxf "libcli.tar.gz" -C "libcli" --strip-components=1 && \
+    cd libcli && \
+    make CC=aarch64-linux-musl-gcc AR=aarch64-linux-musl-ar libcli.a && \
+    cp libcli.h /opt/aarch64-linux-musl-cross/aarch64-linux-musl/include/ && \
+    cp libcli.a /opt/aarch64-linux-musl-cross/aarch64-linux-musl/lib/ && \
+    cd .. && rm -rf libcli
+
+# libpcap
+RUN mkdir -p libpcap && tar xf "libpcap.tar.xz" -C "libpcap" --strip-components=1 && \
+    cd libpcap && \
+    CC=aarch64-linux-musl-gcc ./configure --host=aarch64-linux-musl \
+        --prefix=/opt/aarch64-linux-musl-cross/aarch64-linux-musl \
+        --with-pcap=linux --enable-static --disable-shared \
+        --disable-usb --disable-netmap --disable-bluetooth --disable-dbus && \
+    make -j$(nproc) && make install && \
+    cd .. && rm -rf libpcap
+
+WORKDIR /src/ASK
+
+COPY patches/mono-ask.mk .
+COPY kernel-extra.config /src/kernel-extra.config
+COPY kernel-build/ensure-kconfig.sh /src/ensure-kconfig.sh
+COPY kernel-build/dts/*.dts /src/linux/arch/arm64/boot/dts/freescale/
+
+# 1. This step patches the kernel, merges kernel-extra.config, and builds the modules
+RUN make -f mono-ask.mk modules
+
+# 2. This step cross-compiles fmlib, fmc, cmm, and dpa_app for Alpine
+RUN make -f mono-ask.mk userspace
+
+# 3. Stage the artifacts
+RUN make -f mono-ask.mk dist
+
+# 4. Stage the in-tree Linux kernel modules
+RUN mkdir -p /out/rootfs && \
+    make -C /src/linux ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \
+    modules_install INSTALL_MOD_PATH=/out/rootfs
+
+RUN KERNEL_VER=$(ls /out/rootfs/lib/modules/) && \
+    mkdir -p /out/rootfs/lib/modules/$KERNEL_VER/extra && \
+    mv /src/ASK/dist/*.ko /out/rootfs/lib/modules/$KERNEL_VER/extra/ && \
+    depmod -b /out/rootfs $KERNEL_VER && \
+    cd /out && tar zcf rootfs.tar.gz rootfs
+
+# Export stage
+FROM scratch AS export
+COPY --from=build /src/ASK/dist/ /ask/
+
+# Export the newly staged in-tree modules
+COPY --from=build /out/rootfs.tar.gz /
+COPY --from=build /src/linux/System.map /kernel/System.map
+COPY --from=build /src/linux/.config /kernel/.config
+COPY --from=build /src/linux/arch/arm64/boot/Image /kernel/Image
+COPY --from=build /src/linux/arch/arm64/boot/dts/freescale/mono-gateway-dk-sdk.dtb /kernel/mono-gateway-dk-sdk.dtb
+
+# Grab the proprietary Mono Gateway XML configs
+COPY --from=build /src/ASK/config/gateway-dk/cdx_cfg.xml /xml/cdx_cfg.xml
+COPY --from=build /src/ASK/dpa_app/files/etc/cdx_pcd.xml /xml/cdx_pcd.xml

docker/build-base.Dockerfile

@@ -4,47 +4,55 @@ ENV DEBIAN_FRONTEND=noninteractive
 
 WORKDIR /build
 
+ARG APT_PROXY
+
+RUN if [ -n "${APT_PROXY}" ]; then \
+      echo "Acquire::http::Proxy \"http://${APT_PROXY}\";" > /etc/apt/apt.conf.d/01proxy; \
+    fi
+
 RUN apt-get update && apt-get install -y --no-install-recommends \
     bash \
     bc \
+    binutils-aarch64-linux-gnu \
     bison \
     build-essential \
+    bzip2 \
+    ca-certificates \
     cpio \
-	ca-certificates \
     curl \
+    device-tree-compiler \
     dosfstools \
-    file \
-	fdisk \
-    fuse-overlayfs \
-	gdisk \
+    dwarves \
     e2fsprogs \
+    fdisk \
+    file \
     flex \
-    git \
+    fuse-overlayfs \
+    gcc-aarch64-linux-gnu \
+    gdisk \
     gettext-base \
-	jq \
+    git \
+    jq \
     kmod \
+    libc6-dev-arm64-cross \
+    libelf-dev \
     libelf-dev \
     libssl-dev \
+    linux-libc-dev-arm64-cross \
     make \
     pahole \
-	parted \
+    parted \
+    patch \
     perl \
-	pv \
+    podman \
+    pv \
     python3 \
-	qemu-user-static \
-	podman \
-	skopeo \
+    qemu-user-static \
     rsync \
+    skopeo \
     tar \
-	udev \
-    xz-utils \
-	zstd \
-    dwarves \
-    gcc-aarch64-linux-gnu \
-    binutils-aarch64-linux-gnu \
-    libc6-dev-arm64-cross \
-    linux-libc-dev-arm64-cross \
     u-boot-tools \
-    device-tree-compiler \
+    udev \
+    xz-utils \
+    zstd \
  && rm -rf /var/lib/apt/lists/*
-

docker/download-packages.Dockerfile

@@ -0,0 +1,180 @@
+FROM alpine:3.23.0 AS base
+RUN apk add --no-cache curl ca-certificates
+
+# ---- kubelet ----
+FROM base AS kubelet
+ARG KUBE_VERSION
+ARG ARCH
+WORKDIR /out/kubernetes
+RUN curl -fL --retry 3 -o "kubelet-${KUBE_VERSION}" \
+    "https://dl.k8s.io/${KUBE_VERSION}/bin/linux/${ARCH}/kubelet" && \
+    chmod +x "kubelet-${KUBE_VERSION}"
+
+# ---- kubeadm ----
+FROM base AS kubeadm
+ARG KUBE_VERSION
+ARG ARCH
+WORKDIR /out/kubernetes
+RUN curl -fL --retry 3 -o "kubeadm-${KUBE_VERSION}" \
+    "https://dl.k8s.io/${KUBE_VERSION}/bin/linux/${ARCH}/kubeadm" && \
+    chmod +x "kubeadm-${KUBE_VERSION}"
+
+# ---- kubectl ----
+FROM base AS kubectl
+ARG KUBE_VERSION
+ARG ARCH
+WORKDIR /out/kubernetes
+RUN curl -fL --retry 3 -o "kubectl-${KUBE_VERSION}" \
+    "https://dl.k8s.io/${KUBE_VERSION}/bin/linux/${ARCH}/kubectl" && \
+    chmod +x "kubectl-${KUBE_VERSION}"
+
+# ---- busybox ----
+FROM base AS busybox
+ARG BUSYBOX_VERSION
+WORKDIR /out
+RUN curl -fL --retry 3 -o "busybox-${BUSYBOX_VERSION}.tar.gz" \
+    "https://github.com/mirror/busybox/archive/refs/tags/${BUSYBOX_VERSION}.tar.gz"
+
+# ---- e2fsprogs ----
+FROM base AS e2fsprogs
+ARG E2FSPROGS_VERSION
+WORKDIR /out
+RUN curl -fL --retry 3 -o "e2fsprogs-v${E2FSPROGS_VERSION}.tar.gz" \
+    "https://github.com/tytso/e2fsprogs/archive/refs/tags/v${E2FSPROGS_VERSION}.tar.gz"
+
+# ---- dpdk ----
+FROM base AS dpdk
+ARG DPDK_VERSION
+WORKDIR /out/nxp/dpdk
+RUN curl -fL --retry 3 -o "${DPDK_VERSION}.tar.gz" \
+    "https://github.com/nxp-qoriq/dpdk/archive/refs/tags/${DPDK_VERSION}.tar.gz"
+
+# ---- fmlib ----
+FROM base AS fmlib
+ARG FMLIB_VERSION
+WORKDIR /out/nxp/fmlib
+RUN curl -fL --retry 3 -o "${FMLIB_VERSION}.tar.gz" \
+    "https://github.com/nxp-qoriq/fmlib/archive/refs/tags/${FMLIB_VERSION}.tar.gz"
+
+# ---- fmc ----
+FROM base AS fmc
+ARG FMC_VERSION
+WORKDIR /out/nxp/fmc
+RUN curl -fL --retry 3 -o "${FMC_VERSION}.tar.gz" \
+    "https://github.com/nxp-qoriq/fmc/archive/refs/tags/${FMC_VERSION}.tar.gz"
+
+# ---- vpp ----
+FROM base AS vpp
+ARG VPP_VERSION
+WORKDIR /out/nxp/vpp
+RUN curl -fL --retry 3 -o "${VPP_VERSION}.tar.gz" \
+    "https://github.com/nxp-qoriq/vpp/archive/refs/tags/${VPP_VERSION}.tar.gz"
+
+# ---- MUSL CC ----
+FROM base AS aarch64_musl_cc
+WORKDIR /out
+RUN curl -fL --retry 3 -o "aarch64-linux-musl-cross.tgz" \
+    "https://musl.cc/aarch64-linux-musl-cross.tgz"
+
+# ---- ASK ----
+FROM base AS mono_ask
+ARG MONO_ASK_VERSION
+WORKDIR /out/ask
+RUN curl -fL --retry 3 -o "${MONO_ASK_VERSION}.tar.gz" \
+    "https://github.com/we-are-mono/ASK/archive/refs/tags/${MONO_ASK_VERSION}.tar.gz"
+
+# ---- libnfnetlink ----
+FROM base AS libnfnetlink
+ARG LIBNFNETLINK_VERSION
+WORKDIR /out/ask/libnfnetlink
+RUN curl -fL --retry 3 -o "${LIBNFNETLINK_VERSION}.tar.bz2" \
+    "https://www.netfilter.org/projects/libnfnetlink/files/libnfnetlink-${LIBNFNETLINK_VERSION}.tar.bz2"
+
+# ---- libnfct ----
+FROM base AS libnfct
+ARG LIBNFCT_VERSION
+WORKDIR /out/ask/libnfct
+RUN curl -fL --retry 3 -o "${LIBNFCT_VERSION}.tar.xz" \
+    "https://www.netfilter.org/projects/libnetfilter_conntrack/files/libnetfilter_conntrack-${LIBNFCT_VERSION}.tar.xz"
+
+# ---- libmnl ----
+FROM base AS libmnl
+ARG LIBMNL_VERSION
+WORKDIR /out/ask/libmnl
+RUN curl -fL --retry 3 -o "${LIBMNL_VERSION}.tar.bz2" \
+    "https://www.netfilter.org/projects/libmnl/files/libmnl-${LIBMNL_VERSION}.tar.bz2"
+
+# ---- tclap ----
+FROM base AS tclap
+ARG TCLAP_VERSION
+WORKDIR /out/ask/tclap
+RUN curl -fL --retry 3 -o "${TCLAP_VERSION}.tar.gz" \
+    "https://sourceforge.net/projects/tclap/files/tclap-${TCLAP_VERSION}.tar.gz"
+
+# ---- libxml2 ----
+FROM base AS libxml2
+ARG LIBXML2_VERSION
+WORKDIR /out/ask/libxml2
+RUN curl -fL --retry 3 -o "${LIBXML2_VERSION}.tar.xz" \
+    "https://download.gnome.org/sources/libxml2/2.11/libxml2-${LIBXML2_VERSION}.tar.xz"
+
+# ---- libcli ----
+FROM base AS libcli
+ARG LIBCLI_VERSION
+WORKDIR /out/ask/libcli
+RUN curl -fL --retry 3 -o "${LIBCLI_VERSION}.tar.gz" \
+    "https://github.com/dparrish/libcli/archive/refs/tags/V${LIBCLI_VERSION}.tar.gz"
+
+# ---- libpcap ----
+FROM base AS libpcap
+ARG LIBPCAP_VERSION
+WORKDIR /out/ask/libpcap
+RUN curl -fL --retry 3 -o "${LIBPCAP_VERSION}.tar.xz" \
+    "https://www.tcpdump.org/release/libpcap-${LIBPCAP_VERSION}.tar.xz"
+
+# ---- alpine rootfs ----
+FROM base AS alpine_rootfs
+ARG ALPINE_SERIES
+ARG ALPINE_ARCH
+ARG ALPINE_VER
+WORKDIR /out
+RUN curl -fL --retry 3 -o "alpine-minirootfs-${ALPINE_VER}-${ALPINE_ARCH}.tar.gz" \
+    "https://dl-cdn.alpinelinux.org/alpine/v${ALPINE_SERIES}/releases/${ALPINE_ARCH}/alpine-minirootfs-${ALPINE_VER}-${ALPINE_ARCH}.tar.gz"
+
+# ---- nxp linux ----
+FROM base AS nxp_linux
+ARG NXP_VERSION
+WORKDIR /out/nxp/kernel
+RUN curl -fL --retry 3 -o "${NXP_VERSION}.tar.gz" \
+    "https://github.com/nxp-qoriq/linux/archive/refs/tags/${NXP_VERSION}.tar.gz"
+
+# ---- crio ----
+FROM base AS crio
+ARG CRIO_VERSION
+WORKDIR /out
+RUN curl -fL --retry 3 -o "${CRIO_VERSION}.tar.gz" \
+    "https://storage.googleapis.com/cri-o/artifacts/${CRIO_VERSION}.tar.gz"
+
+# ---- final exported artifact set ----
+FROM scratch
+COPY --from=kubelet /out/ /
+COPY --from=kubeadm /out/ /
+COPY --from=kubectl /out/ /
+COPY --from=busybox /out/ /
+COPY --from=e2fsprogs /out/ /
+COPY --from=dpdk /out/ /
+COPY --from=aarch64_musl_cc /out/ /
+COPY --from=mono_ask /out/ /
+COPY --from=vpp /out/ /
+COPY --from=fmlib /out/ /
+COPY --from=fmc /out/ /
+COPY --from=libnfnetlink /out/ /
+COPY --from=libnfct /out/ /
+COPY --from=libmnl /out/ /
+COPY --from=libcli /out/ /
+COPY --from=libpcap /out/ /
+COPY --from=libxml2 /out/ /
+COPY --from=tclap /out/ /
+COPY --from=alpine_rootfs /out/ /
+COPY --from=nxp_linux /out/ /
+COPY --from=crio /out/ /

docker/itb.Dockerfile

@@ -1,7 +1,5 @@
 ARG TAG=dev
 ARG DOCKER_IMAGE_ROOT=monok8s
-FROM ${DOCKER_IMAGE_ROOT}/kernel-build:${TAG} AS kernel
-FROM ${DOCKER_IMAGE_ROOT}/fit-build:${TAG} AS fit
 
 FROM --platform=$BUILDPLATFORM ${DOCKER_IMAGE_ROOT}/build-base:${TAG} AS build
 
@@ -11,8 +9,6 @@ RUN mkdir /image
 WORKDIR /image
 
 COPY [ "./out/Image.gz" \
-	, "./out/System.map" \
-	, "./out/.config" \
 	, "./out/initramfs.cpio.gz" \
 	, "./out/${DEVICE_TREE_TARGET}.dtb", "./" ]
 

docker/kernel-build.Dockerfile

@@ -14,7 +14,7 @@ RUN test -n "${NXP_VERSION}" || (echo "Please specify NXP_VERSION" >&2; exit 1);
     test -n "${CROSS_COMPILE}" || (echo "Please specify CROSS_COMPILE" >&2; exit 1); \
     test -n "${DEVICE_TREE_TARGET}" || (echo "Please specify DEVICE_TREE_TARGET" >&2; exit 1)
 
-COPY packages/${NXP_VERSION}.tar.gz ./
+COPY packages/nxp/kernel/${NXP_VERSION}.tar.gz ./
 RUN tar -xf ${NXP_VERSION}.tar.gz \
  && mv "linux-${NXP_VERSION}" nxplinux \
  && rm -f ${NXP_VERSION}.tar.gz
@@ -23,6 +23,7 @@ WORKDIR /build/nxplinux
 
 COPY kernel-extra.config /tmp/kernel-extra.config
 COPY kernel-build/dts/*.dts ./arch/arm64/boot/dts/freescale/
+COPY kernel-build/ensure-kconfig.sh /build/
 
 RUN grep -q "^dtb-\\\$(CONFIG_ARCH_LAYERSCAPE) += ${DEVICE_TREE_TARGET}.dtb$" \
       arch/arm64/boot/dts/freescale/Makefile \
@@ -33,7 +34,7 @@ RUN grep -q "^dtb-\\\$(CONFIG_ARCH_LAYERSCAPE) += ${DEVICE_TREE_TARGET}.dtb$" \
 RUN make ARCH="${ARCH}" CROSS_COMPILE="${CROSS_COMPILE}" defconfig lsdk.config \
  && ./scripts/kconfig/merge_config.sh -m .config /tmp/kernel-extra.config \
  && make ARCH="${ARCH}" CROSS_COMPILE="${CROSS_COMPILE}" olddefconfig \
- && grep '^CONFIG_NF_TABLES=' .config \
+ && /build/ensure-kconfig.sh .config /tmp/kernel-extra.config \
  && make ARCH="${ARCH}" CROSS_COMPILE="${CROSS_COMPILE}" -j"$(nproc)"
 
 # artifact collection