Refine controller template and probe listeners

docs/installing-ssh-pod.md

@@ -0,0 +1,204 @@
+# Installing the recovery SSHD pod
+
+This page explains how to install a temporary SSH server pod for break-glass recovery.
+
+Use this when normal Kubernetes access is degraded, for example after the API server certificate expires or rotates and you need to retrieve updated host-side credentials.
+
+The SSHD pod is intended for recovery and debugging only. Remove it when you are done.
+
+## What this does
+
+The recovery pod starts an SSH server on the selected node and authorizes your local SSH public key.
+
+The pod also mounts selected host paths under `/host`, so you can inspect the host filesystem and run some host-side recovery commands through `chroot`.
+
+For example:
+
+```sh
+chroot /host /bin/sh -lc 'rc-status'
+chroot /host /bin/sh -lc 'rc-service crio status'
+chroot /host /bin/sh -lc 'rc-service kubelet status'
+```
+
+## Requirements
+
+You need:
+
+- A working `kubectl` connection to the cluster.
+- Access to the `node-agent` DaemonSet in the `mono-system` namespace.
+- A local SSH public key, usually `~/.ssh/id_rsa.pub` or `~/.ssh/id_ed25519.pub`.
+
+Use a public key file only. Do not pass your private key.
+
+## Generate the SSHD manifest
+
+To print the recovery SSHD manifest:
+
+```bash
+kubectl exec -i -n mono-system ds/node-agent -- \
+  ctl create sshd --authkeys /dev/stdin < ~/.ssh/id_rsa.pub
+```
+
+This reads your local public key and places it into the generated pod's `authorized_keys`.
+
+If you use Ed25519 keys, use:
+
+```bash
+kubectl exec -i -n mono-system ds/node-agent -- \
+  ctl create sshd --authkeys /dev/stdin < ~/.ssh/id_ed25519.pub
+```
+
+## Generate and apply the manifest
+
+To create the recovery SSHD resources in one step:
+
+```bash
+kubectl exec -i -n mono-system ds/node-agent -- \
+  ctl create sshd --authkeys /dev/stdin < ~/.ssh/id_rsa.pub \
+  | kubectl apply -f -
+```
+
+For Ed25519:
+
+```bash
+kubectl exec -i -n mono-system ds/node-agent -- \
+  ctl create sshd --authkeys /dev/stdin < ~/.ssh/id_ed25519.pub \
+  | kubectl apply -f -
+```
+
+## Why `-i` is used instead of `-it`
+
+Use `-i`, not `-it`, when piping the SSH public key.
+
+The `-t` option allocates a pseudo-TTY. A pseudo-TTY can modify piped input, which is not what you want when passing an SSH public key through stdin.
+
+Correct:
+
+```bash
+kubectl exec -i -n mono-system ds/node-agent -- \
+  ctl create sshd --authkeys /dev/stdin < ~/.ssh/id_rsa.pub
+```
+
+Avoid:
+
+```bash
+kubectl exec -it -n mono-system ds/node-agent -- \
+  ctl create sshd --authkeys /dev/stdin < ~/.ssh/id_rsa.pub
+```
+
+## Check that the pod is running
+
+After applying the manifest, check the pod:
+
+```bash
+kubectl get pods -n mono-system -l app.kubernetes.io/name=sshd
+```
+
+Check the service:
+
+```bash
+kubectl get svc -n mono-system -l app.kubernetes.io/name=sshd
+```
+
+If the pod does not start, inspect it:
+
+```bash
+kubectl describe pod -n mono-system -l app.kubernetes.io/name=sshd
+```
+
+## Connect through SSH
+
+The exact SSH command depends on how the generated service exposes the pod.
+
+If the service uses a NodePort such as `30022`, connect with:
+
+```bash
+ssh -p 30022 root@<node-ip>
+```
+
+Replace `<node-ip>` with the node's reachable IP address.
+
+## Access the host environment
+
+Inside the SSH session, the host filesystem is available under `/host`.
+
+Useful checks:
+
+```sh
+ls -la /host
+chroot /host /bin/sh -lc 'rc-status'
+chroot /host /bin/sh -lc 'rc-service crio status'
+chroot /host /bin/sh -lc 'rc-service kubelet status'
+```
+
+Restart CRI-O:
+
+```sh
+chroot /host /bin/sh -lc 'rc-service crio restart'
+```
+
+Restart kubelet:
+
+```sh
+chroot /host /bin/sh -lc 'rc-service kubelet restart'
+```
+
+You can also inspect host processes from the pod because the recovery pod uses the host PID namespace:
+
+```sh
+ps aux | grep -E 'kubelet|crio'
+```
+
+## Notes for monok8s host mounts
+
+The recovery pod does not mount host `/` directly.
+
+On monok8s, `/` and `/var` may be private mounts. Mounting them directly as host paths can fail with errors such as:
+
+```text
+path "/" is mounted on "/" but it is not a shared or slave mount
+```
+
+or:
+
+```text
+path "/var" is mounted on "/var" but it is not a shared or slave mount
+```
+
+Instead, the recovery pod assembles a minimal host root under `/host` from individual host paths.
+
+For `/var`, it uses the backing path:
+
+```text
+/data/var -> /host/var
+```
+
+This avoids the private bind-mount issue.
+
+## Remove the recovery pod
+
+When recovery is complete, remove the generated resources.
+
+If the resources use the default SSHD labels:
+
+```bash
+kubectl delete deployment -n mono-system -l app.kubernetes.io/name=sshd
+kubectl delete service -n mono-system -l app.kubernetes.io/name=sshd
+kubectl delete configmap -n mono-system -l app.kubernetes.io/name=sshd
+```
+
+If your generated manifest uses a fixed resource name, you can also remove them by name:
+
+```bash
+kubectl delete deployment -n mono-system sshd
+kubectl delete service -n mono-system sshd
+kubectl delete configmap -n mono-system sshd-authorized-keys
+```
+
+## Security warning
+
+This pod is powerful.
+
+It runs with root-level recovery access and can inspect or modify host files through `/host`. Treat it as a temporary break-glass tool, not a normal service.
+
+Do not leave it running after recovery.