# Installing the recovery SSHD pod

This page explains how to install a temporary SSH server pod for break-glass recovery.

Use this when normal Kubernetes access is degraded, for example after the API server certificate expires or rotates and you need to retrieve updated host-side credentials.

The SSHD pod is intended for recovery and debugging only. Remove it when you are done.

## What this does

The recovery pod starts an SSH server on the selected node and authorizes your local SSH public key.

The pod also mounts selected host paths under `/host`, so you can inspect the host filesystem and run some host-side recovery commands through `chroot`.

For example:

```sh
chroot /host /bin/sh -lc 'rc-status'
chroot /host /bin/sh -lc 'rc-service crio status'
chroot /host /bin/sh -lc 'rc-service kubelet status'
```

## Requirements

You need:

- A working `kubectl` connection to the cluster.
- Access to the `node-agent` DaemonSet in the `mono-system` namespace.
- A local SSH public key, usually `~/.ssh/id_rsa.pub` or `~/.ssh/id_ed25519.pub`.

Use a public key file only. Do not pass your private key.

## Generate the SSHD manifest

To print the recovery SSHD manifest:

```bash
kubectl exec -i -n mono-system ds/node-agent -- \
  ctl create sshd --authkeys /dev/stdin < ~/.ssh/id_rsa.pub
```

This reads your local public key and places it into the generated pod's `authorized_keys`.

If you use Ed25519 keys, use:

```bash
kubectl exec -i -n mono-system ds/node-agent -- \
  ctl create sshd --authkeys /dev/stdin < ~/.ssh/id_ed25519.pub
```

## Generate and apply the manifest

To create the recovery SSHD resources in one step:

```bash
kubectl exec -i -n mono-system ds/node-agent -- \
  ctl create sshd --authkeys /dev/stdin < ~/.ssh/id_rsa.pub \
  | kubectl apply -f -
```

For Ed25519:

```bash
kubectl exec -i -n mono-system ds/node-agent -- \
  ctl create sshd --authkeys /dev/stdin < ~/.ssh/id_ed25519.pub \
  | kubectl apply -f -
```

## Why `-i` is used instead of `-it`

Use `-i`, not `-it`, when piping the SSH public key.

The `-t` option allocates a pseudo-TTY. A pseudo-TTY can modify piped input, which is not what you want when passing an SSH public key through stdin.

Correct:

```bash
kubectl exec -i -n mono-system ds/node-agent -- \
  ctl create sshd --authkeys /dev/stdin < ~/.ssh/id_rsa.pub
```

Avoid:

```bash
kubectl exec -it -n mono-system ds/node-agent -- \
  ctl create sshd --authkeys /dev/stdin < ~/.ssh/id_rsa.pub
```

## Check that the pod is running

After applying the manifest, check the pod:

```bash
kubectl get pods -n mono-system -l app.kubernetes.io/name=sshd
```

Check the service:

```bash
kubectl get svc -n mono-system -l app.kubernetes.io/name=sshd
```

If the pod does not start, inspect it:

```bash
kubectl describe pod -n mono-system -l app.kubernetes.io/name=sshd
```

## Connect through SSH

The exact SSH command depends on how the generated service exposes the pod.

If the service uses a NodePort such as `30022`, connect with:

```bash
ssh -p 30022 root@<node-ip>
```

Replace `<node-ip>` with the node's reachable IP address.

## Access the host environment

Inside the SSH session, the host filesystem is available under `/host`.

Useful checks:

```sh
ls -la /host
chroot /host /bin/sh -lc 'rc-status'
chroot /host /bin/sh -lc 'rc-service crio status'
chroot /host /bin/sh -lc 'rc-service kubelet status'
```

Restart CRI-O:

```sh
chroot /host /bin/sh -lc 'rc-service crio restart'
```

Restart kubelet:

```sh
chroot /host /bin/sh -lc 'rc-service kubelet restart'
```

You can also inspect host processes from the pod because the recovery pod uses the host PID namespace:

```sh
ps aux | grep -E 'kubelet|crio'
```

## Notes for monok8s host mounts

The recovery pod does not mount host `/` directly.

On monok8s, `/` and `/var` may be private mounts. Mounting them directly as host paths can fail with errors such as:

```text
path "/" is mounted on "/" but it is not a shared or slave mount
```

or:

```text
path "/var" is mounted on "/var" but it is not a shared or slave mount
```

Instead, the recovery pod assembles a minimal host root under `/host` from individual host paths.

For `/var`, it uses the backing path:

```text
/data/var -> /host/var
```

This avoids the private bind-mount issue.

## Remove the recovery pod

When recovery is complete, remove the generated resources.

If the resources use the default SSHD labels:

```bash
kubectl delete deployment -n mono-system -l app.kubernetes.io/name=sshd
kubectl delete service -n mono-system -l app.kubernetes.io/name=sshd
kubectl delete configmap -n mono-system -l app.kubernetes.io/name=sshd
```

If your generated manifest uses a fixed resource name, you can also remove them by name:

```bash
kubectl delete deployment -n mono-system sshd
kubectl delete service -n mono-system sshd
kubectl delete configmap -n mono-system sshd-authorized-keys
```

## Security warning

This pod is powerful.

It runs with root-level recovery access and can inspect or modify host files through `/host`. Treat it as a temporary break-glass tool, not a normal service.

Do not leave it running after recovery.