monok8s/docs/ota.md

Upgrade process

We use a CRD with an agent to handle this. Our versions follows upstream's.

To issue an upgrade. Simply use

kubectl apply -f upgrade.yaml

apiVersion: monok8s.io/v1alpha1
kind: OSUpgrade
metadata:
  name: "my-ugrade-2"
spec:
  version: "v1.35.3"
  imageURL: "https://updates.example.com/monok8s-1.2.3.img.zst"
  checksum: "sha256:..."
  nodeSelector: {}
  catalog:
    inline: |
      stable: v1.35.1
      images:
        - version: v1.34.6
          url: https://example.invalid/images/monok8s-v1.34.6.img.zst
          checksum: sha256:abc
        - version: v1.34.1
          url: https://example.invalid/images/monok8s-v1.34.1.img.zst
          checksum: sha256:abc
        - version: v1.35.0
          url: https://example.invalid/images/monok8s-v1.35.0.img.zst
          checksum: sha256:ghi
        - version: v1.35.4
          url: https://example.invalid/images/monok8s-v1.35.4.img.zst
          checksum: sha256:jkl
        - version: v1.35.1
          url: http://localhost:8000/rootfs.ext4.zst
          checksum: sha256:99af82a263deca44ad91d21d684f0fa944d5d0456a1da540f1c644f8aa59b14b
          size: 1858076672 # expanded image size in bytes, use "zstd -lv image.zst to check"
      blocked:
        - v1.34.0

catalog accepts URL or ConfigMap

catalog:
  URL: https://example.com/images.yaml

catalog:
  ConfigMap: images-cm

Contents should look like this

stable: v1.35.1
images:
  - version: v1.34.6
    url: https://example.invalid/images/monok8s-v1.34.6.img.zst
    checksum: sha256:abc
  - version: v1.34.1
    url: https://example.invalid/images/monok8s-v1.34.1.img.zst
    checksum: sha256:abc
  - version: v1.35.0
    url: https://example.invalid/images/monok8s-v1.35.0.img.zst
    checksum: sha256:ghi
  - version: v1.35.4
    url: https://example.invalid/images/monok8s-v1.35.4.img.zst
    checksum: sha256:jkl
  - version: v1.35.1
    url: http://localhost:8000/rootfs.ext4.zst
    checksum: sha256:99af82a263deca44ad91d21d684f0fa944d5d0456a1da540f1c644f8aa59b14b
    size: 1858076672 # expanded image size in bytes, use "zstd -lv image.zst to check"
blocked:
  - v1.34.0

Monitoring the upgrades

kubectl get osugrades

NAME            DESIRED    RESOLVED   PHASE       TARGETS   OK   FAIL   AGE
my-upgrade-3    stable     v1.35.4    RollingOut  3         1    0      1m
my-upgrade-2    v1.35.3    v1.35.3    Accepted    2         0    0      1m
my-downgrade-1  v1.33.2    v1.33.2    Rejected    2         0    2      1m

kubectl get osupgradeprogress

NAME                NODE        SOURCE        CURRENT  TARGET   STATUS
osupgrade-abc123f   node-1      my-upgrade-2  v1.34.1  v1.35.3  downloading
osupgrade-cde456g   node-2      my-upgrade-2  v1.35.3  v1.35.3  completed

kubectl describe osupgradeprogress osupgrade-abc123f

apiVersion: monok8s.io/v1alpha1
kind: OSUpgradeProgress
metadata:
  name: "osupgrade-abc123f"
spec:
  sourceRef:
    name: my-upgrade-2
  nodeName: node-1
status:
  currentVersion: "v1.34.1"
  targetVersion: "v1.35.3"
  phase: Downloading
  startedAt: null
  completedAt: null
  lastUpdatedAt: null
  retryCount: 0
  inactivePartition: "B"
  failureReason: ""
  message: ""

Development notes

Simulate OTA

Use nmap ncat. Otherwise we'll have all kinds of fabulous issues sending it.

Sending side

pv "out/rootfs.ext4.zst" | ncat 10.0.0.10 1234 --send-only

Receiving side

ncat -l 1234 --recv-only | zstd -d -c | dd of=/dev/sda3 bs=4M status=progress && sync && echo "SUCCESS"