diff --git a/README.md b/README.md index 406a962..2db0fe6 100644 --- a/README.md +++ b/README.md @@ -1,19 +1,25 @@ # monok8s - Alpine-based Kubernetes cluster image for Mono's Gateway Development Kit https://docs.mono.si/gateway-development-kit/getting-started -## DISCLAIMER -USE AT YOUR OWN RISKS. I leverage ChatGPT heavily for this. I am testing them all by myself right now. +## Features + * A/B deployment + * Read-only OS + * k8s style OS upgrade (see Upgrading) ## IMPORTANT NOTES * The 3 RJ45 ports are label in eth1, eth2, eth0 respectively by the kernel (left to right) * So `ip addr eth0` is your right most port * If the fan stopped spinning. Unplug ASAP! Otherwise CPU temp goes to the moon. -## Build +### Table of Contents +1. Flashing + - [USB](docs/flashing-usb.md) + - [Over network (tftp)](docs/flashing-network.md) +2. [Upgrading](docs/ota.md) +## Build Find the latest package versions and update build.env * [kernel](https://github.com/nxp-qoriq/linux/archive/refs/tags/) * [busybox](https://github.com/mirror/busybox/archive/refs/tags/) @@ -25,131 +31,24 @@ Then run make release ``` -Then run, follow the instructions +Inspect and modify config/cluster.env, then run ``` make cluster-config ``` -## Flashing - -#### USB Drive -1. `make release` -2. Format the USB in vfat -3. Copy out/[RELEASE].img.gz and out/board.itb to your usb drive's root -4. Run -``` -usb start -usb tree -fatls usb 0:1 # For fat -ext4ls usb 0:1 # For ext4 if you insist -fatload usb 0 0x80000000 board.itb - -setenv bootargs "console=ttyS0,115200 earlycon=uart8250,mmio,0x21c0500 root=/dev/ram0 rootwait rw" -bootm 0x80000000 -``` - -5. Inside initramfs, run -``` -flash-emmc.sh -``` - -4. If it boots, create the A/B deployment scheme - - (WORK IN PROGRESS) - -#### tftp (network is required) -1. Put out/[RELEASE].img.gz and out/board.itb into your ftp server -``` -setenv ipaddr 10.0.0.153 -setenv serverip 10.0.0.129 -tftp 0x80000000 board.itb - -setenv bootargs "console=ttyS0,115200 earlycon=uart8250,mmio,0x21c0500 root=/dev/ram0 rootwait rw" -bootm 0x80000000 -``` -2. Inside initramfs, run -``` -flash-emmc.sh -``` - - ### Making sub stages ```bash -make build-base # The image the builds kernel and everything +make build-base # The image that builds the kernel and everything make kernel # Builds our kernel from NXP make initramfs make itb # Builds out/board.itb (contains the kernel and the initramfs) ``` -## Architecture - - A/B deployment (So we can just do kubectl apply upgrade.yaml with version jumps. Agent will walk through it automatically) - - Read-only OS - -## Upgrade process - -We use a CRD with an agent to handle this. Our versions follows upstream's. - -kubectl apply -f upgrade.yaml -```yaml -apiVersion: monok8s.io/v1alpha1 -kind: OSUpgrade -metadata: - name: "my-ugrade-2" -spec: - version: "v1.35.3" - imageURL: "https://updates.example.com/monok8s-1.2.3.img.zst" - checksum: "sha256:..." - nodeSelector: {} -status: - phase: Pending - resolvedVersion: "v1.35.3" - observedGeneration: 1 - summary: - targetedNodes: 0 - pendingNodes: 0 - runningNodes: 0 - succeededNodes: 0 - failedNodes: 0 -``` - -kubectl get osugrades -``` -NAME DESIRED RESOLVED PHASE TARGETS OK FAIL AGE -my-upgrade-3 stable v1.35.4 RollingOut 3 1 0 1m -my-upgrade-2 v1.35.3 v1.35.3 Accepted 2 0 0 1m -my-downgrade-1 v1.33.2 v1.33.2 Rejected 2 0 2 1m -``` - -kubectl get osupgradeprogress -``` -NAME NODE SOURCE CURRENT TARGET STATUS -osupgrade-abc123f node-1 my-upgrade-2 v1.34.1 v1.35.3 downloading -osupgrade-cde456g node-2 my-upgrade-2 v1.35.3 v1.35.3 completed -``` - -```yaml -apiVersion: monok8s.io/v1alpha1 -kind: OSUpgradeProgress -metadata: - name: "osupgrade-abc123f" -spec: - sourceRef: - name: my-upgrade-2 - nodeName: node-1 -status: - currentVersion: "v1.34.1" - targetVersion: "v1.35.3" - phase: Downloading - startedAt: null - completedAt: null - lastUpdatedAt: null - retryCount: 0 - inactivePartition: "B" - failureReason: "" - message: "" -``` - ## NOTES ### The device's dts files are located at here https://github.com/we-are-mono/OpenWRT-ASK/tree/mono-25.12.0-rc3/target/linux/layerscape/files/arch/arm64/boot/dts/freescale * We need both `mono-gateway-dk-sdk.dts` and `mono-gateway-dk.dts` since the sdk one includes the non-sdk one. * The actual dts being used is the `mono-gateway-dk-sdk.dts` + +## DISCLAIMER +USE AT YOUR OWN RISKS. I leverage ChatGPT heavily for this. diff --git a/docs/flashing-network.md b/docs/flashing-network.md new file mode 100644 index 0000000..5a45948 --- /dev/null +++ b/docs/flashing-network.md @@ -0,0 +1,17 @@ +Mostly just the same as tutorial from mono + +## tftp (network is required) +1. Put out/[RELEASE].img.gz and out/board.itb into your ftp server +``` +setenv ipaddr 10.0.0.153 +setenv serverip 10.0.0.129 +tftp 0x80000000 board.itb + +setenv bootargs "console=ttyS0,115200 earlycon=uart8250,mmio,0x21c0500 root=/dev/ram0 rootwait rw" +bootm 0x80000000 +``` + +2. Inside initramfs, run +``` +flash-emmc.sh +``` diff --git a/docs/flashing-usb.md b/docs/flashing-usb.md new file mode 100644 index 0000000..1b1e782 --- /dev/null +++ b/docs/flashing-usb.md @@ -0,0 +1,33 @@ +We have two type of flashing +1. Flash the image directly into USB. And boot using it. +2. Copy the files into the USB drive. Then boot into initramfs. Then flash the image into the eMMC. + +## Flashing into eMMC +1. `make release` +2. Format the USB in vfat +3. Copy out/[RELEASE].img.gz and out/board.itb to your usb drive's root +4. Run +``` +usb start +usb tree +fatls usb 0:1 # For fat +ext4ls usb 0:1 # For ext4 if you insist +fatload usb 0 0x80000000 board.itb + +setenv bootargs "console=ttyS0,115200 earlycon=uart8250,mmio,0x21c0500 root=/dev/ram0 rootwait rw" +bootm 0x80000000 +``` + +5. Inside initramfs, run +``` +flash-emmc.sh +``` + +4. If it boots, create the A/B deployment scheme + - (WORK IN PROGRESS) + +## Flashing into USB +On MacOS +1. ./macos/flashusb.sh + +On Windows (Work In Progress) diff --git a/docs/ota.md b/docs/ota.md index 685f545..2f4f41a 100644 --- a/docs/ota.md +++ b/docs/ota.md @@ -1,3 +1,120 @@ +## Upgrade process + +We use a CRD with an agent to handle this. Our versions follows upstream's. + +To issue an upgrade. Simply use + +kubectl apply -f upgrade.yaml +```yaml +apiVersion: monok8s.io/v1alpha1 +kind: OSUpgrade +metadata: + name: "my-ugrade-2" +spec: + version: "v1.35.3" + imageURL: "https://updates.example.com/monok8s-1.2.3.img.zst" + checksum: "sha256:..." + nodeSelector: {} + catalog: + inline: | + stable: v1.35.1 + images: + - version: v1.34.6 + url: https://example.invalid/images/monok8s-v1.34.6.img.zst + checksum: sha256:abc + - version: v1.34.1 + url: https://example.invalid/images/monok8s-v1.34.1.img.zst + checksum: sha256:abc + - version: v1.35.0 + url: https://example.invalid/images/monok8s-v1.35.0.img.zst + checksum: sha256:ghi + - version: v1.35.4 + url: https://example.invalid/images/monok8s-v1.35.4.img.zst + checksum: sha256:jkl + - version: v1.35.1 + url: http://localhost:8000/rootfs.ext4.zst + checksum: sha256:99af82a263deca44ad91d21d684f0fa944d5d0456a1da540f1c644f8aa59b14b + size: 1858076672 # expanded image size in bytes, use "zstd -lv image.zst to check" + blocked: + - v1.34.0 +``` + +catalog accepts URL or ConfigMap +```yaml +catalog: + URL: https://example.com/images.yaml + +catalog: + ConfigMap: images-cm +``` + +Contents should look like this +```yaml +stable: v1.35.1 +images: + - version: v1.34.6 + url: https://example.invalid/images/monok8s-v1.34.6.img.zst + checksum: sha256:abc + - version: v1.34.1 + url: https://example.invalid/images/monok8s-v1.34.1.img.zst + checksum: sha256:abc + - version: v1.35.0 + url: https://example.invalid/images/monok8s-v1.35.0.img.zst + checksum: sha256:ghi + - version: v1.35.4 + url: https://example.invalid/images/monok8s-v1.35.4.img.zst + checksum: sha256:jkl + - version: v1.35.1 + url: http://localhost:8000/rootfs.ext4.zst + checksum: sha256:99af82a263deca44ad91d21d684f0fa944d5d0456a1da540f1c644f8aa59b14b + size: 1858076672 # expanded image size in bytes, use "zstd -lv image.zst to check" +blocked: + - v1.34.0 +``` + +### Monitoring the upgrades + +kubectl get osugrades +``` +NAME DESIRED RESOLVED PHASE TARGETS OK FAIL AGE +my-upgrade-3 stable v1.35.4 RollingOut 3 1 0 1m +my-upgrade-2 v1.35.3 v1.35.3 Accepted 2 0 0 1m +my-downgrade-1 v1.33.2 v1.33.2 Rejected 2 0 2 1m +``` + +kubectl get osupgradeprogress +``` +NAME NODE SOURCE CURRENT TARGET STATUS +osupgrade-abc123f node-1 my-upgrade-2 v1.34.1 v1.35.3 downloading +osupgrade-cde456g node-2 my-upgrade-2 v1.35.3 v1.35.3 completed +``` + + +kubectl describe osupgradeprogress osupgrade-abc123f +```yaml +apiVersion: monok8s.io/v1alpha1 +kind: OSUpgradeProgress +metadata: + name: "osupgrade-abc123f" +spec: + sourceRef: + name: my-upgrade-2 + nodeName: node-1 +status: + currentVersion: "v1.34.1" + targetVersion: "v1.35.3" + phase: Downloading + startedAt: null + completedAt: null + lastUpdatedAt: null + retryCount: 0 + inactivePartition: "B" + failureReason: "" + message: "" +``` + +## Development notes + ### Simulate OTA **Use nmap ncat**. Otherwise we'll have all kinds of fabulous issues sending it.