penguin/monok8s

SHA256

Fork 0

Files

斟酌鵬兄 7167e36bf2 research #5

2026-05-01 01:39:04 +08:00

32 KiB

Raw Blame History

Cross-Compiling ASK for Arm64 in Docker Without Full Emulation

Executive summary

The best default for ASK is not QEMU and not an Alpine-first builder. It is a native-host Docker build stage pinned to --platform=$BUILDPLATFORM, using a real arm64 GNU/Linux cross toolchain plus a matching target sysroot and kernel tree. Docker explicitly recommends cross-compilation with multi-stage builds as one of the three multi-platform strategies, and it explicitly warns that QEMU emulation is much slower for compilation-heavy workloads. The key pattern is: keep the build container native to the builder host, and let the compiler target arm64. citeturn10view4turn17search2

For a Linux arm64/glibc target, the cleanest path is a Debian-based builder with crossbuild-essential-arm64, gcc-aarch64-linux-gnu, g++-aarch64-linux-gnu, libc6-dev-arm64-cross, and linux-libc-dev-arm64-cross. Debian’s package metadata makes the intent explicit: crossbuild-essential-arm64 is the cross-build meta-package, gcc-aarch64-linux-gnu is the default arm64 GNU C cross-compiler, and libc6-dev-arm64-cross provides the arm64 glibc development headers and objects for cross-compiling. That stack is the shortest route to reproducible arm64 binaries without emulation. citeturn10view0turn10view1turn10view2

For kernel modules, a matching kernel source tree is not optional. The kernel docs are explicit: external modules need the kernel’s configuration and headers, modules_prepare exists for that purpose, and modules_prepare does not generate Module.symvers when CONFIG_MODVERSIONS is in play. If your module ABI depends on symbol versioning, you need a full kernel build of the matching tree/config to produce Module.symvers. citeturn10view5

For ASK specifically, the practical recommendation is:

Use a Debian cross-toolchain builder for both userspace and module builds.
Use a glibc target sysroot if the runtime target is Debian/Ubuntu or another glibc-based rootfs.
Only use musl-cross or Alpine when the deployment target is actually musl-based.
Use Clang/LLVM only if ASK and its module path are known to be Clang-clean.
Treat crosstool-ng and custom toolchain builds as a last-mile optimization for pinned enterprise toolchains, not the day-one setup. citeturn10view8turn14view0turn10view10turn10view11turn10view12

Assumptions and the recommended path

Because some project details remain unspecified, I am making these assumptions instead of inventing facts:

ASK is a C/C++ project with both userspace code and out-of-tree kernel modules.
The arm64 target runtime is more likely glibc than musl, because the current build direction and requested Debian-cross path point that way.
The source enters the build as tarballs, not Git checkouts.
A matching kernel source tarball is available as KERNEL_TAR when module builds are required.
Third-party target libraries are either vendored as source tarballs or staged into a target sysroot before ASK is compiled.
The build host may be amd64 or arm64, but the goal is to avoid full target emulation either way.

Under those assumptions, the recommended path is:

Use docker buildx build --platform=linux/arm64 for target metadata and artifact export.
Pin every FROM that actually runs build commands to --platform=$BUILDPLATFORM, so those stages run natively on the builder and do not invoke emulation.
Install the Debian arm64 cross toolchain in the builder.
Stage a target sysroot for arm64 glibc.
Build vendored target libraries into that sysroot.
Build ASK userspace with CC/CXX pointing at the cross compiler and --sysroot or an equivalent staged sysroot layout.
Build ASK modules with ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- KDIR=/opt/kernel.
Export artifacts with --output type=local. citeturn17search1turn17search2turn11view0turn11view1

flowchart LR
    A[ASK.tar.gz + dependency tarballs + optional KERNEL_TAR] --> B[buildx build request --platform=linux/arm64]
    B --> C[build stages pinned to --platform=$BUILDPLATFORM]
    C --> D[install Debian arm64 cross toolchain]
    D --> E[stage arm64 glibc sysroot]
    E --> F[build vendored target libs into sysroot]
    F --> G[extract ASK tarball]
    G --> H[build userspace with aarch64-linux-gnu-gcc/g++]
    E --> I[extract matching kernel tree]
    I --> J[merge config fragments and olddefconfig]
    J --> K[modules_prepare or full kernel build]
    K --> L[build external modules]
    H --> M[collect artifacts]
    L --> M
    M --> N[scratch artifacts stage]
    N --> O[--output type=local]

Toolchain choices without emulation

The main choices are not equal.

Debian cross packages

This is the best default for ASK if the target runtime is arm64 GNU/Linux with glibc. Debian already publishes a coherent arm64 GNU cross stack: crossbuild-essential-arm64, gcc-aarch64-linux-gnu, g++-aarch64-linux-gnu, libc6-dev-arm64-cross, linux-libc-dev-arm64-cross, and the matching arm64 libstdc++ development package. That gives you a native-host compilation environment that directly targets arm64 GNU/Linux without needing emulation or a self-built compiler. citeturn10view0turn10view1turn10view2turn10view3turn15view0turn15view1

Arm GNU Toolchain and Linaro-delivered releases

This is the right fallback when you need a prebuilt, distro-independent GNU cross compiler outside Debian’s cadence, or you want the exact Arm-distributed toolchain family. Linaro’s current downloads page explicitly points users to the Arm Developer site for the official prebuilt GNU cross-toolchain releases for AArch64 and A-profile Arm targets. That makes Arm GNU Toolchain a legitimate alternative to Debian packages, especially when you want one pinned tarball instead of distro package resolution. citeturn10view10turn5search3

Clang and LLVM cross-targeting

Clang is viable when you want one compiler binary that can emit code for many targets, and the official Clang cross-compilation docs explain the exact model: use -target and, where needed, --sysroot, plus explicit include/library paths. For the kernel specifically, the official kernel LLVM docs say arm64 is supported, make LLVM=1 ARCH=arm64 is the standard form, and if you use only LLVM tools then CROSS_COMPILE becomes unnecessary; if you mix GNU binutils with LLVM, you set those utilities explicitly. This is a strong option, but only if ASK’s userspace and module code are already known to behave under Clang. citeturn14view0turn14view1

crosstool-ng

crosstool-ng is a toolchain generator, not a fast path. Its official site describes it as a versatile cross-toolchain generator with a menuconfig-style interface. That is useful when you need a custom-pinned compiler, libc, binutils, and threading model combination, but it is slower to bootstrap and heavier to maintain than using Debian’s built packages. Use it when you need that control, not because you think you are being clever. citeturn10view8

musl-cross and Alpine-based targeting

Use this only when the target is musl-based. Alpine’s own docs say Alpine uses musl, and musl does not implement most glibc locale behavior. Alpine also documents gcompat as a compatibility layer for simpler glibc programs, not as a universal solution. musl’s own site links to musl-cross-make as the automated cross toolchain builder, and the musl-cross-make project describes itself as a simple, relocatable way to produce musl-targeting cross compilers. That is good for a musl target. It is the wrong default for a glibc target. citeturn10view11turn10view12turn11view8turn11view9

Decision table

Option	Best when	Strengths	Weaknesses	Verdict for ASK
Debian cross packages	Target is arm64 GNU/Linux with glibc	Fast setup, distro-integrated sysroot, easiest CI	Tied to Debian package cadence	Best default
Arm GNU Toolchain	You want a pinned prebuilt toolchain tarball	Portable, explicit toolchain versioning	You still need a matching sysroot	Strong alternative
Clang/LLVM + GNU sysroot	Codebase is Clang-clean and you want one compiler binary	Good cross model, kernel arm64 support	Tool/library path tuning can be fussier	Good if already validated
crosstool-ng	You need a custom compiler/libc/binutils combo	Maximum control	Slow bootstrap, more maintenance	Use only if necessary
musl-cross / Alpine	Target runtime is musl	Small runtimes, relocatable toolchains possible	glibc mismatch risk	Use only for musl targets
QEMU emulation	You need runtime smoke tests or no cross path exists	Easy conceptually	Slow for compilation	Avoid for primary builds

The table above is grounded in Docker’s documented multi-platform strategies, Debian’s cross-package metadata, Clang’s cross-compilation docs, the kernel’s LLVM docs, crosstool-ng’s own documentation, and Alpine/musl documentation. Docker explicitly states that QEMU is usually much slower for compilation-heavy workloads and recommends cross-compilation or native multi-node builders when possible. citeturn10view4turn10view0turn10view1turn14view0turn14view1turn10view8turn10view11turn10view12

Debian and Alpine comparison for arm64-targeted Linux builds

Need	Debian / glibc path	Alpine / musl path	Bottom line
Native build meta-package	`build-essential`	`build-base`	Straight equivalents for native builds
Arm64 Linux cross meta-package	`crossbuild-essential-arm64`	no close stable equivalent	Debian is much better here
Arm64 Linux GNU C compiler	`gcc-aarch64-linux-gnu`	no obvious stable `aarch64-linux-gnu-gcc` package surfaced	Debian wins for glibc Linux targets
Arm64 Linux GNU C++ compiler	`g++-aarch64-linux-gnu`	no obvious stable `aarch64-linux-gnu-g++` package surfaced	Debian wins
glibc target sysroot headers/libs	`libc6-dev-arm64-cross`, `linux-libc-dev-arm64-cross`, `libstdc++-14-dev-arm64-cross`	Alpine is musl-first, not glibc-first	Use Debian for glibc sysroots
musl target compiler	extra work on Debian or `musl-cross-make`	Alpine naturally targets musl	Alpine or musl-cross only if target is musl
glibc compatibility on musl	n/a	`gcompat` for simpler cases	Useful only as a runtime compatibility hack

This comparison comes directly from Debian package pages and Alpine’s own package/wiki materials. Alpine clearly documents build-base as the standard build meta-package and documents musl as the system libc. The Alpine package index examples that do surface clear cross packages in this space are *-none-elf embedded toolchains, which are not the same thing as a glibc/Linux arm64 cross stack. citeturn10view0turn10view1turn10view2turn15view0turn10view13turn11view6turn11view7turn19view0turn10view11

Concrete Docker implementation

The primary design below has two Dockerfiles.

docker/arm64-sysroot.Dockerfile builds and caches a reusable arm64 target sysroot.
docker/ask.Dockerfile uses that sysroot, the Debian arm64 cross compiler, and an optional matching kernel tree to build ASK userspace and modules without emulation.

The key trick is that both Dockerfiles pin real build stages to FROM --platform=$BUILDPLATFORM .... That means the build steps run natively on the builder host even when the overall build request targets linux/arm64. Docker’s own docs explicitly describe this pattern and the automatic BUILDPLATFORM / TARGET* build args that make it work. citeturn17search1turn17search2turn10view4

Root Makefile

.RECIPEPREFIX := >

DOCKER_BUILDX      ?= docker buildx build
TARGET_PLATFORM    ?= linux/arm64
DEBIAN_SUITE       ?= trixie

ASK_TAR            ?= packages/ASK.tar.gz
KERNEL_TAR         ?=
FMLIB_TAR          ?= packages/fmlib-lf-6.12.49-2.2.0.tar.gz
FMC_TAR            ?= packages/fmc-lf-6.12.49-2.2.0.tar.gz
LIBNFNETLINK_TAR   ?= packages/libnfnetlink-1.0.2.tar.bz2
LIBNFCT_TAR        ?= packages/libnetfilter_conntrack-1.1.0.tar.xz
LIBCLI_TAR         ?= packages/libcli-1.10.7.tar.gz

ASK_OUT            ?= out/ask
ASK_IMAGE_NAME     ?= local/ask:dev
SYSROOT_IMAGE      ?= local/ask-arm64-sysroot:dev

TARGET_ARCH        ?= arm64
TARGET_TRIPLE      ?= aarch64-linux-gnu
BUILD_TARGET       ?= dist
KERNEL_FULL_BUILD  ?= 0
SOURCE_DATE_EPOCH  ?= 1714521600

COMMON_ARGS = \
> --build-arg DEBIAN_SUITE=$(DEBIAN_SUITE) \
> --build-arg TARGET_TRIPLE=$(TARGET_TRIPLE) \
> --build-arg SOURCE_DATE_EPOCH=$(SOURCE_DATE_EPOCH)

.PHONY: ASK_SYSROOT ASK ASK_IMAGE

ASK_SYSROOT:
> $(DOCKER_BUILDX) \
>   --platform $(TARGET_PLATFORM) \
>   -f docker/arm64-sysroot.Dockerfile \
>   $(COMMON_ARGS) \
>   --build-arg FMLIB_TAR=$(FMLIB_TAR) \
>   --build-arg FMC_TAR=$(FMC_TAR) \
>   --build-arg LIBNFNETLINK_TAR=$(LIBNFNETLINK_TAR) \
>   --build-arg LIBNFCT_TAR=$(LIBNFCT_TAR) \
>   --build-arg LIBCLI_TAR=$(LIBCLI_TAR) \
>   --load \
>   -t $(SYSROOT_IMAGE) \
>   .

ASK: ASK_SYSROOT
> mkdir -p $(ASK_OUT)
> $(DOCKER_BUILDX) \
>   --platform $(TARGET_PLATFORM) \
>   -f docker/ask.Dockerfile \
>   $(COMMON_ARGS) \
>   --build-arg SYSROOT_IMAGE=$(SYSROOT_IMAGE) \
>   --build-arg ASK_TAR=$(ASK_TAR) \
>   --build-arg KERNEL_TAR=$(KERNEL_TAR) \
>   --build-arg BUILD_TARGET=$(BUILD_TARGET) \
>   --build-arg KERNEL_FULL_BUILD=$(KERNEL_FULL_BUILD) \
>   --target artifacts \
>   --output type=local,dest=$(ASK_OUT) \
>   .

ASK_IMAGE: ASK_SYSROOT
> $(DOCKER_BUILDX) \
>   --platform $(TARGET_PLATFORM) \
>   -f docker/ask.Dockerfile \
>   $(COMMON_ARGS) \
>   --build-arg SYSROOT_IMAGE=$(SYSROOT_IMAGE) \
>   --build-arg ASK_TAR=$(ASK_TAR) \
>   --build-arg KERNEL_TAR=$(KERNEL_TAR) \
>   --build-arg BUILD_TARGET=$(BUILD_TARGET) \
>   --build-arg KERNEL_FULL_BUILD=$(KERNEL_FULL_BUILD) \
>   --target runtime \
>   --load \
>   -t $(ASK_IMAGE_NAME) \
>   .

Example invocations:

# Userspace-only build
make ASK ASK_TAR=packages/ASK.tar.gz BUILD_TARGET=userspace

# Full build with matching kernel tree and full kernel build to get Module.symvers
make ASK \
  ASK_TAR=packages/ASK.tar.gz \
  KERNEL_TAR=packages/lf-6.12.49-2.2.0.tar.gz \
  BUILD_TARGET=dist \
  KERNEL_FULL_BUILD=1

# Minimal artifact image instead of local export
make ASK_IMAGE ASK_TAR=packages/ASK.tar.gz BUILD_TARGET=userspace

Helper sysroot Dockerfile

# syntax=docker/dockerfile:1.7
ARG DEBIAN_SUITE=trixie

FROM --platform=$BUILDPLATFORM debian:${DEBIAN_SUITE}-slim AS sysroot-build
ARG TARGET_TRIPLE=aarch64-linux-gnu
ARG SOURCE_DATE_EPOCH

ARG FMLIB_TAR=
ARG FMC_TAR=
ARG LIBNFNETLINK_TAR=
ARG LIBNFCT_TAR=
ARG LIBCLI_TAR=

ENV DEBIAN_FRONTEND=noninteractive
ENV LC_ALL=C.UTF-8
ENV TZ=UTC
ENV SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH}

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && apt-get install -y --no-install-recommends \
      ca-certificates \
      build-essential \
      crossbuild-essential-arm64 \
      gcc-aarch64-linux-gnu \
      g++-aarch64-linux-gnu \
      binutils-aarch64-linux-gnu \
      libc6-dev-arm64-cross \
      linux-libc-dev-arm64-cross \
      libstdc++-14-dev-arm64-cross \
      autoconf automake libtool pkgconf make patch perl python3 rsync xz-utils bzip2 file \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /work
COPY packages/ /vendor/packages/

# Build a clean target sysroot rooted at /opt/sysroot.
RUN mkdir -p /opt/sysroot/usr && rsync -a /usr/aarch64-linux-gnu/ /opt/sysroot/usr/

# Optional: cross-build vendored target libraries into the sysroot.
RUN if [ -n "${LIBNFNETLINK_TAR}" ] && [ -f "/vendor/${LIBNFNETLINK_TAR}" ]; then \
      mkdir -p /tmp/libnfnetlink && \
      tar -xf "/vendor/${LIBNFNETLINK_TAR}" --strip-components=1 -C /tmp/libnfnetlink && \
      cd /tmp/libnfnetlink && \
      ./configure --host=${TARGET_TRIPLE} --prefix=/usr --libdir=/usr/lib/aarch64-linux-gnu && \
      make -j"$(nproc)" && \
      make DESTDIR=/opt/sysroot install; \
    fi

RUN if [ -n "${LIBNFCT_TAR}" ] && [ -f "/vendor/${LIBNFCT_TAR}" ]; then \
      mkdir -p /tmp/libnfct && \
      tar -xf "/vendor/${LIBNFCT_TAR}" --strip-components=1 -C /tmp/libnfct && \
      cd /tmp/libnfct && \
      PKG_CONFIG_SYSROOT_DIR=/opt/sysroot \
      PKG_CONFIG_LIBDIR=/opt/sysroot/usr/lib/aarch64-linux-gnu/pkgconfig:/opt/sysroot/usr/share/pkgconfig \
      ./configure --host=${TARGET_TRIPLE} --prefix=/usr --libdir=/usr/lib/aarch64-linux-gnu && \
      make -j"$(nproc)" && \
      make DESTDIR=/opt/sysroot install; \
    fi

RUN if [ -n "${LIBCLI_TAR}" ] && [ -f "/vendor/${LIBCLI_TAR}" ]; then \
      mkdir -p /tmp/libcli && \
      tar -xf "/vendor/${LIBCLI_TAR}" --strip-components=1 -C /tmp/libcli && \
      make -C /tmp/libcli \
        CC="${TARGET_TRIPLE}-gcc --sysroot=/opt/sysroot" \
        AR="${TARGET_TRIPLE}-ar" && \
      make -C /tmp/libcli PREFIX=/usr DESTDIR=/opt/sysroot install; \
    fi

FROM scratch AS sysroot
COPY --from=sysroot-build /opt/sysroot/ /

Main ASK Dockerfile

# syntax=docker/dockerfile:1.7
ARG DEBIAN_SUITE=trixie
ARG SYSROOT_IMAGE=local/ask-arm64-sysroot:dev

FROM ${SYSROOT_IMAGE} AS sysroot

FROM --platform=$BUILDPLATFORM debian:${DEBIAN_SUITE}-slim AS build
ARG TARGETPLATFORM
ARG TARGETOS
ARG TARGETARCH
ARG TARGET_TRIPLE=aarch64-linux-gnu
ARG ASK_TAR=packages/ASK.tar.gz
ARG KERNEL_TAR=
ARG BUILD_TARGET=dist
ARG KERNEL_FULL_BUILD=0
ARG SOURCE_DATE_EPOCH

ENV DEBIAN_FRONTEND=noninteractive
ENV LC_ALL=C.UTF-8
ENV TZ=UTC
ENV SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH}
ENV KBUILD_BUILD_TIMESTAMP=@${SOURCE_DATE_EPOCH}
ENV KBUILD_BUILD_USER=repro
ENV KBUILD_BUILD_HOST=repro-host

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && apt-get install -y --no-install-recommends \
      ca-certificates \
      build-essential \
      crossbuild-essential-arm64 \
      gcc-aarch64-linux-gnu \
      g++-aarch64-linux-gnu \
      binutils-aarch64-linux-gnu \
      libc6-dev-arm64-cross \
      linux-libc-dev-arm64-cross \
      libstdc++-14-dev-arm64-cross \
      bc bison cpio file flex kmod libelf-dev make openssl patch perl pkgconf python3 rsync xz-utils \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /work
COPY --from=sysroot / /opt/sysroot/
COPY packages/ /vendor/packages/
COPY scripts/filter-kconfig-fragment.sh /usr/local/bin/filter-kconfig-fragment.sh
COPY docker/kernel-extra.config /tmp/kernel-extra.config
COPY docker/overrides/ /docker-overrides/
RUN chmod +x /usr/local/bin/filter-kconfig-fragment.sh

# Extract ASK tarball and inject cross-build overrides.
RUN mkdir -p /src/ASK && \
    tar -xf "/vendor/${ASK_TAR}" --strip-components=1 -C /src/ASK && \
    rm -rf /src/ASK/.git && \
    install -m 0644 /docker-overrides/Makefile /src/ASK/Makefile && \
    install -m 0644 /docker-overrides/toolchain.mk /src/ASK/build/toolchain.mk

# Optional matching kernel tree for out-of-tree module builds.
RUN if [ -n "${KERNEL_TAR}" ]; then \
      mkdir -p /opt/kernel && \
      tar -xf "/vendor/${KERNEL_TAR}" --strip-components=1 -C /opt/kernel && \
      make -C /opt/kernel ARCH=arm64 CROSS_COMPILE=${TARGET_TRIPLE}- defconfig && \
      /usr/local/bin/filter-kconfig-fragment.sh /opt/kernel /tmp/kernel-extra.config > /tmp/kernel-extra.effective.config && \
      /opt/kernel/scripts/kconfig/merge_config.sh -m /opt/kernel/.config /tmp/kernel-extra.effective.config && \
      KCONFIG_WARN_UNKNOWN_SYMBOLS=1 KCONFIG_WERROR=1 \
      make -C /opt/kernel ARCH=arm64 CROSS_COMPILE=${TARGET_TRIPLE}- olddefconfig && \
      if [ "${KERNEL_FULL_BUILD}" = "1" ]; then \
        make -C /opt/kernel ARCH=arm64 CROSS_COMPILE=${TARGET_TRIPLE}- -j"$(nproc)" Image modules dtbs; \
      else \
        make -C /opt/kernel ARCH=arm64 CROSS_COMPILE=${TARGET_TRIPLE}- -j"$(nproc)" modules_prepare; \
      fi; \
    fi

WORKDIR /src/ASK

RUN export SYSROOT=/opt/sysroot; \
    export CROSS_COMPILE=${TARGET_TRIPLE}-; \
    export ARCH=arm64; \
    export CC="${TARGET_TRIPLE}-gcc --sysroot=${SYSROOT}"; \
    export CXX="${TARGET_TRIPLE}-g++ --sysroot=${SYSROOT}"; \
    export AR="${TARGET_TRIPLE}-ar"; \
    export STRIP="${TARGET_TRIPLE}-strip"; \
    export PKG_CONFIG=pkg-config; \
    export PKG_CONFIG_SYSROOT_DIR="${SYSROOT}"; \
    export PKG_CONFIG_LIBDIR="${SYSROOT}/usr/lib/aarch64-linux-gnu/pkgconfig:${SYSROOT}/usr/share/pkgconfig"; \
    case "${BUILD_TARGET}" in \
      userspace) make userspace ;; \
      modules)   test -n "${KERNEL_TAR}" && make KDIR=/opt/kernel modules ;; \
      dist)      test -n "${KERNEL_TAR}" && make KDIR=/opt/kernel dist ;; \
      *)         echo "unsupported BUILD_TARGET=${BUILD_TARGET}" >&2; exit 2 ;; \
    esac && \
    mkdir -p /out && cp -a dist/. /out/

FROM scratch AS artifacts
COPY --from=build /out/ /

FROM scratch AS runtime
COPY --from=build /out/ /opt/ask/

Cross-aware Makefile override excerpt

If ASK’s upstream Makefile already has tarball-only source handling from your earlier reproducible-build work, the critical cross additions are these flags and environment variables:

ARCH           ?= arm64
TARGET_TRIPLE  ?= aarch64-linux-gnu
CROSS_COMPILE  ?= $(TARGET_TRIPLE)-
SYSROOT        ?= /opt/sysroot

CC             ?= $(TARGET_TRIPLE)-gcc --sysroot=$(SYSROOT)
CXX            ?= $(TARGET_TRIPLE)-g++ --sysroot=$(SYSROOT)
AR             ?= $(TARGET_TRIPLE)-ar
STRIP          ?= $(TARGET_TRIPLE)-strip
PKG_CONFIG     ?= pkg-config

export PKG_CONFIG_SYSROOT_DIR := $(SYSROOT)
export PKG_CONFIG_LIBDIR := \
  $(SYSROOT)/usr/lib/aarch64-linux-gnu/pkgconfig:$(SYSROOT)/usr/share/pkgconfig

KBUILD_ARGS := ARCH=$(ARCH) CROSS_COMPILE=$(CROSS_COMPILE)

userspace:
	$(MAKE) -C cmm CC="$(CC)" CXX="$(CXX)"
	$(MAKE) -C dpa_app CC="$(CC)" CXX="$(CXX)"

modules:
	test -d "$(KDIR)"
	$(MAKE) -C cdx KERNELDIR="$(KDIR)" $(KBUILD_ARGS)
	$(MAKE) -C fci KERNEL_SOURCE="$(KDIR)" $(KBUILD_ARGS)
	$(MAKE) -C auto_bridge KERNEL_SOURCE="$(KDIR)" $(KBUILD_ARGS)

Optional Arm GNU Toolchain stage

If you prefer a prebuilt Arm-distributed toolchain tarball over Debian packages, swap in a stage like this and keep the rest of the sysroot/kernel logic the same:

FROM --platform=$BUILDPLATFORM debian:trixie-slim AS armgnu
ARG ARM_GNU_TARBALL=packages/arm-gnu-toolchain-aarch64-linux-gnu.tar.xz
COPY packages/ /vendor/packages/
RUN mkdir -p /opt/toolchain && \
    tar -xf "/vendor/${ARM_GNU_TARBALL}" --strip-components=1 -C /opt/toolchain
ENV PATH=/opt/toolchain/bin:${PATH}

The Docker pieces above rely on Docker’s documented multi-stage builds, automatic platform args, native-stage --platform=$BUILDPLATFORM pattern, local artifact exporter, and build-arg semantics. For private tarballs or credentials, Docker explicitly says to use secret mounts rather than ARG or ENV. citeturn11view1turn17search1turn17search2turn11view0turn11view2turn11view3

Kernel modules, sysroots, and config handling

The kernel side is where most “cross-compilation” guides turn to mush. The correct model is sharper than that.

If ASK builds only userspace, you need:

a target compiler,
a target libc/sysroot,
target .pc metadata or explicit include/library paths.

If ASK builds external kernel modules, you additionally need:

the exact target kernel source tree or a prepared build tree,
the exact target .config after fragment merging,
generated headers under include/generated,
and, when CONFIG_MODVERSIONS=y, a matching Module.symvers from a full kernel build, not merely modules_prepare. citeturn10view5turn11view5

That distinction matters because linux-libc-dev-arm64-cross is for userspace development headers. Debian’s own package metadata says those are Linux kernel headers for cross-compiling development, not a substitute for the actual configured kernel build tree you need for external modules. So: use Debian cross libc/sysroot packages for userspace, and use KERNEL_TAR for modules. citeturn10view2turn10view3

Minimal sysroot extraction patterns

If you build the sysroot from Debian cross packages:

mkdir -p /opt/sysroot/usr
rsync -a /usr/aarch64-linux-gnu/ /opt/sysroot/usr/

If you receive a prebuilt sysroot tarball instead:

mkdir -p /opt/sysroot
tar -xf packages/arm64-glibc-sysroot.tar.xz -C /opt/sysroot

If you receive a matching kernel tree tarball:

mkdir -p /opt/kernel
tar -xf packages/lf-6.12.49-2.2.0.tar.gz --strip-components=1 -C /opt/kernel

Kernel config merge and mismatch handling

Use this sequence every time:

make -C /opt/kernel ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- defconfig

filter-kconfig-fragment.sh /opt/kernel docker/kernel-extra.config \
  > /tmp/kernel-extra.effective.config

/opt/kernel/scripts/kconfig/merge_config.sh -m \
  /opt/kernel/.config \
  /tmp/kernel-extra.effective.config

KCONFIG_WARN_UNKNOWN_SYMBOLS=1 KCONFIG_WERROR=1 \
make -C /opt/kernel ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- olddefconfig

Why the filter exists is simple: config fragments drift across kernel lines. merge_config.sh is the right merge tool, but it does not magically make a symbol exist in a tree that no longer defines it. Filtering before merge prevents stale fragment entries from poisoning the build.

The kernel docs explicitly document KCONFIG_WARN_UNKNOWN_SYMBOLS and KCONFIG_WERROR, and the kernel tree ships merge_config.sh explicitly for fragment merging. citeturn16search1turn11view5

Auto-gating missing symbols with `scripts/config`

For features that are optional or kernel-version-dependent, gate them before olddefconfig:

cd /opt/kernel

if grep -RqsE '^[[:space:]]*(menu)?config[[:space:]]+NETFILTER_XTABLES_LEGACY([[:space:]]|$)' .; then
  scripts/config --file .config -e NETFILTER_XTABLES_LEGACY
fi

if grep -RqsE '^[[:space:]]*(menu)?config[[:space:]]+IP_NF_IPTABLES_LEGACY([[:space:]]|$)' .; then
  scripts/config --file .config -e IP_NF_IPTABLES_LEGACY
fi

KCONFIG_WARN_UNKNOWN_SYMBOLS=1 KCONFIG_WERROR=1 \
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- olddefconfig

This is the right cure for errors of the form “fragment expects CONFIG_X=y but the resolved config says <missing>”: first check whether the symbol exists in the target tree, then enable it only when it exists, then let Kconfig resolve dependencies. scripts/config is the official in-tree command-line .config manipulator. citeturn16search0turn16search1turn16search3

When to stop at `modules_prepare` and when to do a full kernel build

Use modules_prepare when:

you only need generated headers and basic preparation,
and CONFIG_MODVERSIONS is not required for module ABI matching.

Do a full kernel build when:

CONFIG_MODVERSIONS=y,
you need Module.symvers,
or you want the strongest ABI match signal against the actual shipping board kernel.

A practical decision rule:

If you need only compile-time headers: modules_prepare may be enough.
If you need symbol versioning correctness: build the kernel fully.

That rule is not opinion; it is straight from the kernel’s external-modules documentation. citeturn10view5

Verification, CI, and trade-offs

The verification story should be mechanical, not aspirational.

To verify arm64 userspace binaries:

file out/ask/cmm
readelf -h out/ask/cmm | grep 'Machine:'
readelf -l out/ask/cmm | grep 'Requesting program interpreter'
readelf -d out/ask/cmm | grep NEEDED

You want:

file to report an AArch64 ELF,
readelf -h to report Machine: AArch64,
the ELF interpreter to match the target runtime’s loader,
and NEEDED entries to resolve against the target sysroot or rootfs, not your host.

To verify kernel modules:

file out/ask/cdx.ko
readelf -h out/ask/cdx.ko | grep 'Machine:'
modinfo -F vermagic out/ask/cdx.ko
readelf -S out/ask/cdx.ko | grep __versions || true

You want:

Machine: AArch64,
vermagic matching the target kernel release/build flags,
and, when CONFIG_MODVERSIONS=y, version sections consistent with the kernel build products.

For stricter ABI checks, compare the module against the exact Module.symvers and shipping kernel release, not a hand-wavy “same major version” guess.

A light smoke test under QEMU is acceptable after build if you want one, but it should be optional and narrow. The primary build should remain non-emulated. Docker’s own docs explicitly recommend cross-compilation or native multi-node builders over QEMU where possible because QEMU is slower for compute-heavy work. citeturn10view4

CI guidance that actually matters

Use these practices:

Pin builder base images by digest.
Keep toolchain and sysroot in separate reusable stages or images.
Verify SHA256SUMS for every vendored tarball before extraction.
Set SOURCE_DATE_EPOCH, KBUILD_BUILD_TIMESTAMP, KBUILD_BUILD_USER, and KBUILD_BUILD_HOST.
Remove .git from tarball-extracted sources unless VCS metadata is a deliberate build input.
Use BuildKit cache mounts for apt and, if applicable, compiler caches.
Use --output type=local for artifacts rather than hiding everything inside an image layer.

Those recommendations are directly aligned with Docker’s best-practices guidance and the kernel’s reproducible-build guidance. The kernel docs are explicit that timestamps, user, and host leakage must be overridden for reproducible output, and Docker’s docs explicitly recommend multi-stage builds and local exporters for clean build outputs. citeturn11view1turn11view0turn11view4

Trade-offs

If you want the blunt version:

Debian cross packages are the best speed-to-value option for ASK.
Arm GNU Toolchain is best when you want a pinned vendor-distributed compiler tarball.
Clang/LLVM is attractive if you already know the project and module path build cleanly with it.
crosstool-ng is for teams that truly need custom toolchains and are willing to own them.
musl-cross only makes sense when the target runtime is musl.
QEMU is a fallback or a spot-check tool, not the backbone of a serious CI build.

flowchart TD
    A[Need arm64 ASK artifacts in Docker] --> B{Target runtime glibc?}
    B -- yes --> C{Need fastest reliable setup?}
    C -- yes --> D[Debian cross packages + glibc sysroot]
    C -- no --> E{Need custom pinned toolchain?}
    E -- yes --> F[Arm GNU Toolchain or crosstool-ng]
    E -- no --> D
    B -- no --> G{Target runtime musl?}
    G -- yes --> H[musl-cross or Alpine/musl sysroot]
    G -- no --> I[Clarify runtime first]
    D --> J{Kernel modules involved?}
    F --> J
    H --> J
    J -- no --> K[userspace cross build only]
    J -- yes --> L[provide KERNEL_TAR + config + headers]
    L --> M{CONFIG_MODVERSIONS?}
    M -- no --> N[modules_prepare may be enough]
    M -- yes --> O[full kernel build to get Module.symvers]
    K --> P[verify ELF headers]
    N --> Q[verify vermagic and symbols]
    O --> Q

Sources and notes

This report prioritizes official or project-authoritative sources. Docker guidance is from the official Docker documentation on multi-platform builds, BUILDPLATFORM/TARGET* build arguments, build secrets, local exporters, and Dockerfile best practices. Debian package metadata is from the official Debian package pages for crossbuild-essential-arm64, gcc-aarch64-linux-gnu, g++-aarch64-linux-gnu, libc6-dev-arm64-cross, linux-libc-dev-arm64-cross, and libstdc++-14-dev-arm64-cross. Kernel guidance is from the official Linux kernel docs on external modules, modules_prepare, Module.symvers, reproducible builds, Kconfig controls, Clang/LLVM kernel builds, and the in-tree merge_config.sh / scripts/config utilities. Alternative cross-toolchain options are grounded in the official crosstool-ng site, the musl site and musl-cross-make, and Linaro’s downloads page pointing to the Arm Developer site for official Arm GNU toolchain releases. Alpine references are from the Alpine wiki and package index for build-base, musl, and gcompat. citeturn10view4turn17search1turn17search2turn11view0turn11view1turn11view2turn11view3turn10view0turn10view1turn10view2turn10view3turn15view0turn15view1turn10view5turn11view4turn11view5turn16search0turn16search1turn14view1turn10view8turn10view10turn11view8turn11view9turn10view11turn10view12turn11view6turn11view7turn19view0turn19view1

32 KiB Raw Blame History Unescape Escape

Cross-Compiling ASK for Arm64 in Docker Without Full Emulation

Executive summary

Assumptions and the recommended path

Toolchain choices without emulation

Debian cross packages

Arm GNU Toolchain and Linaro-delivered releases

Clang and LLVM cross-targeting

crosstool-ng

musl-cross and Alpine-based targeting

Decision table

Debian and Alpine comparison for arm64-targeted Linux builds

Concrete Docker implementation

Root Makefile

Helper sysroot Dockerfile

Main ASK Dockerfile

Cross-aware Makefile override excerpt

Optional Arm GNU Toolchain stage

Kernel modules, sysroots, and config handling

Minimal sysroot extraction patterns

Kernel config merge and mismatch handling

Auto-gating missing symbols with scripts/config

When to stop at modules_prepare and when to do a full kernel build

Verification, CI, and trade-offs

CI guidance that actually matters

Trade-offs

Sources and notes

32 KiB

Raw Blame History

Auto-gating missing symbols with `scripts/config`

When to stop at `modules_prepare` and when to do a full kernel build