diff --git a/docs/ask-deepresearch-4.md b/docs/ask-deepresearch-4.md new file mode 100644 index 0000000..597c2e4 --- /dev/null +++ b/docs/ask-deepresearch-4.md @@ -0,0 +1,261 @@ +# `filter-kconfig-fragment.sh` for Reproducible Kernel-Config Filtering + +## Executive summary + +I did **not** find a file named `filter-kconfig-fragment.sh` in the uploaded project files or in the supplied ASK/kernel tarballs. What **is** present is the standard kernel-side machinery you would use around such a filter: the Linux kernel’s own `scripts/kconfig/merge_config.sh` for fragment merging, `scripts/config` for imperative `.config` edits, and documented Kconfig controls such as `KCONFIG_WARN_UNKNOWN_SYMBOLS`, `KCONFIG_WERROR`, and `KCONFIG_ALLCONFIG`. The kernel tree itself also uses `merge_config.sh` followed by `olddefconfig` when it builds from config fragments. citeturn3view1turn0search2turn3view2 + +A small pre-filter is still useful, because neither `merge_config.sh` nor `scripts/config` is meant to silently rewrite a fragment for a **different kernel generation**. `merge_config.sh` merges and warns; `scripts/config` edits `.config`; Kconfig enforces symbol existence and dependency rules. When a fragment contains a symbol that simply does not exist in the target tree, such as `CONFIG_NETFILTER_XTABLES_LEGACY` in your reported 6.12-based tree, a pre-filter avoids an avoidable mismatch by dropping only the absent symbol while preserving ordering, comments, and all valid settings. That is the narrow job of the replacement script below. citeturn3view1turn3view2turn3view6 + +The replacement script in this report is POSIX `sh`, offline-safe, requires only standard Unix tools (`find`, `grep`, `sed`, `mktemp`, `cat`), validates inputs, checks for `scripts/kconfig/merge_config.sh`, preserves comments and ordering, emits only symbols that exist in the target Kconfig tree, and exits non-zero if nothing usable remains after filtering. It is designed specifically for Docker builder stages and reproducible kernel-config flows. citeturn3view1turn3view2turn3view4 + +## What I found and what to use instead + +The Linux kernel already ships the two main utilities you should treat as authoritative. + +`merge_config.sh` is the standard fragment-merging tool in the kernel tree. Its own header says it “takes a list of config fragment values, and merges them one by one,” and that it warns about overridden values and symbols that did not make it into the final `.config` because of dependencies or symbol removal. The kernel build system also invokes `scripts/kconfig/merge_config.sh -m $(KCONFIG_CONFIG) ...` and then runs `olddefconfig` for fragment-based config targets. citeturn3view1turn0search2 + +`scripts/config` is the standard imperative editor for `.config`. Its built-in usage text documents `--enable`, `--disable`, `--module`, `--set-str`, `--set-val`, `--undefine`, `--state`, and `--file`, and also states that it does **not** validate `.config` immediately; that validation happens at the next `make` step. The kernel docs also contain concrete examples of `./scripts/config -e ...` and `-m ...` being used to enable required options. citeturn3view0turn3view5 + +Kconfig itself exposes the exact controls you want for drift detection: `KCONFIG_WARN_UNKNOWN_SYMBOLS` warns on unrecognized config symbols, `KCONFIG_WERROR` turns those warnings into errors, and `KCONFIG_ALLCONFIG` provides a documented “mini-config” mechanism for all*config targets. The same docs also recommend `make listnewconfig` to surface newly introduced symbols. citeturn3view2turn5view0 + +For external module builds, the kernel docs also matter here: `modules_prepare` can prepare a tree for external modules, but it does **not** produce `Module.symvers` if `CONFIG_MODVERSIONS` matters, so a full kernel build is still required in that case. That is the main reason a strict builder pipeline should validate kernel config and preparation before invoking out-of-tree module builds. citeturn3view4 + +## Minimal portable replacement script + +Assumptions for this script: + +- `KERNEL_DIR` is a Linux kernel source tree root. +- The tree contains `scripts/kconfig/merge_config.sh`. +- Config symbols are defined in `Kconfig*` files somewhere under `KERNEL_DIR`. +- The fragment format is standard Kconfig fragment syntax: `CONFIG_FOO=...` and `# CONFIG_FOO is not set`. +- The builder environment has POSIX `sh`, `find`, `grep`, `sed`, `mktemp`, `cat`, and `rm`. + +**`scripts/filter-kconfig-fragment.sh`** + +```sh +#!/bin/sh +# filter-kconfig-fragment.sh +# +# Purpose: +# Emit an "effective" Kconfig fragment that preserves comments and ordering +# but drops CONFIG symbols that do not exist in the target kernel tree. +# +# Usage: +# filter-kconfig-fragment.sh KERNEL_DIR FRAGMENT_FILE > effective.config +# +# Exit codes: +# 2 invalid usage / missing required files +# 3 no Kconfig files were found in KERNEL_DIR +# 4 fragment had no surviving CONFIG lines after filtering + +set -eu + +usage() { + echo "usage: $0 KERNEL_DIR FRAGMENT_FILE" >&2 + exit 2 +} + +[ "$#" -eq 2 ] || usage + +KERNEL_DIR=$1 +FRAGMENT_FILE=$2 +MERGE_SCRIPT=$KERNEL_DIR/scripts/kconfig/merge_config.sh + +[ -d "$KERNEL_DIR" ] || { + echo "error: KERNEL_DIR is not a directory: $KERNEL_DIR" >&2 + exit 2 +} + +[ -r "$FRAGMENT_FILE" ] || { + echo "error: FRAGMENT_FILE is not readable: $FRAGMENT_FILE" >&2 + exit 2 +} + +[ -s "$FRAGMENT_FILE" ] || { + echo "error: FRAGMENT_FILE is empty: $FRAGMENT_FILE" >&2 + exit 2 +} + +[ -r "$MERGE_SCRIPT" ] || { + echo "error: required kernel utility missing: $MERGE_SCRIPT" >&2 + exit 2 +} + +TMP_KCONFIGS=$(mktemp) +TMP_OUT=$(mktemp) +trap 'rm -f "$TMP_KCONFIGS" "$TMP_OUT"' EXIT HUP INT TERM + +# Index all Kconfig files once. Searching the whole tree is simpler and more +# robust than hardcoding a small directory subset. +find "$KERNEL_DIR" -type f \( -name 'Kconfig' -o -name 'Kconfig*' \) | sort > "$TMP_KCONFIGS" + +[ -s "$TMP_KCONFIGS" ] || { + echo "error: no Kconfig files found under $KERNEL_DIR" >&2 + exit 3 +} + +symbol_exists() { + # Accept either CONFIG_FOO or FOO. + sym=$1 + case "$sym" in + CONFIG_*) sym=${sym#CONFIG_} ;; + esac + + # We treat both "config FOO" and "menuconfig FOO" as symbol definitions. + # GNU grep and busybox grep both support -E. + while IFS= read -r kf; do + if grep -Eq "^[[:space:]]*(menu)?config[[:space:]]+$sym([[:space:]]|\$)" "$kf"; then + return 0 + fi + done < "$TMP_KCONFIGS" + + return 1 +} + +kept_symbols=0 + +while IFS= read -r line || [ -n "$line" ]; do + # Preserve blank lines exactly. + if [ -z "$line" ]; then + printf '\n' >> "$TMP_OUT" + continue + fi + + case "$line" in + '# 'CONFIG_*' is not set') + # Example: "# CONFIG_FOO is not set" + sym=$(printf '%s\n' "$line" | sed -n 's/^# \(CONFIG_[A-Za-z0-9_][A-Za-z0-9_]*\) is not set$/\1/p') + if [ -n "$sym" ] && symbol_exists "$sym"; then + printf '%s\n' "$line" >> "$TMP_OUT" + kept_symbols=$((kept_symbols + 1)) + fi + ;; + CONFIG_*=*) + # Example: "CONFIG_FOO=y" or "CONFIG_BAR=\"abc\"" + sym=${line%%=*} + if symbol_exists "$sym"; then + printf '%s\n' "$line" >> "$TMP_OUT" + kept_symbols=$((kept_symbols + 1)) + fi + ;; + '#'*) + # Preserve comments and headings. + printf '%s\n' "$line" >> "$TMP_OUT" + ;; + *) + # Preserve any other non-empty line verbatim; fragments normally + # shouldn't contain these, but preserving them is safer than rewriting. + printf '%s\n' "$line" >> "$TMP_OUT" + ;; + esac +done < "$FRAGMENT_FILE" + +if [ "$kept_symbols" -eq 0 ]; then + echo "error: no CONFIG entries remain after filtering $FRAGMENT_FILE" >&2 + exit 4 +fi + +cat "$TMP_OUT" +``` + +This script is intentionally narrow. It does **not** try to solve dependency resolution, force values, or replace `merge_config.sh`. It only answers one question: “Does this symbol exist anywhere in the target kernel’s Kconfig tree?” That is the right preflight step when a fragment spans multiple kernel generations and you need a portable Docker-safe filter before Kconfig proper enforces dependencies. citeturn3view1turn3view2turn3view6 + +## Integration examples + +### Dockerfile usage + +```dockerfile +COPY scripts/filter-kconfig-fragment.sh /usr/local/bin/filter-kconfig-fragment.sh +COPY docker/kernel-extra.config /tmp/kernel-extra.config + +RUN chmod +x /usr/local/bin/filter-kconfig-fragment.sh && \ + test -d /opt/kernel && \ + /usr/local/bin/filter-kconfig-fragment.sh /opt/kernel /tmp/kernel-extra.config \ + > /tmp/kernel-extra.effective.config && \ + /opt/kernel/scripts/kconfig/merge_config.sh -m /opt/kernel/.config /tmp/kernel-extra.effective.config && \ + KCONFIG_WARN_UNKNOWN_SYMBOLS=1 KCONFIG_WERROR=1 \ + make -C /opt/kernel olddefconfig && \ + make -C /opt/kernel modules_prepare +``` + +If `CONFIG_MODVERSIONS=y` or your external modules depend on `Module.symvers`, replace the final `modules_prepare` with a full kernel build step for the relevant targets. The kernel docs explicitly warn that `modules_prepare` does not produce `Module.symvers`. citeturn3view4 + +### Makefile usage + +```make +KERNEL_DIR ?= /opt/kernel +KERNEL_FRAGMENT ?= docker/kernel-extra.config +KERNEL_EFFECTIVE?= build/kernel-extra.effective.config +KCONFIG_FILTER ?= scripts/filter-kconfig-fragment.sh + +$(KERNEL_EFFECTIVE): $(KERNEL_FRAGMENT) $(KCONFIG_FILTER) + mkdir -p $(dir $@) + $(KCONFIG_FILTER) $(KERNEL_DIR) $(KERNEL_FRAGMENT) > $@ + +kernel-merge: $(KERNEL_EFFECTIVE) + $(KERNEL_DIR)/scripts/kconfig/merge_config.sh -m $(KERNEL_DIR)/.config $(KERNEL_EFFECTIVE) + KCONFIG_WARN_UNKNOWN_SYMBOLS=1 KCONFIG_WERROR=1 $(MAKE) -C $(KERNEL_DIR) olddefconfig + +kernel-prepare: kernel-merge + $(MAKE) -C $(KERNEL_DIR) modules_prepare +``` + +### Why the filter is needed + +This filter exists to handle **kernel-version mismatches in fragments**, not to bypass Kconfig. A fragment can be perfectly valid for kernel line A and partially invalid for kernel line B because symbols were renamed, split, removed, or introduced later. Your concrete example, `CONFIG_NETFILTER_XTABLES_LEGACY`, is exactly that class of problem: the symbol may be meaningful in one kernel generation and absent in another. Filtering it out before `merge_config.sh` and `olddefconfig` prevents a false failure from one stale line while still allowing Kconfig to validate all remaining, real symbols and dependencies. Kconfig’s own docs explain that symbols live in a dependency-aware tree and that visibility and legality depend on those relationships, which is why the filter should check **existence only** and leave dependency resolution to Kconfig itself. citeturn3view6turn3view2 + +```mermaid +flowchart LR + A[kernel-extra.config] --> B[filter-kconfig-fragment.sh] + B --> C[kernel-extra.effective.config] + C --> D[merge_config.sh -m .config effective.fragment] + D --> E[make olddefconfig] + E --> F[make modules_prepare or full kernel build] +``` + +## Quick checks and validation commands + +Use these checks before or after wiring the script into Docker: + +### Check whether a symbol exists in the target tree + +```sh +grep -Rnw --include='Kconfig*' \ + -E '^[[:space:]]*(menu)?config[[:space:]]+NETFILTER_XTABLES_LEGACY([[:space:]]|$)' \ + "$KERNEL_DIR" || echo "missing" +``` + +### Show which lines survive filtering + +```sh +scripts/filter-kconfig-fragment.sh "$KERNEL_DIR" docker/kernel-extra.config \ + | grep -E '^(CONFIG_|# CONFIG_)' +``` + +### Confirm the filtered fragment is non-empty + +```sh +scripts/filter-kconfig-fragment.sh "$KERNEL_DIR" docker/kernel-extra.config \ + > /tmp/kernel-extra.effective.config && \ +test -s /tmp/kernel-extra.effective.config +``` + +### Validate after merge and `olddefconfig` + +```sh +"$KERNEL_DIR"/scripts/kconfig/merge_config.sh -m "$KERNEL_DIR"/.config /tmp/kernel-extra.effective.config +KCONFIG_WARN_UNKNOWN_SYMBOLS=1 KCONFIG_WERROR=1 \ +make -C "$KERNEL_DIR" olddefconfig +``` + +### Inspect config drift + +```sh +make -C "$KERNEL_DIR" listnewconfig +``` + +The kernel docs explicitly recommend `listnewconfig` when older configs are reused against newer kernels, and they document `KCONFIG_WARN_UNKNOWN_SYMBOLS` / `KCONFIG_WERROR` for exactly the “unknown symbol in config input” problem this workflow is guarding against. citeturn5view0turn3view2 + +## Sources and notes + +I did not find `filter-kconfig-fragment.sh` in the provided project files, so the replacement above is intentionally conservative and built around **official kernel interfaces** instead of inventing a parallel config system. The authoritative equivalents are: the kernel’s own `scripts/kconfig/merge_config.sh`, whose header states that it merges config fragments and warns about overridden or unresolved values; `scripts/config`, whose built-in usage documents `--enable`, `--disable`, `--module`, `--file`, and related options; the Kconfig documentation for `KCONFIG_WARN_UNKNOWN_SYMBOLS`, `KCONFIG_WERROR`, `KCONFIG_ALLCONFIG`, and `listnewconfig`; the Kconfig language docs for symbol definitions and dependency semantics; and the external-modules docs for `modules_prepare` and the `Module.symvers` caveat. The reproducible-builds kernel docs are also relevant when the surrounding Docker pipeline sets `SOURCE_DATE_EPOCH`, `KBUILD_BUILD_TIMESTAMP`, `KBUILD_BUILD_USER`, and `KBUILD_BUILD_HOST`. citeturn3view1turn3view0turn3view2turn5view0turn3view6turn3view4turn3view3 \ No newline at end of file