ASK research docs 4

docs/ask-deepresearch-4.md

@@ -0,0 +1,261 @@
+# `filter-kconfig-fragment.sh` for Reproducible Kernel-Config Filtering
+
+## Executive summary
+
+I did **not** find a file named `filter-kconfig-fragment.sh` in the uploaded project files or in the supplied ASK/kernel tarballs. What **is** present is the standard kernel-side machinery you would use around such a filter: the Linux kernel’s own `scripts/kconfig/merge_config.sh` for fragment merging, `scripts/config` for imperative `.config` edits, and documented Kconfig controls such as `KCONFIG_WARN_UNKNOWN_SYMBOLS`, `KCONFIG_WERROR`, and `KCONFIG_ALLCONFIG`. The kernel tree itself also uses `merge_config.sh` followed by `olddefconfig` when it builds from config fragments. citeturn3view1turn0search2turn3view2
+
+A small pre-filter is still useful, because neither `merge_config.sh` nor `scripts/config` is meant to silently rewrite a fragment for a **different kernel generation**. `merge_config.sh` merges and warns; `scripts/config` edits `.config`; Kconfig enforces symbol existence and dependency rules. When a fragment contains a symbol that simply does not exist in the target tree, such as `CONFIG_NETFILTER_XTABLES_LEGACY` in your reported 6.12-based tree, a pre-filter avoids an avoidable mismatch by dropping only the absent symbol while preserving ordering, comments, and all valid settings. That is the narrow job of the replacement script below. citeturn3view1turn3view2turn3view6
+
+The replacement script in this report is POSIX `sh`, offline-safe, requires only standard Unix tools (`find`, `grep`, `sed`, `mktemp`, `cat`), validates inputs, checks for `scripts/kconfig/merge_config.sh`, preserves comments and ordering, emits only symbols that exist in the target Kconfig tree, and exits non-zero if nothing usable remains after filtering. It is designed specifically for Docker builder stages and reproducible kernel-config flows. citeturn3view1turn3view2turn3view4
+
+## What I found and what to use instead
+
+The Linux kernel already ships the two main utilities you should treat as authoritative.
+
+`merge_config.sh` is the standard fragment-merging tool in the kernel tree. Its own header says it “takes a list of config fragment values, and merges them one by one,” and that it warns about overridden values and symbols that did not make it into the final `.config` because of dependencies or symbol removal. The kernel build system also invokes `scripts/kconfig/merge_config.sh -m $(KCONFIG_CONFIG) ...` and then runs `olddefconfig` for fragment-based config targets. citeturn3view1turn0search2
+
+`scripts/config` is the standard imperative editor for `.config`. Its built-in usage text documents `--enable`, `--disable`, `--module`, `--set-str`, `--set-val`, `--undefine`, `--state`, and `--file`, and also states that it does **not** validate `.config` immediately; that validation happens at the next `make` step. The kernel docs also contain concrete examples of `./scripts/config -e ...` and `-m ...` being used to enable required options. citeturn3view0turn3view5
+
+Kconfig itself exposes the exact controls you want for drift detection: `KCONFIG_WARN_UNKNOWN_SYMBOLS` warns on unrecognized config symbols, `KCONFIG_WERROR` turns those warnings into errors, and `KCONFIG_ALLCONFIG` provides a documented “mini-config” mechanism for all*config targets. The same docs also recommend `make listnewconfig` to surface newly introduced symbols. citeturn3view2turn5view0
+
+For external module builds, the kernel docs also matter here: `modules_prepare` can prepare a tree for external modules, but it does **not** produce `Module.symvers` if `CONFIG_MODVERSIONS` matters, so a full kernel build is still required in that case. That is the main reason a strict builder pipeline should validate kernel config and preparation before invoking out-of-tree module builds. citeturn3view4
+
+## Minimal portable replacement script
+
+Assumptions for this script:
+
+- `KERNEL_DIR` is a Linux kernel source tree root.
+- The tree contains `scripts/kconfig/merge_config.sh`.
+- Config symbols are defined in `Kconfig*` files somewhere under `KERNEL_DIR`.
+- The fragment format is standard Kconfig fragment syntax: `CONFIG_FOO=...` and `# CONFIG_FOO is not set`.
+- The builder environment has POSIX `sh`, `find`, `grep`, `sed`, `mktemp`, `cat`, and `rm`.
+
+**`scripts/filter-kconfig-fragment.sh`**
+
+```sh
+#!/bin/sh
+# filter-kconfig-fragment.sh
+#
+# Purpose:
+#   Emit an "effective" Kconfig fragment that preserves comments and ordering
+#   but drops CONFIG symbols that do not exist in the target kernel tree.
+#
+# Usage:
+#   filter-kconfig-fragment.sh KERNEL_DIR FRAGMENT_FILE > effective.config
+#
+# Exit codes:
+#   2  invalid usage / missing required files
+#   3  no Kconfig files were found in KERNEL_DIR
+#   4  fragment had no surviving CONFIG lines after filtering
+
+set -eu
+
+usage() {
+    echo "usage: $0 KERNEL_DIR FRAGMENT_FILE" >&2
+    exit 2
+}
+
+[ "$#" -eq 2 ] || usage
+
+KERNEL_DIR=$1
+FRAGMENT_FILE=$2
+MERGE_SCRIPT=$KERNEL_DIR/scripts/kconfig/merge_config.sh
+
+[ -d "$KERNEL_DIR" ] || {
+    echo "error: KERNEL_DIR is not a directory: $KERNEL_DIR" >&2
+    exit 2
+}
+
+[ -r "$FRAGMENT_FILE" ] || {
+    echo "error: FRAGMENT_FILE is not readable: $FRAGMENT_FILE" >&2
+    exit 2
+}
+
+[ -s "$FRAGMENT_FILE" ] || {
+    echo "error: FRAGMENT_FILE is empty: $FRAGMENT_FILE" >&2
+    exit 2
+}
+
+[ -r "$MERGE_SCRIPT" ] || {
+    echo "error: required kernel utility missing: $MERGE_SCRIPT" >&2
+    exit 2
+}
+
+TMP_KCONFIGS=$(mktemp)
+TMP_OUT=$(mktemp)
+trap 'rm -f "$TMP_KCONFIGS" "$TMP_OUT"' EXIT HUP INT TERM
+
+# Index all Kconfig files once. Searching the whole tree is simpler and more
+# robust than hardcoding a small directory subset.
+find "$KERNEL_DIR" -type f \( -name 'Kconfig' -o -name 'Kconfig*' \) | sort > "$TMP_KCONFIGS"
+
+[ -s "$TMP_KCONFIGS" ] || {
+    echo "error: no Kconfig files found under $KERNEL_DIR" >&2
+    exit 3
+}
+
+symbol_exists() {
+    # Accept either CONFIG_FOO or FOO.
+    sym=$1
+    case "$sym" in
+        CONFIG_*) sym=${sym#CONFIG_} ;;
+    esac
+
+    # We treat both "config FOO" and "menuconfig FOO" as symbol definitions.
+    # GNU grep and busybox grep both support -E.
+    while IFS= read -r kf; do
+        if grep -Eq "^[[:space:]]*(menu)?config[[:space:]]+$sym([[:space:]]|\$)" "$kf"; then
+            return 0
+        fi
+    done < "$TMP_KCONFIGS"
+
+    return 1
+}
+
+kept_symbols=0
+
+while IFS= read -r line || [ -n "$line" ]; do
+    # Preserve blank lines exactly.
+    if [ -z "$line" ]; then
+        printf '\n' >> "$TMP_OUT"
+        continue
+    fi
+
+    case "$line" in
+        '# 'CONFIG_*' is not set')
+            # Example: "# CONFIG_FOO is not set"
+            sym=$(printf '%s\n' "$line" | sed -n 's/^# \(CONFIG_[A-Za-z0-9_][A-Za-z0-9_]*\) is not set$/\1/p')
+            if [ -n "$sym" ] && symbol_exists "$sym"; then
+                printf '%s\n' "$line" >> "$TMP_OUT"
+                kept_symbols=$((kept_symbols + 1))
+            fi
+            ;;
+        CONFIG_*=*)
+            # Example: "CONFIG_FOO=y" or "CONFIG_BAR=\"abc\""
+            sym=${line%%=*}
+            if symbol_exists "$sym"; then
+                printf '%s\n' "$line" >> "$TMP_OUT"
+                kept_symbols=$((kept_symbols + 1))
+            fi
+            ;;
+        '#'*)
+            # Preserve comments and headings.
+            printf '%s\n' "$line" >> "$TMP_OUT"
+            ;;
+        *)
+            # Preserve any other non-empty line verbatim; fragments normally
+            # shouldn't contain these, but preserving them is safer than rewriting.
+            printf '%s\n' "$line" >> "$TMP_OUT"
+            ;;
+    esac
+done < "$FRAGMENT_FILE"
+
+if [ "$kept_symbols" -eq 0 ]; then
+    echo "error: no CONFIG entries remain after filtering $FRAGMENT_FILE" >&2
+    exit 4
+fi
+
+cat "$TMP_OUT"
+```
+
+This script is intentionally narrow. It does **not** try to solve dependency resolution, force values, or replace `merge_config.sh`. It only answers one question: “Does this symbol exist anywhere in the target kernel’s Kconfig tree?” That is the right preflight step when a fragment spans multiple kernel generations and you need a portable Docker-safe filter before Kconfig proper enforces dependencies. citeturn3view1turn3view2turn3view6
+
+## Integration examples
+
+### Dockerfile usage
+
+```dockerfile
+COPY scripts/filter-kconfig-fragment.sh /usr/local/bin/filter-kconfig-fragment.sh
+COPY docker/kernel-extra.config /tmp/kernel-extra.config
+
+RUN chmod +x /usr/local/bin/filter-kconfig-fragment.sh && \
+    test -d /opt/kernel && \
+    /usr/local/bin/filter-kconfig-fragment.sh /opt/kernel /tmp/kernel-extra.config \
+      > /tmp/kernel-extra.effective.config && \
+    /opt/kernel/scripts/kconfig/merge_config.sh -m /opt/kernel/.config /tmp/kernel-extra.effective.config && \
+    KCONFIG_WARN_UNKNOWN_SYMBOLS=1 KCONFIG_WERROR=1 \
+    make -C /opt/kernel olddefconfig && \
+    make -C /opt/kernel modules_prepare
+```
+
+If `CONFIG_MODVERSIONS=y` or your external modules depend on `Module.symvers`, replace the final `modules_prepare` with a full kernel build step for the relevant targets. The kernel docs explicitly warn that `modules_prepare` does not produce `Module.symvers`. citeturn3view4
+
+### Makefile usage
+
+```make
+KERNEL_DIR      ?= /opt/kernel
+KERNEL_FRAGMENT ?= docker/kernel-extra.config
+KERNEL_EFFECTIVE?= build/kernel-extra.effective.config
+KCONFIG_FILTER  ?= scripts/filter-kconfig-fragment.sh
+
+$(KERNEL_EFFECTIVE): $(KERNEL_FRAGMENT) $(KCONFIG_FILTER)
+	mkdir -p $(dir $@)
+	$(KCONFIG_FILTER) $(KERNEL_DIR) $(KERNEL_FRAGMENT) > $@
+
+kernel-merge: $(KERNEL_EFFECTIVE)
+	$(KERNEL_DIR)/scripts/kconfig/merge_config.sh -m $(KERNEL_DIR)/.config $(KERNEL_EFFECTIVE)
+	KCONFIG_WARN_UNKNOWN_SYMBOLS=1 KCONFIG_WERROR=1 $(MAKE) -C $(KERNEL_DIR) olddefconfig
+
+kernel-prepare: kernel-merge
+	$(MAKE) -C $(KERNEL_DIR) modules_prepare
+```
+
+### Why the filter is needed
+
+This filter exists to handle **kernel-version mismatches in fragments**, not to bypass Kconfig. A fragment can be perfectly valid for kernel line A and partially invalid for kernel line B because symbols were renamed, split, removed, or introduced later. Your concrete example, `CONFIG_NETFILTER_XTABLES_LEGACY`, is exactly that class of problem: the symbol may be meaningful in one kernel generation and absent in another. Filtering it out before `merge_config.sh` and `olddefconfig` prevents a false failure from one stale line while still allowing Kconfig to validate all remaining, real symbols and dependencies. Kconfig’s own docs explain that symbols live in a dependency-aware tree and that visibility and legality depend on those relationships, which is why the filter should check **existence only** and leave dependency resolution to Kconfig itself. citeturn3view6turn3view2
+
+```mermaid
+flowchart LR
+    A[kernel-extra.config] --> B[filter-kconfig-fragment.sh]
+    B --> C[kernel-extra.effective.config]
+    C --> D[merge_config.sh -m .config effective.fragment]
+    D --> E[make olddefconfig]
+    E --> F[make modules_prepare or full kernel build]
+```
+
+## Quick checks and validation commands
+
+Use these checks before or after wiring the script into Docker:
+
+### Check whether a symbol exists in the target tree
+
+```sh
+grep -Rnw --include='Kconfig*' \
+  -E '^[[:space:]]*(menu)?config[[:space:]]+NETFILTER_XTABLES_LEGACY([[:space:]]|$)' \
+  "$KERNEL_DIR" || echo "missing"
+```
+
+### Show which lines survive filtering
+
+```sh
+scripts/filter-kconfig-fragment.sh "$KERNEL_DIR" docker/kernel-extra.config \
+  | grep -E '^(CONFIG_|# CONFIG_)'
+```
+
+### Confirm the filtered fragment is non-empty
+
+```sh
+scripts/filter-kconfig-fragment.sh "$KERNEL_DIR" docker/kernel-extra.config \
+  > /tmp/kernel-extra.effective.config && \
+test -s /tmp/kernel-extra.effective.config
+```
+
+### Validate after merge and `olddefconfig`
+
+```sh
+"$KERNEL_DIR"/scripts/kconfig/merge_config.sh -m "$KERNEL_DIR"/.config /tmp/kernel-extra.effective.config
+KCONFIG_WARN_UNKNOWN_SYMBOLS=1 KCONFIG_WERROR=1 \
+make -C "$KERNEL_DIR" olddefconfig
+```
+
+### Inspect config drift
+
+```sh
+make -C "$KERNEL_DIR" listnewconfig
+```
+
+The kernel docs explicitly recommend `listnewconfig` when older configs are reused against newer kernels, and they document `KCONFIG_WARN_UNKNOWN_SYMBOLS` / `KCONFIG_WERROR` for exactly the “unknown symbol in config input” problem this workflow is guarding against. citeturn5view0turn3view2
+
+## Sources and notes
+
+I did not find `filter-kconfig-fragment.sh` in the provided project files, so the replacement above is intentionally conservative and built around **official kernel interfaces** instead of inventing a parallel config system. The authoritative equivalents are: the kernel’s own `scripts/kconfig/merge_config.sh`, whose header states that it merges config fragments and warns about overridden or unresolved values; `scripts/config`, whose built-in usage documents `--enable`, `--disable`, `--module`, `--file`, and related options; the Kconfig documentation for `KCONFIG_WARN_UNKNOWN_SYMBOLS`, `KCONFIG_WERROR`, `KCONFIG_ALLCONFIG`, and `listnewconfig`; the Kconfig language docs for symbol definitions and dependency semantics; and the external-modules docs for `modules_prepare` and the `Module.symvers` caveat. The reproducible-builds kernel docs are also relevant when the surrounding Docker pipeline sets `SOURCE_DATE_EPOCH`, `KBUILD_BUILD_TIMESTAMP`, `KBUILD_BUILD_USER`, and `KBUILD_BUILD_HOST`. citeturn3view1turn3view0turn3view2turn5view0turn3view6turn3view4turn3view3