261 lines
13 KiB
Markdown
261 lines
13 KiB
Markdown
# `filter-kconfig-fragment.sh` for Reproducible Kernel-Config Filtering
|
||
|
||
## Executive summary
|
||
|
||
I did **not** find a file named `filter-kconfig-fragment.sh` in the uploaded project files or in the supplied ASK/kernel tarballs. What **is** present is the standard kernel-side machinery you would use around such a filter: the Linux kernel’s own `scripts/kconfig/merge_config.sh` for fragment merging, `scripts/config` for imperative `.config` edits, and documented Kconfig controls such as `KCONFIG_WARN_UNKNOWN_SYMBOLS`, `KCONFIG_WERROR`, and `KCONFIG_ALLCONFIG`. The kernel tree itself also uses `merge_config.sh` followed by `olddefconfig` when it builds from config fragments. citeturn3view1turn0search2turn3view2
|
||
|
||
A small pre-filter is still useful, because neither `merge_config.sh` nor `scripts/config` is meant to silently rewrite a fragment for a **different kernel generation**. `merge_config.sh` merges and warns; `scripts/config` edits `.config`; Kconfig enforces symbol existence and dependency rules. When a fragment contains a symbol that simply does not exist in the target tree, such as `CONFIG_NETFILTER_XTABLES_LEGACY` in your reported 6.12-based tree, a pre-filter avoids an avoidable mismatch by dropping only the absent symbol while preserving ordering, comments, and all valid settings. That is the narrow job of the replacement script below. citeturn3view1turn3view2turn3view6
|
||
|
||
The replacement script in this report is POSIX `sh`, offline-safe, requires only standard Unix tools (`find`, `grep`, `sed`, `mktemp`, `cat`), validates inputs, checks for `scripts/kconfig/merge_config.sh`, preserves comments and ordering, emits only symbols that exist in the target Kconfig tree, and exits non-zero if nothing usable remains after filtering. It is designed specifically for Docker builder stages and reproducible kernel-config flows. citeturn3view1turn3view2turn3view4
|
||
|
||
## What I found and what to use instead
|
||
|
||
The Linux kernel already ships the two main utilities you should treat as authoritative.
|
||
|
||
`merge_config.sh` is the standard fragment-merging tool in the kernel tree. Its own header says it “takes a list of config fragment values, and merges them one by one,” and that it warns about overridden values and symbols that did not make it into the final `.config` because of dependencies or symbol removal. The kernel build system also invokes `scripts/kconfig/merge_config.sh -m $(KCONFIG_CONFIG) ...` and then runs `olddefconfig` for fragment-based config targets. citeturn3view1turn0search2
|
||
|
||
`scripts/config` is the standard imperative editor for `.config`. Its built-in usage text documents `--enable`, `--disable`, `--module`, `--set-str`, `--set-val`, `--undefine`, `--state`, and `--file`, and also states that it does **not** validate `.config` immediately; that validation happens at the next `make` step. The kernel docs also contain concrete examples of `./scripts/config -e ...` and `-m ...` being used to enable required options. citeturn3view0turn3view5
|
||
|
||
Kconfig itself exposes the exact controls you want for drift detection: `KCONFIG_WARN_UNKNOWN_SYMBOLS` warns on unrecognized config symbols, `KCONFIG_WERROR` turns those warnings into errors, and `KCONFIG_ALLCONFIG` provides a documented “mini-config” mechanism for all*config targets. The same docs also recommend `make listnewconfig` to surface newly introduced symbols. citeturn3view2turn5view0
|
||
|
||
For external module builds, the kernel docs also matter here: `modules_prepare` can prepare a tree for external modules, but it does **not** produce `Module.symvers` if `CONFIG_MODVERSIONS` matters, so a full kernel build is still required in that case. That is the main reason a strict builder pipeline should validate kernel config and preparation before invoking out-of-tree module builds. citeturn3view4
|
||
|
||
## Minimal portable replacement script
|
||
|
||
Assumptions for this script:
|
||
|
||
- `KERNEL_DIR` is a Linux kernel source tree root.
|
||
- The tree contains `scripts/kconfig/merge_config.sh`.
|
||
- Config symbols are defined in `Kconfig*` files somewhere under `KERNEL_DIR`.
|
||
- The fragment format is standard Kconfig fragment syntax: `CONFIG_FOO=...` and `# CONFIG_FOO is not set`.
|
||
- The builder environment has POSIX `sh`, `find`, `grep`, `sed`, `mktemp`, `cat`, and `rm`.
|
||
|
||
**`scripts/filter-kconfig-fragment.sh`**
|
||
|
||
```sh
|
||
#!/bin/sh
|
||
# filter-kconfig-fragment.sh
|
||
#
|
||
# Purpose:
|
||
# Emit an "effective" Kconfig fragment that preserves comments and ordering
|
||
# but drops CONFIG symbols that do not exist in the target kernel tree.
|
||
#
|
||
# Usage:
|
||
# filter-kconfig-fragment.sh KERNEL_DIR FRAGMENT_FILE > effective.config
|
||
#
|
||
# Exit codes:
|
||
# 2 invalid usage / missing required files
|
||
# 3 no Kconfig files were found in KERNEL_DIR
|
||
# 4 fragment had no surviving CONFIG lines after filtering
|
||
|
||
set -eu
|
||
|
||
usage() {
|
||
echo "usage: $0 KERNEL_DIR FRAGMENT_FILE" >&2
|
||
exit 2
|
||
}
|
||
|
||
[ "$#" -eq 2 ] || usage
|
||
|
||
KERNEL_DIR=$1
|
||
FRAGMENT_FILE=$2
|
||
MERGE_SCRIPT=$KERNEL_DIR/scripts/kconfig/merge_config.sh
|
||
|
||
[ -d "$KERNEL_DIR" ] || {
|
||
echo "error: KERNEL_DIR is not a directory: $KERNEL_DIR" >&2
|
||
exit 2
|
||
}
|
||
|
||
[ -r "$FRAGMENT_FILE" ] || {
|
||
echo "error: FRAGMENT_FILE is not readable: $FRAGMENT_FILE" >&2
|
||
exit 2
|
||
}
|
||
|
||
[ -s "$FRAGMENT_FILE" ] || {
|
||
echo "error: FRAGMENT_FILE is empty: $FRAGMENT_FILE" >&2
|
||
exit 2
|
||
}
|
||
|
||
[ -r "$MERGE_SCRIPT" ] || {
|
||
echo "error: required kernel utility missing: $MERGE_SCRIPT" >&2
|
||
exit 2
|
||
}
|
||
|
||
TMP_KCONFIGS=$(mktemp)
|
||
TMP_OUT=$(mktemp)
|
||
trap 'rm -f "$TMP_KCONFIGS" "$TMP_OUT"' EXIT HUP INT TERM
|
||
|
||
# Index all Kconfig files once. Searching the whole tree is simpler and more
|
||
# robust than hardcoding a small directory subset.
|
||
find "$KERNEL_DIR" -type f \( -name 'Kconfig' -o -name 'Kconfig*' \) | sort > "$TMP_KCONFIGS"
|
||
|
||
[ -s "$TMP_KCONFIGS" ] || {
|
||
echo "error: no Kconfig files found under $KERNEL_DIR" >&2
|
||
exit 3
|
||
}
|
||
|
||
symbol_exists() {
|
||
# Accept either CONFIG_FOO or FOO.
|
||
sym=$1
|
||
case "$sym" in
|
||
CONFIG_*) sym=${sym#CONFIG_} ;;
|
||
esac
|
||
|
||
# We treat both "config FOO" and "menuconfig FOO" as symbol definitions.
|
||
# GNU grep and busybox grep both support -E.
|
||
while IFS= read -r kf; do
|
||
if grep -Eq "^[[:space:]]*(menu)?config[[:space:]]+$sym([[:space:]]|\$)" "$kf"; then
|
||
return 0
|
||
fi
|
||
done < "$TMP_KCONFIGS"
|
||
|
||
return 1
|
||
}
|
||
|
||
kept_symbols=0
|
||
|
||
while IFS= read -r line || [ -n "$line" ]; do
|
||
# Preserve blank lines exactly.
|
||
if [ -z "$line" ]; then
|
||
printf '\n' >> "$TMP_OUT"
|
||
continue
|
||
fi
|
||
|
||
case "$line" in
|
||
'# 'CONFIG_*' is not set')
|
||
# Example: "# CONFIG_FOO is not set"
|
||
sym=$(printf '%s\n' "$line" | sed -n 's/^# \(CONFIG_[A-Za-z0-9_][A-Za-z0-9_]*\) is not set$/\1/p')
|
||
if [ -n "$sym" ] && symbol_exists "$sym"; then
|
||
printf '%s\n' "$line" >> "$TMP_OUT"
|
||
kept_symbols=$((kept_symbols + 1))
|
||
fi
|
||
;;
|
||
CONFIG_*=*)
|
||
# Example: "CONFIG_FOO=y" or "CONFIG_BAR=\"abc\""
|
||
sym=${line%%=*}
|
||
if symbol_exists "$sym"; then
|
||
printf '%s\n' "$line" >> "$TMP_OUT"
|
||
kept_symbols=$((kept_symbols + 1))
|
||
fi
|
||
;;
|
||
'#'*)
|
||
# Preserve comments and headings.
|
||
printf '%s\n' "$line" >> "$TMP_OUT"
|
||
;;
|
||
*)
|
||
# Preserve any other non-empty line verbatim; fragments normally
|
||
# shouldn't contain these, but preserving them is safer than rewriting.
|
||
printf '%s\n' "$line" >> "$TMP_OUT"
|
||
;;
|
||
esac
|
||
done < "$FRAGMENT_FILE"
|
||
|
||
if [ "$kept_symbols" -eq 0 ]; then
|
||
echo "error: no CONFIG entries remain after filtering $FRAGMENT_FILE" >&2
|
||
exit 4
|
||
fi
|
||
|
||
cat "$TMP_OUT"
|
||
```
|
||
|
||
This script is intentionally narrow. It does **not** try to solve dependency resolution, force values, or replace `merge_config.sh`. It only answers one question: “Does this symbol exist anywhere in the target kernel’s Kconfig tree?” That is the right preflight step when a fragment spans multiple kernel generations and you need a portable Docker-safe filter before Kconfig proper enforces dependencies. citeturn3view1turn3view2turn3view6
|
||
|
||
## Integration examples
|
||
|
||
### Dockerfile usage
|
||
|
||
```dockerfile
|
||
COPY scripts/filter-kconfig-fragment.sh /usr/local/bin/filter-kconfig-fragment.sh
|
||
COPY docker/kernel-extra.config /tmp/kernel-extra.config
|
||
|
||
RUN chmod +x /usr/local/bin/filter-kconfig-fragment.sh && \
|
||
test -d /opt/kernel && \
|
||
/usr/local/bin/filter-kconfig-fragment.sh /opt/kernel /tmp/kernel-extra.config \
|
||
> /tmp/kernel-extra.effective.config && \
|
||
/opt/kernel/scripts/kconfig/merge_config.sh -m /opt/kernel/.config /tmp/kernel-extra.effective.config && \
|
||
KCONFIG_WARN_UNKNOWN_SYMBOLS=1 KCONFIG_WERROR=1 \
|
||
make -C /opt/kernel olddefconfig && \
|
||
make -C /opt/kernel modules_prepare
|
||
```
|
||
|
||
If `CONFIG_MODVERSIONS=y` or your external modules depend on `Module.symvers`, replace the final `modules_prepare` with a full kernel build step for the relevant targets. The kernel docs explicitly warn that `modules_prepare` does not produce `Module.symvers`. citeturn3view4
|
||
|
||
### Makefile usage
|
||
|
||
```make
|
||
KERNEL_DIR ?= /opt/kernel
|
||
KERNEL_FRAGMENT ?= docker/kernel-extra.config
|
||
KERNEL_EFFECTIVE?= build/kernel-extra.effective.config
|
||
KCONFIG_FILTER ?= scripts/filter-kconfig-fragment.sh
|
||
|
||
$(KERNEL_EFFECTIVE): $(KERNEL_FRAGMENT) $(KCONFIG_FILTER)
|
||
mkdir -p $(dir $@)
|
||
$(KCONFIG_FILTER) $(KERNEL_DIR) $(KERNEL_FRAGMENT) > $@
|
||
|
||
kernel-merge: $(KERNEL_EFFECTIVE)
|
||
$(KERNEL_DIR)/scripts/kconfig/merge_config.sh -m $(KERNEL_DIR)/.config $(KERNEL_EFFECTIVE)
|
||
KCONFIG_WARN_UNKNOWN_SYMBOLS=1 KCONFIG_WERROR=1 $(MAKE) -C $(KERNEL_DIR) olddefconfig
|
||
|
||
kernel-prepare: kernel-merge
|
||
$(MAKE) -C $(KERNEL_DIR) modules_prepare
|
||
```
|
||
|
||
### Why the filter is needed
|
||
|
||
This filter exists to handle **kernel-version mismatches in fragments**, not to bypass Kconfig. A fragment can be perfectly valid for kernel line A and partially invalid for kernel line B because symbols were renamed, split, removed, or introduced later. Your concrete example, `CONFIG_NETFILTER_XTABLES_LEGACY`, is exactly that class of problem: the symbol may be meaningful in one kernel generation and absent in another. Filtering it out before `merge_config.sh` and `olddefconfig` prevents a false failure from one stale line while still allowing Kconfig to validate all remaining, real symbols and dependencies. Kconfig’s own docs explain that symbols live in a dependency-aware tree and that visibility and legality depend on those relationships, which is why the filter should check **existence only** and leave dependency resolution to Kconfig itself. citeturn3view6turn3view2
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
A[kernel-extra.config] --> B[filter-kconfig-fragment.sh]
|
||
B --> C[kernel-extra.effective.config]
|
||
C --> D[merge_config.sh -m .config effective.fragment]
|
||
D --> E[make olddefconfig]
|
||
E --> F[make modules_prepare or full kernel build]
|
||
```
|
||
|
||
## Quick checks and validation commands
|
||
|
||
Use these checks before or after wiring the script into Docker:
|
||
|
||
### Check whether a symbol exists in the target tree
|
||
|
||
```sh
|
||
grep -Rnw --include='Kconfig*' \
|
||
-E '^[[:space:]]*(menu)?config[[:space:]]+NETFILTER_XTABLES_LEGACY([[:space:]]|$)' \
|
||
"$KERNEL_DIR" || echo "missing"
|
||
```
|
||
|
||
### Show which lines survive filtering
|
||
|
||
```sh
|
||
scripts/filter-kconfig-fragment.sh "$KERNEL_DIR" docker/kernel-extra.config \
|
||
| grep -E '^(CONFIG_|# CONFIG_)'
|
||
```
|
||
|
||
### Confirm the filtered fragment is non-empty
|
||
|
||
```sh
|
||
scripts/filter-kconfig-fragment.sh "$KERNEL_DIR" docker/kernel-extra.config \
|
||
> /tmp/kernel-extra.effective.config && \
|
||
test -s /tmp/kernel-extra.effective.config
|
||
```
|
||
|
||
### Validate after merge and `olddefconfig`
|
||
|
||
```sh
|
||
"$KERNEL_DIR"/scripts/kconfig/merge_config.sh -m "$KERNEL_DIR"/.config /tmp/kernel-extra.effective.config
|
||
KCONFIG_WARN_UNKNOWN_SYMBOLS=1 KCONFIG_WERROR=1 \
|
||
make -C "$KERNEL_DIR" olddefconfig
|
||
```
|
||
|
||
### Inspect config drift
|
||
|
||
```sh
|
||
make -C "$KERNEL_DIR" listnewconfig
|
||
```
|
||
|
||
The kernel docs explicitly recommend `listnewconfig` when older configs are reused against newer kernels, and they document `KCONFIG_WARN_UNKNOWN_SYMBOLS` / `KCONFIG_WERROR` for exactly the “unknown symbol in config input” problem this workflow is guarding against. citeturn5view0turn3view2
|
||
|
||
## Sources and notes
|
||
|
||
I did not find `filter-kconfig-fragment.sh` in the provided project files, so the replacement above is intentionally conservative and built around **official kernel interfaces** instead of inventing a parallel config system. The authoritative equivalents are: the kernel’s own `scripts/kconfig/merge_config.sh`, whose header states that it merges config fragments and warns about overridden or unresolved values; `scripts/config`, whose built-in usage documents `--enable`, `--disable`, `--module`, `--file`, and related options; the Kconfig documentation for `KCONFIG_WARN_UNKNOWN_SYMBOLS`, `KCONFIG_WERROR`, `KCONFIG_ALLCONFIG`, and `listnewconfig`; the Kconfig language docs for symbol definitions and dependency semantics; and the external-modules docs for `modules_prepare` and the `Module.symvers` caveat. The reproducible-builds kernel docs are also relevant when the surrounding Docker pipeline sets `SOURCE_DATE_EPOCH`, `KBUILD_BUILD_TIMESTAMP`, `KBUILD_BUILD_USER`, and `KBUILD_BUILD_HOST`. citeturn3view1turn3view0turn3view2turn5view0turn3view6turn3view4turn3view3 |