RFC: Package Necessary Files To Build out-of-tree modules

RFC: Package Necessary Files To Build out-of-tree modules

[Federico Vaga]

Hello,

I did this little patch to do some experiments around the idea of having the
kernel building system being able to produce a tarball containing all the
required files for building out-of-tree modules. The direct advantage would be
mainly for distributions' packagers who do not need anymore to specify the list
of required files - for all distributions.

This is just a draft, to see if the idea has some value for the community. For
sure the list of files is incomplete, and in other cases it adds more than
necessary. I see `rsync` is already in use in other files, so I'm not
introducing a new dependency, but of course it can be easily replaced with
something else. I copied from `scripts/package/buildtar` the parts about
creating the tarball.

so, ready for comments :)

[PATCH 1/2] package: add tar development package for 3rd party modules

[Federico Vaga]

Most, if not all, Linux distributions provides a Linux development
package which purpose is to support the building of out-of-tree modules
without providing the entire source tree.

What ends up in this development directory is a mixture of source
files (mainly headers) and generated ones (headers, and tools produced
by `make modules_prepare`).

This patch is an attempt to generate a tarball archive containing all
required files to build external modules. It could be than reused by
packagers.

Signed-off-by: Federico Vaga <federico.vaga@cern.ch>
---
 Makefile                       |   2 +-
 scripts/Makefile.package       |  13 +++
 scripts/package/buildtar-devel | 207 +++++++++++++++++++++++++++++++++
 3 files changed, 221 insertions(+), 1 deletion(-)
 create mode 100644 scripts/package/buildtar-devel

diff --git a/Makefile b/Makefile
index cfbe6a7de640..36a58394ce16 100644
--- a/Makefile
+++ b/Makefile
@@ -1578,7 +1578,7 @@ CLEAN_FILES += include/ksym vmlinux.symvers modules-only.symvers \
 # Directories & files removed with 'make mrproper'
 MRPROPER_FILES += include/config include/generated          \
 		  arch/$(SRCARCH)/include/generated .objdiff \
-		  debian snap tar-install \
+		  debian snap tar-install* \
 		  .config .config.old .version \
 		  Module.symvers \
 		  certs/signing_key.pem \
diff --git a/scripts/Makefile.package b/scripts/Makefile.package
index 8bbcced67c22..9523a4dfaee5 100644
--- a/scripts/Makefile.package
+++ b/scripts/Makefile.package
@@ -112,6 +112,13 @@ $(tar-pkgs):
 	$(MAKE) -f $(srctree)/Makefile
 	+$(CONFIG_SHELL) $(srctree)/scripts/package/buildtar $@
 
+tar-dev-pkgs := dir-dev-pkg tar-dev-pkg targz-dev-pkg tarbz2-dev-pkg
+tar-dev-pkgs += tarxz-dev-pkg tarzst-dev-pkg
+PHONY += $(tar-dev-pkgs)
+$(tar-dev-pkgs):
+	$(MAKE) -f $(srctree)/Makefile
+	+$(CONFIG_SHELL) $(srctree)/scripts/package/buildtar-devel $@
+
 # perf-pkg - generate a source tarball with perf source
 # ---------------------------------------------------------------------------
 
@@ -159,6 +166,12 @@ help:
 	@echo '  tarbz2-pkg          - Build the kernel as a bzip2 compressed tarball'
 	@echo '  tarxz-pkg           - Build the kernel as a xz compressed tarball'
 	@echo '  tarzst-pkg          - Build the kernel as a zstd compressed tarball'
+	@echo '  dir-dev-pkg         - Module development as a plain directory structure'
+	@echo '  tar-dev-pkg         - Module development as an uncompressed tarball'
+	@echo '  targz-dev-pkg       - Module development as a gzip compressed tarball'
+	@echo '  tarbz2-dev-pkg      - Module development as a bzip2 compressed tarball'
+	@echo '  tarxz-dev-pkg       - Module development as a xz compressed tarball'
+	@echo '  tarzst-dev-pkg      - Module development as a zstd compressed tarball'
 	@echo '  perf-tar-src-pkg    - Build $(perf-tar).tar source tarball'
 	@echo '  perf-targz-src-pkg  - Build $(perf-tar).tar.gz source tarball'
 	@echo '  perf-tarbz2-src-pkg - Build $(perf-tar).tar.bz2 source tarball'
diff --git a/scripts/package/buildtar-devel b/scripts/package/buildtar-devel
new file mode 100644
index 000000000000..87706d50a302
--- /dev/null
+++ b/scripts/package/buildtar-devel
@@ -0,0 +1,207 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+
+#
+# (C) 2022 CERN (home.cern)
+# Author Federico Vaga <federico.vaga@cern.ch>
+#
+# This script is used to build a 3rd party kernel module development tarball
+# from the currently prepared kernel.
+#
+
+set -ex
+
+#
+# Some variables and settings used throughout the script
+#
+tmpdir="${abs_objtree}/tar-install-dev"
+tarball="${abs_objtree}/linux-${KERNELRELEASE}-${ARCH}-dev.tar"
+
+#
+# Figure out how to compress, if requested at all
+#
+case "${1}" in
+    dir-pkg-dev|tar-pkg-dev)
+        opts=
+		;;
+	targz-pkg-dev)
+		opts="-I ${KGZIP}"
+		tarball=${tarball}.gz
+		;;
+	tarbz2-pkg-dev)
+		opts="-I ${KBZIP2}"
+		tarball=${tarball}.bz2
+		;;
+	tarxz-pkg-dev)
+		opts="-I ${XZ}"
+		tarball=${tarball}.xz
+		;;
+	tarzst-pkg-dev)
+		opts="-I ${ZSTD}"
+		tarball=${tarball}.zst
+		;;
+	*)
+		echo "Unknown tarball target \"${1}\" requested, please add it to ${0}." >&2
+		exit 1
+		;;
+esac
+
+#
+# Clean-up and re-create the temporary directory
+#
+
+rm -rf -- "${tmpdir}"
+mkdir -p -- "${tmpdir}"
+
+#
+# Copy required files
+#
+FILTER_FILE=$(mktemp)
+cat <<EOF > ${FILTER_FILE}
+
+#
+# Include the following files and directories required to build external modules
+#
+
++ /arch/
++ /arch/${ARCH}/
++ /arch/${ARCH}/**.h
++ /arch/${ARCH}/**/Makefile*
++ /.config
++ /drivers/
++ /drivers/**/Kbuild*
++ /drivers/**/Kconfig*
++ /drivers/**/Makefile*
++ /include/*
++ /include/**/*.h
++ /Kbuild
++ /Kconfig
++ /Makefile
++ /Module.symvers
++ /scripts/Makefile.*
++ /scripts/basic
++ /scripts/basic/fixdep
++ /scripts/bin2c
++ /scripts/checkincludes.pl
++ /scripts/checkstack.pl
++ /scripts/checkversion.pl
++ /scripts/check-local-export
++ /scripts/depmod.sh
++ /scripts/extract-cert
++ /scripts/gcc-goto.sh
++ /scripts/gcc-version.sh
++ /scripts/gcc-x86_32-has-stack-protector.sh
++ /scripts/gcc-x86_64-has-stack-protector.sh
++ /scripts/genksyms
++ /scripts/genksyms/genksyms
++ /scripts/headers_install.sh
++ /scripts/kallsyms
++ /scripts/Kbuild.include
++ /scripts/kconfig
++ /scripts/kconfig/conf
++ /scripts/kernel-doc
++ /scripts/ld-version.sh
++ /scripts/Lindent
++ /scripts/makelst
++ /scripts/mksysmap
++ /scripts/mkuboot.sh
++ /scripts/mod/
++ /scripts/mod/modpost
++ /scripts/module.lds
++ /scripts/patch-kernel
++ /scripts/pahole-flags.sh
++ /scripts/recordmcount
++ /scripts/recordmcount.pl
++ /scripts/setlocalversion
++ /scripts/sign-file
++ /scripts/subarch.include
++ /scripts/unifdef
++ /scripts/ver_linux
++ /tools
++ /tools/objtool/
++ /tools/objtool/objtool
++ /.version
+
+#
+# Completly ignore the following directories
+#
+- /arch/*
+- /Documentation
+- /LICENSES
+- /block
+- /certs
+- /crypto
+- /drivers/*
+- /fs
+- /init
+- /io_uring
+- /ipc
+- /kernel
+- /lib
+- /MAINTAINERS
+- /mm
+- /modules.*
+- /net
+- /README
+- /rust
+- /samples
+- /scripts/*
+- /scripts/basic/Makefile
+- /scripts/genksyms/*
+- /scripts/kconfig/*
+- /scripts/mod/*
+- /security
+- /sound
+- /System.map
+- /tools/*
+- /tools/objtool/*
+- /usr
+- /virt
+
+#
+# Completly ignore intermediate directories
+#
+- /source
+- /tar-install*
+- /${O}
+#
+# Completly ignore the following files
+#
+
+- .*
+- *.a
+- *.c
+- *.o
+- *.S
+- *vmlinux*
+EOF
+
+#
+# Copy from object directory as well when it is not the same as the source one
+#
+if [ "${abs_objtree}" != "${abs_srctree}" ]
+then
+    rsync -a --filter="merge ${FILTER_FILE}" ${abs_objtree}/ ${tmpdir}
+fi
+rsync -a --filter="merge ${FILTER_FILE}" ${abs_srctree}/ ${tmpdir}
+rm ${FILTER_FILE}
+
+if [ "${1}" = dir-pkg-dev ]; then
+	echo "Kernel tree successfully created in $tmpdir"
+	exit 0
+fi
+
+#
+# Create the tarball
+#
+if tar --owner=root --group=root --help >/dev/null 2>&1; then
+	opts="$opts --owner=root --group=root"
+fi
+
+opts="$opts"
+
+tar cf $tarball -C $tmpdir $opts .
+
+echo "Tarball successfully created in $tarball"
+
+exit 0
-- 
2.27.0

Re: [PATCH 1/2] package: add tar development package for 3rd party modules

[Randy Dunlap]

Hi--

On 10/21/22 03:14, Federico Vaga wrote:
> Most, if not all, Linux distributions provides a Linux development
> package which purpose is to support the building of out-of-tree modules
> without providing the entire source tree.
> 
> What ends up in this development directory is a mixture of source
> files (mainly headers) and generated ones (headers, and tools produced
> by `make modules_prepare`).
> 
> This patch is an attempt to generate a tarball archive containing all
> required files to build external modules. It could be than reused by
> packagers.
> 
> Signed-off-by: Federico Vaga <federico.vaga@cern.ch>
> ---
>  Makefile                       |   2 +-
>  scripts/Makefile.package       |  13 +++
>  scripts/package/buildtar-devel | 207 +++++++++++++++++++++++++++++++++
>  3 files changed, 221 insertions(+), 1 deletion(-)
>  create mode 100644 scripts/package/buildtar-devel

Is there a patch 2/2?  I don't see it anywhere.


thanks.
-- 
~Randy

Re: [PATCH 1/2] package: add tar development package for 3rd party modules

[Federico Vaga]

On Fri, Oct 21, 2022 at 01:11:18PM -0700, Randy Dunlap wrote:
>Hi--
>
>On 10/21/22 03:14, Federico Vaga wrote:
>> Most, if not all, Linux distributions provides a Linux development
>> package which purpose is to support the building of out-of-tree modules
>> without providing the entire source tree.
>>
>> What ends up in this development directory is a mixture of source
>> files (mainly headers) and generated ones (headers, and tools produced
>> by `make modules_prepare`).
>>
>> This patch is an attempt to generate a tarball archive containing all
>> required files to build external modules. It could be than reused by
>> packagers.
>>
>> Signed-off-by: Federico Vaga <federico.vaga@cern.ch>
>> ---
>>  Makefile                       |   2 +-
>>  scripts/Makefile.package       |  13 +++
>>  scripts/package/buildtar-devel | 207 +++++++++++++++++++++++++++++++++
>>  3 files changed, 221 insertions(+), 1 deletion(-)
>>  create mode 100644 scripts/package/buildtar-devel
>
>Is there a patch 2/2?  I don't see it anywhere.

My mistake.

Yes there is a second one but I did not want to send it becuase it is about
generalizing buildtar to build 3 type of tarballs: the linux binaries to be
placed in /boot, the header files for user-space, and the development headers
and tools for out-of-tree modules (this patch).

The second one makes sense, only if this one makes sense. That's why I wrote few
lines in the RFC cover letter. I should have used the format-patch option to not
enumerate patches :)

>thanks.
>-- 
>~Randy

-- 
-------------------------------
Federico Vaga - CERN BE-CEM-EDL

Re: [PATCH 1/2] package: add tar development package for 3rd party modules

[Randy Dunlap]

On 10/21/22 14:48, Federico Vaga wrote:
> On Fri, Oct 21, 2022 at 01:11:18PM -0700, Randy Dunlap wrote:
>> Hi--
>>
>> On 10/21/22 03:14, Federico Vaga wrote:
>>> Most, if not all, Linux distributions provides a Linux development
>>> package which purpose is to support the building of out-of-tree modules
>>> without providing the entire source tree.
>>>
>>> What ends up in this development directory is a mixture of source
>>> files (mainly headers) and generated ones (headers, and tools produced
>>> by `make modules_prepare`).
>>>
>>> This patch is an attempt to generate a tarball archive containing all
>>> required files to build external modules. It could be than reused by
>>> packagers.
>>>
>>> Signed-off-by: Federico Vaga <federico.vaga@cern.ch>
>>> ---
>>>  Makefile                       |   2 +-
>>>  scripts/Makefile.package       |  13 +++
>>>  scripts/package/buildtar-devel | 207 +++++++++++++++++++++++++++++++++
>>>  3 files changed, 221 insertions(+), 1 deletion(-)
>>>  create mode 100644 scripts/package/buildtar-devel
>>
>> Is there a patch 2/2?  I don't see it anywhere.
> 
> My mistake.
> 
> Yes there is a second one but I did not want to send it becuase it is about
> generalizing buildtar to build 3 type of tarballs: the linux binaries to be
> placed in /boot, the header files for user-space, and the development headers
> and tools for out-of-tree modules (this patch).

Ah, I was wondering why this change was included:

diff --git a/Makefile b/Makefile
index cfbe6a7de640..36a58394ce16 100644
--- a/Makefile
+++ b/Makefile
@@ -1578,7 +1578,7 @@ CLEAN_FILES += include/ksym vmlinux.symvers modules-only.symvers \
 # Directories & files removed with 'make mrproper'
 MRPROPER_FILES += include/config include/generated          \
 		  arch/$(SRCARCH)/include/generated .objdiff \
-		  debian snap tar-install \
+		  debian snap tar-install* \
 		  .config .config.old .version \
 		  Module.symvers \
 		  certs/signing_key.pem \

so I think that you just explained it.

> The second one makes sense, only if this one makes sense. That's why I wrote few
> lines in the RFC cover letter. I should have used the format-patch option to not
> enumerate patches :)
> 
>> thanks.
>> -- 
>> ~Randy
> 

-- 
~Randy