[PATCH 0/7] builddeb: Improve Debian packaging

[PATCH 0/7] builddeb: Improve Debian packaging

[Guillem Jover]

Hi!

This series fixes some packaging issues and brings it closer to
current standards.

Thanks,
Guillem

Guillem Jover (7):
  builddeb: Pass -n to gzip for reproducible packages
  builddeb: Move upstream repository URL into the upstream metadata file
  builddeb: Switch to a machine readable copyright file
  builddeb: Add support for all required debian/rules targets
  builddeb: Enable rootless builds
  builddeb: Map rc versions to use an earlier than version scheme
  builddeb: Try to get the user full name from the GECOS field

 scripts/package/builddeb | 11 ++++++--
 scripts/package/mkdebian | 59 +++++++++++++++++++++++++++-------------
 2 files changed, 48 insertions(+), 22 deletions(-)

-- 
2.28.0.297.g1956fa8f8d

[PATCH 1/7] builddeb: Pass -n to gzip for reproducible packages

[Guillem Jover]

We should not be encoding the timestamp, otherwise we end up generating
unreproducible files that cascade into unreproducible packages.

Signed-off-by: Guillem Jover <guillem@hadrons.org>
---
 scripts/package/builddeb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/package/builddeb b/scripts/package/builddeb
index 6df3c9f8b2da..b14abbe5afbb 100755
--- a/scripts/package/builddeb
+++ b/scripts/package/builddeb
@@ -31,7 +31,7 @@ create_package() {
 	mkdir -p "$pdir/usr/share/doc/$pname"
 	cp debian/copyright "$pdir/usr/share/doc/$pname/"
 	cp debian/changelog "$pdir/usr/share/doc/$pname/changelog.Debian"
-	gzip -9 "$pdir/usr/share/doc/$pname/changelog.Debian"
+	gzip -n -9 "$pdir/usr/share/doc/$pname/changelog.Debian"
 	sh -c "cd '$pdir'; find . -type f ! -path './DEBIAN/*' -printf '%P\0' \
 		| xargs -r0 md5sum > DEBIAN/md5sums"
 
-- 
2.28.0.297.g1956fa8f8d

Re: [PATCH 1/7] builddeb: Pass -n to gzip for reproducible packages

[Masahiro Yamada]

On Mon, Sep 21, 2020 at 7:48 AM Guillem Jover <guillem@hadrons.org> wrote:
>
> We should not be encoding the timestamp, otherwise we end up generating
> unreproducible files that cascade into unreproducible packages.
>
> Signed-off-by: Guillem Jover <guillem@hadrons.org>


Applied to linux-kbuild.
Thanks.

> ---
>  scripts/package/builddeb | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/scripts/package/builddeb b/scripts/package/builddeb
> index 6df3c9f8b2da..b14abbe5afbb 100755
> --- a/scripts/package/builddeb
> +++ b/scripts/package/builddeb
> @@ -31,7 +31,7 @@ create_package() {
>         mkdir -p "$pdir/usr/share/doc/$pname"
>         cp debian/copyright "$pdir/usr/share/doc/$pname/"
>         cp debian/changelog "$pdir/usr/share/doc/$pname/changelog.Debian"
> -       gzip -9 "$pdir/usr/share/doc/$pname/changelog.Debian"
> +       gzip -n -9 "$pdir/usr/share/doc/$pname/changelog.Debian"
>         sh -c "cd '$pdir'; find . -type f ! -path './DEBIAN/*' -printf '%P\0' \
>                 | xargs -r0 md5sum > DEBIAN/md5sums"
>
> --
> 2.28.0.297.g1956fa8f8d
>


-- 
Best Regards
Masahiro Yamada

[PATCH 2/7] builddeb: Move upstream repository URL into the upstream metadata file

[Guillem Jover]

This file is structured in machine readable form (YAML), so makes it
easier to process by other tools. Switch from a git to https URL.

Signed-off-by: Guillem Jover <guillem@hadrons.org>
---
 scripts/package/mkdebian | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian
index 48fbd3d0284a..cd2c8c39aa54 100755
--- a/scripts/package/mkdebian
+++ b/scripts/package/mkdebian
@@ -148,6 +148,13 @@ $sourcename ($packageversion) $distribution; urgency=low
  -- $maintainer  $(date -R)
 EOF
 
+mkdir -p debian/upstream/
+# Generate upstream metadata file
+cat <<EOF > debian/upstream/metadata
+---
+Repository: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
+EOF
+
 # Generate copyright file
 cat <<EOF > debian/copyright
 This is a packacked upstream version of the Linux kernel.
@@ -157,9 +164,6 @@ https://www.kernel.org/pub/linux/kernel
 
 Copyright: 1991 - 2018 Linus Torvalds and others.
 
-The git repository for mainline kernel development is at:
-git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
-
     This program is free software; you can redistribute it and/or modify
     it under the terms of the GNU General Public License as published by
     the Free Software Foundation; version 2 dated June, 1991.
-- 
2.28.0.297.g1956fa8f8d

[PATCH 3/7] builddeb: Switch to a machine readable copyright file

[Guillem Jover]

This format makes it possible to automatically process the licenses
within a system for compatibility or compliance checks, for example.
Update the copyright years.

Signed-off-by: Guillem Jover <guillem@hadrons.org>
---
 scripts/package/mkdebian | 36 +++++++++++++++++++++++-------------
 1 file changed, 23 insertions(+), 13 deletions(-)

diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian
index cd2c8c39aa54..69f2a6d7f009 100755
--- a/scripts/package/mkdebian
+++ b/scripts/package/mkdebian
@@ -157,19 +157,29 @@ EOF
 
 # Generate copyright file
 cat <<EOF > debian/copyright
-This is a packacked upstream version of the Linux kernel.
-
-The sources may be found at most Linux archive sites, including:
-https://www.kernel.org/pub/linux/kernel
-
-Copyright: 1991 - 2018 Linus Torvalds and others.
-
-    This program is free software; you can redistribute it and/or modify
-    it under the terms of the GNU General Public License as published by
-    the Free Software Foundation; version 2 dated June, 1991.
-
-On Debian GNU/Linux systems, the complete text of the GNU General Public
-License version 2 can be found in \`/usr/share/common-licenses/GPL-2'.
+Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
+Source: https://www.kernel.org/pub/linux/kernel
+
+Files: *
+Copyright:
+ 1991-2020 Linus Torvalds and others.
+License: GPL-2
+
+License: GPL-2
+ This program is free software; you can redistribute it and/or modify
+ it under the terms of the GNU General Public License as published by
+ the Free Software Foundation; version 2 dated June, 1991.
+ .
+ This program is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ GNU General Public License for more details.
+ .
+ You should have received a copy of the GNU General Public License
+ along with this program.  If not, see <https://www.gnu.org/licenses/>.
+Comment:
+ On Debian systems, the complete text of the GNU General Public License
+ version 2 can be found in '/usr/share/common-licenses/GPL-2'.
 EOF
 
 # Generate a control file
-- 
2.28.0.297.g1956fa8f8d

Re: [PATCH 3/7] builddeb: Switch to a machine readable copyright file

[Masahiro Yamada]

(+CC ben)

On Mon, Sep 21, 2020 at 7:48 AM Guillem Jover <guillem@hadrons.org> wrote:
>
> This format makes it possible to automatically process the licenses
> within a system for compatibility or compliance checks, for example.
> Update the copyright years.


Not all kernel developers are familiar with the
background info.
I think the commit description can describe a little more about
the rationale.


For example,

This was accepted as DEP-5
(https://dep-team.pages.debian.net/deps/dep5/)

and the debian kernel package switched to
the machine-readable debian/copyright by [1].

[1]: https://salsa.debian.org/kernel-team/linux/-/commit/d4f730e3ba9c76aff4c522b1a87aeb617f2f655e







> Signed-off-by: Guillem Jover <guillem@hadrons.org>
> ---
>  scripts/package/mkdebian | 36 +++++++++++++++++++++++-------------
>  1 file changed, 23 insertions(+), 13 deletions(-)
>
> diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian
> index cd2c8c39aa54..69f2a6d7f009 100755
> --- a/scripts/package/mkdebian
> +++ b/scripts/package/mkdebian
> @@ -157,19 +157,29 @@ EOF
>
>  # Generate copyright file
>  cat <<EOF > debian/copyright
> -This is a packacked upstream version of the Linux kernel.
> -
> -The sources may be found at most Linux archive sites, including:
> -https://www.kernel.org/pub/linux/kernel
> -
> -Copyright: 1991 - 2018 Linus Torvalds and others.
> -
> -    This program is free software; you can redistribute it and/or modify
> -    it under the terms of the GNU General Public License as published by
> -    the Free Software Foundation; version 2 dated June, 1991.
> -
> -On Debian GNU/Linux systems, the complete text of the GNU General Public
> -License version 2 can be found in \`/usr/share/common-licenses/GPL-2'.
> +Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
> +Source: https://www.kernel.org/pub/linux/kernel
> +
> +Files: *
> +Copyright:
> + 1991-2020 Linus Torvalds and others.
> +License: GPL-2

This specifies GPL-2 for 'Files: *'.

This is wrong because not all files in the kernel are GPL-2.



The debian-kernel maintains the large list of
exceptional files:

https://salsa.debian.org/kernel-team/linux/-/blob/debian/5.8.7-1/debian/copyright

But, I doubt it is maintainable.


I wonder if it is a good thing to convert it
to machine-interpretable, but apparently wrong information.


I am reluctant to maintain the huge debian/copyright...


Ben,
Do you have any idea how to handle this?




> +License: GPL-2
> + This program is free software; you can redistribute it and/or modify
> + it under the terms of the GNU General Public License as published by
> + the Free Software Foundation; version 2 dated June, 1991.
> + .
> + This program is distributed in the hope that it will be useful,
> + but WITHOUT ANY WARRANTY; without even the implied warranty of
> + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + GNU General Public License for more details.
> + .
> + You should have received a copy of the GNU General Public License
> + along with this program.  If not, see <https://www.gnu.org/licenses/>.
> +Comment:
> + On Debian systems, the complete text of the GNU General Public License
> + version 2 can be found in '/usr/share/common-licenses/GPL-2'.
>  EOF
>
>  # Generate a control file
> --
> 2.28.0.297.g1956fa8f8d
>


--
Best Regards
Masahiro Yamada

[PATCH 4/7] builddeb: Add support for all required debian/rules targets

[Guillem Jover]

These have been required by the Debian policy for a while, even though
the tooling can detect and workaround their omission, but are a hard
requirement when using rootless builds.

Signed-off-by: Guillem Jover <guillem@hadrons.org>
---
 scripts/package/mkdebian | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian
index 69f2a6d7f009..dae6c0ec0bd6 100755
--- a/scripts/package/mkdebian
+++ b/scripts/package/mkdebian
@@ -231,11 +231,15 @@ cat <<EOF > debian/rules
 
 srctree ?= .
 
-build:
+build-indep:
+build-arch:
 	\$(MAKE) KERNELRELEASE=${version} ARCH=${ARCH} \
 	KBUILD_BUILD_VERSION=${revision} -f \$(srctree)/Makefile
 
-binary-arch:
+build: build-arch
+
+binary-indep:
+binary-arch: build-arch
 	\$(MAKE) KERNELRELEASE=${version} ARCH=${ARCH} \
 	KBUILD_BUILD_VERSION=${revision} -f \$(srctree)/Makefile intdeb-pkg
 
-- 
2.28.0.297.g1956fa8f8d

Re: [PATCH 4/7] builddeb: Add support for all required debian/rules targets

[Masahiro Yamada]

On Mon, Sep 21, 2020 at 7:48 AM Guillem Jover <guillem@hadrons.org> wrote:
>
> These have been required by the Debian policy for a while, even though
> the tooling can detect and workaround their omission, but are a hard
> requirement when using rootless builds.
>


Applied to linux-kbuild. Thanks.


I added a little more explanation about why this patch is needed.

[masahiro:
The following Debian policy is particularly important for rootless builds:
"Both binary-* targets should depend on the build target, or on the
 appropriate build-arch or build-indep target, so that the package is
 built if it has not been already."
]






> Signed-off-by: Guillem Jover <guillem@hadrons.org>
> ---
>  scripts/package/mkdebian | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian
> index 69f2a6d7f009..dae6c0ec0bd6 100755
> --- a/scripts/package/mkdebian
> +++ b/scripts/package/mkdebian
> @@ -231,11 +231,15 @@ cat <<EOF > debian/rules
>
>  srctree ?= .
>
> -build:
> +build-indep:
> +build-arch:
>         \$(MAKE) KERNELRELEASE=${version} ARCH=${ARCH} \
>         KBUILD_BUILD_VERSION=${revision} -f \$(srctree)/Makefile
>
> -binary-arch:
> +build: build-arch
> +
> +binary-indep:
> +binary-arch: build-arch
>         \$(MAKE) KERNELRELEASE=${version} ARCH=${ARCH} \
>         KBUILD_BUILD_VERSION=${revision} -f \$(srctree)/Makefile intdeb-pkg
>
> --
> 2.28.0.297.g1956fa8f8d
>


-- 
Best Regards
Masahiro Yamada

[PATCH 5/7] builddeb: Enable rootless builds

[Guillem Jover]

This makes it possible to build the Debian packages without requiring
(pseudo-)root privileges, when the build drivers support this mode
of operation.

See-Also: /usr/share/doc/dpkg/rootless-builds.txt.gz
Signed-off-by: Guillem Jover <guillem@hadrons.org>
---
 scripts/package/builddeb | 9 +++++++--
 scripts/package/mkdebian | 1 +
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/scripts/package/builddeb b/scripts/package/builddeb
index b14abbe5afbb..d428c0945d4b 100755
--- a/scripts/package/builddeb
+++ b/scripts/package/builddeb
@@ -26,6 +26,7 @@ if_enabled_echo() {
 
 create_package() {
 	local pname="$1" pdir="$2"
+	local dpkg_deb_opts
 
 	mkdir -m 755 -p "$pdir/DEBIAN"
 	mkdir -p "$pdir/usr/share/doc/$pname"
@@ -36,14 +37,18 @@ create_package() {
 		| xargs -r0 md5sum > DEBIAN/md5sums"
 
 	# Fix ownership and permissions
-	chown -R root:root "$pdir"
+	if [ "$DEB_RULES_REQUIRES_ROOT" = "no" ]; then
+		dpkg_deb_opts="--root-owner-group"
+	else
+		chown -R root:root "$pdir"
+	fi
 	chmod -R go-w "$pdir"
 	# in case we are in a restrictive umask environment like 0077
 	chmod -R a+rX "$pdir"
 
 	# Create the package
 	dpkg-gencontrol -p$pname -P"$pdir"
-	dpkg-deb ${KDEB_COMPRESS:+-Z$KDEB_COMPRESS} --build "$pdir" ..
+	dpkg-deb $dpkg_deb_opts ${KDEB_COMPRESS:+-Z$KDEB_COMPRESS} --build "$pdir" ..
 }
 
 deploy_kernel_headers () {
diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian
index dae6c0ec0bd6..b98c382f967a 100755
--- a/scripts/package/mkdebian
+++ b/scripts/package/mkdebian
@@ -188,6 +188,7 @@ Source: $sourcename
 Section: kernel
 Priority: optional
 Maintainer: $maintainer
+Rules-Requires-Root: no
 Build-Depends: bc, rsync, kmod, cpio, bison, flex | flex:native $extra_build_depends
 Homepage: https://www.kernel.org/
 
-- 
2.28.0.297.g1956fa8f8d

Re: [PATCH 5/7] builddeb: Enable rootless builds

[Masahiro Yamada]

On Mon, Sep 21, 2020 at 7:48 AM Guillem Jover <guillem@hadrons.org> wrote:
>
> This makes it possible to build the Debian packages without requiring
> (pseudo-)root privileges, when the build drivers support this mode
> of operation.
>
> See-Also: /usr/share/doc/dpkg/rootless-builds.txt.gz
> Signed-off-by: Guillem Jover <guillem@hadrons.org>
> ---


Applied to linux-kbuild. Thanks.


>  scripts/package/builddeb | 9 +++++++--
>  scripts/package/mkdebian | 1 +
>  2 files changed, 8 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/package/builddeb b/scripts/package/builddeb
> index b14abbe5afbb..d428c0945d4b 100755
> --- a/scripts/package/builddeb
> +++ b/scripts/package/builddeb
> @@ -26,6 +26,7 @@ if_enabled_echo() {
>
>  create_package() {
>         local pname="$1" pdir="$2"
> +       local dpkg_deb_opts
>
>         mkdir -m 755 -p "$pdir/DEBIAN"
>         mkdir -p "$pdir/usr/share/doc/$pname"
> @@ -36,14 +37,18 @@ create_package() {
>                 | xargs -r0 md5sum > DEBIAN/md5sums"
>
>         # Fix ownership and permissions
> -       chown -R root:root "$pdir"
> +       if [ "$DEB_RULES_REQUIRES_ROOT" = "no" ]; then
> +               dpkg_deb_opts="--root-owner-group"
> +       else
> +               chown -R root:root "$pdir"
> +       fi
>         chmod -R go-w "$pdir"
>         # in case we are in a restrictive umask environment like 0077
>         chmod -R a+rX "$pdir"
>
>         # Create the package
>         dpkg-gencontrol -p$pname -P"$pdir"
> -       dpkg-deb ${KDEB_COMPRESS:+-Z$KDEB_COMPRESS} --build "$pdir" ..
> +       dpkg-deb $dpkg_deb_opts ${KDEB_COMPRESS:+-Z$KDEB_COMPRESS} --build "$pdir" ..
>  }
>
>  deploy_kernel_headers () {
> diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian
> index dae6c0ec0bd6..b98c382f967a 100755
> --- a/scripts/package/mkdebian
> +++ b/scripts/package/mkdebian
> @@ -188,6 +188,7 @@ Source: $sourcename
>  Section: kernel
>  Priority: optional
>  Maintainer: $maintainer
> +Rules-Requires-Root: no
>  Build-Depends: bc, rsync, kmod, cpio, bison, flex | flex:native $extra_build_depends
>  Homepage: https://www.kernel.org/
>
> --
> 2.28.0.297.g1956fa8f8d
>


-- 
Best Regards
Masahiro Yamada

[PATCH 6/7] builddeb: Map rc versions to use an earlier than version scheme

[Guillem Jover]

This makes it possible to build release candidates that will have a
version earlier than the final release.

Signed-off-by: Guillem Jover <guillem@hadrons.org>
---
 scripts/package/mkdebian | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian
index b98c382f967a..ed77d0948fa3 100755
--- a/scripts/package/mkdebian
+++ b/scripts/package/mkdebian
@@ -93,6 +93,7 @@ else
 	revision=$(cat .version 2>/dev/null||echo 1)
 	packageversion=$version-$revision
 fi
+packageversion=$(echo $packageversion | sed -e 's/-rc/~rc/')
 sourcename=$KDEB_SOURCENAME
 packagename=linux-image-$version
 kernel_headers_packagename=linux-headers-$version
-- 
2.28.0.297.g1956fa8f8d

[PATCH 7/7] builddeb: Try to get the user full name from the GECOS field

[Guillem Jover]

The GECOS field might contain the full name for the username, try to
use that and fallback to using the username if that is empty.

Signed-off-by: Guillem Jover <guillem@hadrons.org>
---
 scripts/package/mkdebian | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian
index ed77d0948fa3..e3d0b7ac9057 100755
--- a/scripts/package/mkdebian
+++ b/scripts/package/mkdebian
@@ -113,7 +113,8 @@ if echo $email | grep -q '<.*>'; then
 else
 	# or construct the maintainer string
 	user=${KBUILD_BUILD_USER-$(id -nu)}
-	name=${DEBFULLNAME-$user}
+	name=${DEBFULLNAME-$(getent passwd $user | cut -d: -f5 | cut -d, -f1)}
+	: "${name:=$user}"
 	if [ -z "$email" ]; then
 		buildhost=${KBUILD_BUILD_HOST-$(hostname -f 2>/dev/null || hostname)}
 		email="$user@$buildhost"
-- 
2.28.0.297.g1956fa8f8d

Re: [PATCH 0/7] builddeb: Improve Debian packaging

[Masahiro Yamada]

(+CC Ben)

On Mon, Sep 21, 2020 at 7:48 AM Guillem Jover <guillem@hadrons.org> wrote:
>
> Hi!
>
> This series fixes some packaging issues and brings it closer to
> current standards.


I am not tracking the latest development in the Debian project.

Could you give me a little more context about
what you call "current standards" ?

(It is defined in Debian Policy, or accepted as DEP-*, etc.)

For example, as for "3/7: machine-readable debian/copyright",
it is DEP-5, and can be mentioned in the commit log
for clarification.

What is the rationale for "2/7: debian/upstream/metadata" ?


Ben:
If possible, could you take a look at this series?





> Thanks,
> Guillem
>
> Guillem Jover (7):
>   builddeb: Pass -n to gzip for reproducible packages
>   builddeb: Move upstream repository URL into the upstream metadata file
>   builddeb: Switch to a machine readable copyright file
>   builddeb: Add support for all required debian/rules targets
>   builddeb: Enable rootless builds
>   builddeb: Map rc versions to use an earlier than version scheme
>   builddeb: Try to get the user full name from the GECOS field
>
>  scripts/package/builddeb | 11 ++++++--
>  scripts/package/mkdebian | 59 +++++++++++++++++++++++++++-------------
>  2 files changed, 48 insertions(+), 22 deletions(-)
>
> --
> 2.28.0.297.g1956fa8f8d
>


--
Best Regards

Masahiro Yamada