Debian build polishing

Debian build polishing

[Enrico Weigelt, metux IT consult]

Hi folks,


here're some patches for polishing up the Debian packaging stuff,
so it can be directly used w/ usual Debian machinery like
pbuilder, git-buildpackage, dck-buildpackage, etc.

These expect debian/rules to exist in the unpacked/patched tree
and drive the whole build. Currently 'make deb-pkg' does it in
the opposite direction - it creates debian/rules and fills in
some data, that's derived from .config etc.

My goal is building the kernel package in exactly the same way as
any other Debian package - so there must be a debian/rules as the
primary entry point. To do that, w/ minimal change and w/o breaking
the existing machinery, I'm going in several steps:

#1: add Makefile rules for retrieving missing makefile-internal
    variables kernel config system .config (eg. kernel arch).

    this could be used for other build systems, too.
    just call: `make kernelarch` or `make kernellocalversion`

#2: add an env variable for changing the name of the rules file
    generated by mkdebian. When coming from an existing rules
    file, we can prevent this from being overwritten.

#3: add a generic debian/rules file, that calls mkdebian to
    create the remaining debian control files (w/ rules redirected
    into nirvana)

The existing `make deb-pkg` is bypassed and remains ontouched.

One point still puzzling me: once the debian/rules is applied and
somebody calls `make deb-pkg`, he'll end up w/ unclean tree, as
now a git-tracked file is changed.

Perhaps I just change deb-pkg to call debian/rules then, but I'd
like to hear your oppinions about this, before.


What do you think about that ?


--mtx

[PATCH v1 1/5] Makefile: rules for printing kernel architecture and localversion

[Enrico Weigelt, metux IT consult]

trivial rule to print out the kernel arch and localversion, so
external tools, like distro packagers, can easily get it.

Signed-off-by: Enrico Weigelt, metux IT consult <info@metux.net>
---
 Makefile | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/Makefile b/Makefile
index f070e0d..95b50f8 100644
--- a/Makefile
+++ b/Makefile
@@ -1664,6 +1664,12 @@ kernelrelease:
 kernelversion:
 	@echo $(KERNELVERSION)
 
+kernellocalversion:
+	@$(CONFIG_SHELL) $(srctree)/scripts/setlocalversion $(srctree) | sed -e 's~^\-~~'
+
+kernelarch:
+	@echo $(ARCH)
+
 image_name:
 	@echo $(KBUILD_IMAGE)
 
-- 
1.9.1

[PATCH v1 2/5] scripts: mkdebian: allow renaming generated debian/rules via env

[Enrico Weigelt, metux IT consult]

Add new environment variable KDEB_RULES for controlling where the
generated debian rules are written to. By defaults, it's debian/rules,
but packagers might override it for providing their own rules file.

Signed-off-by: Enrico Weigelt, metux IT consult <info@metux.net>
---
 scripts/package/mkdebian | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian
index edcad61..ff226be 100755
--- a/scripts/package/mkdebian
+++ b/scripts/package/mkdebian
@@ -99,6 +99,9 @@ kernel_headers_packagename=linux-headers-$version
 dbg_packagename=$packagename-dbg
 debarch=
 set_debarch
+if [ -z "$KDEB_RULES" ]; then
+    KDEB_RULES=debian/rules
+fi
 
 if [ "$ARCH" = "um" ] ; then
 	packagename=user-mode-linux-$version
@@ -202,7 +205,7 @@ Description: Linux kernel debugging symbols for $version
  all the necessary debug symbols for the kernel and its modules.
 EOF
 
-cat <<EOF > debian/rules
+cat <<EOF > $KDEB_RULES
 #!$(command -v $MAKE) -f
 
 build:
-- 
1.9.1

[PATCH v1 3/5] scripts: mkdebian: fix missing dependencies

[Enrico Weigelt, metux IT consult]

We missed some build dependencies in the generated debian/control file.

Signed-off-by: Enrico Weigelt, metux IT consult <info@metux.net>
---
 scripts/package/mkdebian | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian
index ff226be..1caaf0b 100755
--- a/scripts/package/mkdebian
+++ b/scripts/package/mkdebian
@@ -173,7 +173,7 @@ Source: $sourcename
 Section: kernel
 Priority: optional
 Maintainer: $maintainer
-Build-Depends: bc, kmod, cpio
+Build-Depends: bc, kmod, cpio, bison, libelf-dev, libssl-dev, flex
 Homepage: http://www.kernel.org/
 
 Package: $packagename
-- 
1.9.1

[PATCH v1 4/5] scripts: checkpatch.pl: don't complain that debian/rules is executable

[Enrico Weigelt, metux IT consult]

checkpatch.pl complains when adding executable "debian/rules",
obviously a false alarm. Therefore add an exception for that.

Signed-off-by: Enrico Weigelt, metux IT consult <info@metux.net>
---
 scripts/checkpatch.pl | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/checkpatch.pl b/scripts/checkpatch.pl
index b737ca9..7bcc70a 100755
--- a/scripts/checkpatch.pl
+++ b/scripts/checkpatch.pl
@@ -2576,6 +2576,7 @@ sub process {
 		if ($line =~ /^new (file )?mode.*[7531]\d{0,2}$/) {
 			my $permhere = $here . "FILE: $realfile\n";
 			if ($realfile !~ m@scripts/@ &&
+			    $realfile !~ "debian/rules" &&
 			    $realfile !~ /\.(py|pl|awk|sh)$/) {
 				ERROR("EXECUTE_PERMISSIONS",
 				      "do not set execute permissions for source files\n" . $permhere);
-- 
1.9.1

Re: [PATCH v1 4/5] scripts: checkpatch.pl: don't complain that debian/rules is executable

[Joe Perches]

On Fri, 2019-03-08 at 13:44 +0100, Enrico Weigelt, metux IT consult wrote:
> checkpatch.pl complains when adding executable "debian/rules",
> obviously a false alarm. Therefore add an exception for that.

This is predicated on a debian/ directory actually being added
to the generic kernel tree.

I'm not sure that's a good idea.

As debian/ does not currently exist in the generic kernel source,
I think this can be kept locally in their tree if they choose.

> diff --git a/scripts/checkpatch.pl b/scripts/checkpatch.pl
[]
> @@ -2576,6 +2576,7 @@ sub process {
>  		if ($line =~ /^new (file )?mode.*[7531]\d{0,2}$/) {
>  			my $permhere = $here . "FILE: $realfile\n";
>  			if ($realfile !~ m@scripts/@ &&
> +			    $realfile !~ "debian/rules" &&
>  			    $realfile !~ /\.(py|pl|awk|sh)$/) {
>  				ERROR("EXECUTE_PERMISSIONS",
>  				      "do not set execute permissions for source files\n" . $permhere);

[PATCH v1 5/5] debian: add generic rule file

[Enrico Weigelt, metux IT consult]

Adding a generic debian rule file, so we can build the directly
via usual Debian package build tools (eg. git-buildpackage,
dck-buildpackage, etc). It expects the .config file already
placed in the source tree.

The rule file contains a rule for creating debian/control and
other metadata - this is done similar to the 'deb-pkg' make rule,
scripts/packaging/mkdebian.

Signed-off-by: Enrico Weigelt, metux IT consult <info@metux.net>
---
 .gitignore   |  1 +
 MAINTAINERS  |  6 ++++++
 debian/rules | 27 +++++++++++++++++++++++++++
 3 files changed, 34 insertions(+)
 create mode 100755 debian/rules

diff --git a/.gitignore b/.gitignore
index a20ac26..2d8081f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -68,6 +68,7 @@ modules.builtin
 # Debian directory (make deb-pkg)
 #
 /debian/
+!/debian/rules
 
 #
 # Snap directory (make snap-pkg)
diff --git a/MAINTAINERS b/MAINTAINERS
index 09d47cd..1edc169 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4363,6 +4363,12 @@ F:	include/uapi/linux/dccp.h
 F:	include/linux/tfrc.h
 F:	net/dccp/
 
+DEBIAN PACKAGING FILES
+M:	Enrico Weigelt <info@metux.net>
+L:	linux-kbuild@vger.kernel.org
+S:	Maintained
+F:	debian/
+
 DECnet NETWORK LAYER
 W:	http://linux-decnet.sourceforge.net
 L:	linux-decnet-user@lists.sourceforge.net
diff --git a/debian/rules b/debian/rules
new file mode 100755
index 0000000..c2f0319
--- /dev/null
+++ b/debian/rules
@@ -0,0 +1,27 @@
+#!/usr/bin/make -f
+# SPDX-License-Identifier: GPL-2.0
+
+export MAKE
+export KERNELARCH           = $(shell $(MAKE) kernelarch)
+export KERNELRELEASE        = $(shell $(MAKE) kernelrelease)
+export KBUILD_DEBARCH       = $(shell dpkg-architecture -qDEB_HOST_ARCH)
+export KBUILD_BUILD_VERSION = $(shell $(MAKE) kernellocalversion)
+export KDEB_RULES           = debian/rules.auto
+export KDEB_SOURCENAME      = linux-source
+export ARCH                 = $(KERNELARCH)
+
+debian/control debian/changelong debian/arch debian/copyright:
+debian/control:
+	./scripts/package/mkdebian
+
+build:	debian/control
+	$(MAKE) KERNELRELEASE=$(KERNELRELEASE) ARCH=$(KERNELARCH) KBUILD_BUILD_VERSION=$(KBUILD_BUILD_VERSION) KBUILD_SRC=
+
+binary-arch:	debian/control
+	$(MAKE) KERNELRELEASE=$(KERNELRELEASE) ARCH=$(KERNELARCH) KBUILD_BUILD_VERSION=$(KBUILD_BUILD_VERSION) KBUILD_SRC= intdeb-pkg
+
+clean:
+	rm -rf debian/*tmp debian/files debian/changelog debian/control debian/copyright debian/rules.auto debian/arch
+	$(MAKE) clean
+
+binary: binary-arch
-- 
1.9.1

Re: Debian build polishing

[Theodore Ts'o]

On Fri, Mar 08, 2019 at 01:44:19PM +0100, Enrico Weigelt, metux IT consult wrote:
> One point still puzzling me: once the debian/rules is applied and
> somebody calls `make deb-pkg`, he'll end up w/ unclean tree, as
> now a git-tracked file is changed.

This is not something I've noticed, but I build my Debian packages
like this:

make O=/build/linux-build bindeb-pkg

This works really well for me, since all of the build-artificats land
in /build, and I can use a HDD (or a PD-HDD when building using Google
Compute Engine) for /build, while I use a SSD for my source tree.  I
find that using a HDD for a target of a build doesn't really slow
things down, and this allows me to save $$$ (when using a Cloud VM)
and reduce flash wearout and capital cost (on my personal machines).

So I really hope your patches don't break this.  Also, are there any
changes the performance of building the Debian packages before and
after your changes?  And are there any differences in the packages in
terms of any pre-or-post install/removal scripts?

There are a lot of things I really dislike about the "official" Debian
kernel build processes (they're optimzied for distribution release
engineers, not kernel developers), so I'm really hoping that making
things more like the "official Debian way" doesn't break some of the
things I really like about the existing "make bindeb-pkg" build
system.

Cheers,

						- Ted

Re: Debian build polishing

[Enrico Weigelt, metux IT consult]

On 08.03.19 18:57, Theodore Ts'o wrote:
> On Fri, Mar 08, 2019 at 01:44:19PM +0100, Enrico Weigelt, metux IT consult wrote:
>> One point still puzzling me: once the debian/rules is applied and
>> somebody calls `make deb-pkg`, he'll end up w/ unclean tree, as
>> now a git-tracked file is changed.
> 
> This is not something I've noticed, but I build my Debian packages
> like this:
> 
> make O=/build/linux-build bindeb-pkg

This doesn't work for my environment:

The requirement is that every debian source tree has a ./debian/rules
file (w/ the standard rules like binary, clean, ...). This is called by
upper level build machinery (git-buildpackage, dpkg-buildpackage, ...)

The idea behind is building all packages exactly the same way - one
standard workflow, that just takes a source tree (from git) and spits
out some .deb files.

> So I really hope your patches don't break this.  

It shouldn't (at least I dont intend to). Haven't tried where exactly
the generated debian/rules file lands in your case.

> Also, are there any changes the performance of building the Debian
> packages before and after your changes?  And are there any differences 
> in the packages in terms of any pre-or-post install/removal scripts?

I don't think so. In essence I just replace the generated rules file by
a fixed one. Okay, it will take *a little* bit longer, as the rules file
now calls `make kernelarch`, `make kernelrelease` and `make
kernellocalversion`, before starting actual build. But that shouldn't be
the big deal, IMHO.

> There are a lot of things I really dislike about the "official" Debian
> kernel build processes (they're optimzied for distribution release
> engineers, not kernel developers), 

I try to bring these two world a bit closer together.

> so I'm really hoping that making
> things more like the "official Debian way" doesn't break some of the
> things I really like about the existing "make bindeb-pkg" build
> system.

I intend not to.

Actually, the official Debian way for Kernel build is entirely
different, and that's really complicated and hard to really understand
(that's why I'm  doing this here).

But here I'm just coping w/ the first entry point into the build
process, which now becomes `./debian/rules binary` instead of
`make bindeb-pkg`. The final result (in the .deb files) should be
the time (unless I did something wrong).


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
info@metux.net -- +49-151-27565287

Re: Debian build polishing

[Theodore Ts'o]

On Fri, Mar 08, 2019 at 10:10:19PM +0100, Enrico Weigelt, metux IT consult wrote:
> > So I really hope your patches don't break this.  
> 
> It shouldn't (at least I dont intend to). Haven't tried where exactly
> the generated debian/rules file lands in your case.

The full debian directory lands in the O= directory.  So presumably
you could make it work for use case if there was a "make O=/build/dir
debian-prepare" which copies the necessary debian/ files into the
build directory and then someone could cd into /build/dir and run
"dpkg-buildpackage"?

						- Ted

Re: Debian build polishing

[Enrico Weigelt, metux IT consult]

On 08.03.19 22:57, Theodore Ts'o wrote:

> The full debian directory lands in the O= directory.  So presumably
> you could make it work for use case if there was a "make O=/build/dir
> debian-prepare" which copies the necessary debian/ files into the
> build directory and then someone could cd into /build/dir and run
> "dpkg-buildpackage"?

Didn't touch these rules at all, so it should still work as it used to.

I thought about changing the old deb rules to just call the new
debian/rules file, but haven't done so yet.

Maybe you could find some time to test whether my approach also
works for you ?


--mtx

-- 
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
info@metux.net -- +49-151-27565287

Re: Debian build polishing

[Masahiro Yamada]

(+CC more debian folks)


On Fri, Mar 8, 2019 at 9:45 PM Enrico Weigelt, metux IT consult
<info@metux.net> wrote:
>
> Hi folks,
>
>
> here're some patches for polishing up the Debian packaging stuff,
> so it can be directly used w/ usual Debian machinery like
> pbuilder, git-buildpackage, dck-buildpackage, etc.
>
> These expect debian/rules to exist in the unpacked/patched tree
> and drive the whole build. Currently 'make deb-pkg' does it in
> the opposite direction - it creates debian/rules and fills in
> some data, that's derived from .config etc.
>
> My goal is building the kernel package in exactly the same way as
> any other Debian package - so there must be a debian/rules as the
> primary entry point. To do that, w/ minimal change and w/o breaking
> the existing machinery, I'm going in several steps:
>
> #1: add Makefile rules for retrieving missing makefile-internal
>     variables kernel config system .config (eg. kernel arch).
>
>     this could be used for other build systems, too.
>     just call: `make kernelarch` or `make kernellocalversion`
>
> #2: add an env variable for changing the name of the rules file
>     generated by mkdebian. When coming from an existing rules
>     file, we can prevent this from being overwritten.
>
> #3: add a generic debian/rules file, that calls mkdebian to
>     create the remaining debian control files (w/ rules redirected
>     into nirvana)
>
> The existing `make deb-pkg` is bypassed and remains ontouched.
>
> One point still puzzling me: once the debian/rules is applied and
> somebody calls `make deb-pkg`, he'll end up w/ unclean tree, as
> now a git-tracked file is changed.


Then, setlocalversion will set -dirty flag.

Committing debian/rules looks questionable to me.


> Perhaps I just change deb-pkg to call debian/rules then, but I'd
> like to hear your oppinions about this, before.
>
>
> What do you think about that ?
>
>
> --mtx
>
>


-- 
Best Regards
Masahiro Yamada