[PATCH v2 1/3] scripts/package/Makefile: put proper config in source tarball

[PATCH v2 1/3] scripts/package/Makefile: put proper config in source tarball

[Arseny Maslennikov]

It is widely known that one can build a kernel without a .config in the
source tree by setting KCONFIG_CONFIG equal to the actual configuration
file path.

When making a *-pkg target, make(1) prepares a source tarball and tries
to pack `.config' in there regardless of the value of KCONFIG_CONFIG,
failing spectacularly if .config is absent and packing the wrong config
if it exists.
Let's fix that by including whatever KCONFIG_CONFIG points to in the
archive under the traditional name `.config'.

We have to pass --absolute-names to tar, since it seems to trim the
paths of archive members first and only then consider --transform, which
breaks KCONFIG_CONFIG=../../some.config, as well as absolute paths.

Signed-off-by: Arseny Maslennikov <ar@cs.msu.ru>
---
 scripts/package/Makefile | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/scripts/package/Makefile b/scripts/package/Makefile
index 2c6de21e5152..134cefee068d 100644
--- a/scripts/package/Makefile
+++ b/scripts/package/Makefile
@@ -27,7 +27,7 @@ KDEB_SOURCENAME ?= linux-$(KERNELRELEASE)
 KBUILD_PKG_ROOTCMD ?="fakeroot -u"
 export KDEB_SOURCENAME
 # Include only those top-level files that are needed by make, plus the GPL copy
-TAR_CONTENT := $(KBUILD_ALLDIRS) .config .scmversion Makefile \
+TAR_CONTENT := $(KBUILD_ALLDIRS) $(KCONFIG_CONFIG) .scmversion Makefile \
                Kbuild Kconfig COPYING $(wildcard localversion*)
 MKSPEC     := $(srctree)/scripts/package/mkspec
 
@@ -44,6 +44,7 @@ if test "$(objtree)" != "$(srctree)"; then \
 fi ; \
 $(srctree)/scripts/setlocalversion --save-scmversion; \
 tar -cz $(RCS_TAR_IGNORE) -f $(2).tar.gz \
+	--absolute-names --transform 's:^$(KCONFIG_CONFIG)$$:.config:S' \
 	--transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3); \
 rm -f $(objtree)/.scmversion
 
-- 
2.20.1

[PATCH v2 2/3] scripts/package/mkdebian: avoid implicit effects

[Arseny Maslennikov]

* The man page for dpkg-source(1) notes:

>      -b, --build directory [format-specific-parameters]
>             Build  a  source  package  (--build since dpkg 1.17.14).
>             <...>
>
>             dpkg-source will build the source package with the first
>             format found in this ordered list: the format  indicated
>             with  the  --format  command  line  option,  the  format
>             indicated in debian/source/format, “1.0”.  The  fallback
>             to “1.0” is deprecated and will be removed at some point
>             in the future, you should always  document  the  desired
>             source   format  in  debian/source/format.  See  section
>             SOURCE PACKAGE FORMATS for an extensive  description  of
>             the various source package formats.

  Thus it would be more foolproof to explicitly use 1.0 (as we always
  did) than to rely on dpkg-source's defaults.

* In a similar vein, debian/rules is not made executable by mkdebian,
  and dpkg-source warns about that but still silently fixes the file.
  Let's be explicit once again.

Signed-off-by: Arseny Maslennikov <ar@cs.msu.ru>
---
 scripts/package/mkdebian | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian
index f030961c5165..d276eb671a27 100755
--- a/scripts/package/mkdebian
+++ b/scripts/package/mkdebian
@@ -132,7 +132,9 @@ else
         echo >&2 "Install lsb-release or set \$KDEB_CHANGELOG_DIST explicitly"
 fi
 
-mkdir -p debian/
+mkdir -p debian/source/
+echo "1.0" > debian/source/format
+
 echo $debarch > debian/arch
 
 # Generate a simple changelog template
@@ -221,5 +223,6 @@ clean:
 
 binary: binary-arch
 EOF
+chmod +x debian/rules
 
 exit 0
-- 
2.20.1

[PATCH v2 3/3] package: {mkdebian,Makefile}: ignore .config in dpkg-source diffs

[Arseny Maslennikov]

* We'd like to make dpkg-source ignore certain filepath patterns while
  producing the .diff.gz as part of a Debian source package.
  The relationship between dpkg-source's -i and --extend-diff-ignore
  options is quite complicated[1]. It's thus not very convenient to
  specify file path patterns to ignore on dpkg-buildpackage's command
  line (they do not end up in the source package as well), so we turn
  `-i.git' into a record in debian/source/options.
  Besides, the dot in `-i.git' is a metacharacter and matches a tad
  too much.

* In cases where a .config exists in the source tree, but the .config
  in the source tarball is provided by wherever KCONFIG_CONFIG pointed
  to, dpkg-source would attempt to replace the second, proper one with
  the first one, which is undesirable and nullifies the effort made
  to put the proper config in the source tarball.
  To prevent this, force dpkg-source to ignore /^[.]config$/ as well.

This makes it possible to build working Debian source packages with a
redefined KCONFIG_CONFIG:
 - that file is put in the source tarball at its traditional place,
   and the diff.gz does not dare touch it;
 - when the source package is rebuilt, debian/rules initiates the build
   process with the desired config already in the standard location.

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=625789#22

Signed-off-by: Arseny Maslennikov <ar@cs.msu.ru>
---
 scripts/package/Makefile | 2 +-
 scripts/package/mkdebian | 6 ++++++
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/scripts/package/Makefile b/scripts/package/Makefile
index 134cefee068d..f2fca7e7d9fa 100644
--- a/scripts/package/Makefile
+++ b/scripts/package/Makefile
@@ -73,7 +73,7 @@ deb-pkg: FORCE
 	$(call cmd,src_tar,$(KDEB_SOURCENAME))
 	origversion=$$(dpkg-parsechangelog -SVersion |sed 's/-[^-]*$$//');\
 		mv $(KDEB_SOURCENAME).tar.gz ../$(KDEB_SOURCENAME)_$${origversion}.orig.tar.gz
-	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) -i.git -us -uc
+	+dpkg-buildpackage -r$(KBUILD_PKG_ROOTCMD) -a$$(cat debian/arch) $(DPKG_FLAGS) -us -uc
 
 bindeb-pkg: FORCE
 	$(CONFIG_SHELL) $(srctree)/scripts/package/mkdebian
diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian
index d276eb671a27..74158c7f28ea 100755
--- a/scripts/package/mkdebian
+++ b/scripts/package/mkdebian
@@ -135,6 +135,12 @@ fi
 mkdir -p debian/source/
 echo "1.0" > debian/source/format
 
+cat <<EOF > debian/source/options
+# The regexes are matched against paths from the source tree.
+diff-ignore = "[.]git"
+extend-diff-ignore = "^[.]config$"
+EOF
+
 echo $debarch > debian/arch
 
 # Generate a simple changelog template
-- 
2.20.1

Re: [PATCH v2 1/3] scripts/package/Makefile: put proper config in source tarball

[Arseny Maslennikov]

[-- Attachment #1: Type: text/plain, Size: 2197 bytes --]

On Sat, Mar 09, 2019 at 06:43:05PM +0300, Arseny Maslennikov wrote:
> It is widely known that one can build a kernel without a .config in the
> source tree by setting KCONFIG_CONFIG equal to the actual configuration
> file path.
> 
> When making a *-pkg target, make(1) prepares a source tarball and tries
> to pack `.config' in there regardless of the value of KCONFIG_CONFIG,
> failing spectacularly if .config is absent and packing the wrong config
> if it exists.
> Let's fix that by including whatever KCONFIG_CONFIG points to in the
> archive under the traditional name `.config'.
> 
> We have to pass --absolute-names to tar, since it seems to trim the
> paths of archive members first and only then consider --transform, which
> breaks KCONFIG_CONFIG=../../some.config, as well as absolute paths.

One way to avoid --absolute-names here is to do the dance below:

> 
> Signed-off-by: Arseny Maslennikov <ar@cs.msu.ru>
> ---
>  scripts/package/Makefile | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/package/Makefile b/scripts/package/Makefile
> index 2c6de21e5152..134cefee068d 100644
> --- a/scripts/package/Makefile
> +++ b/scripts/package/Makefile
> @@ -27,7 +27,7 @@ KDEB_SOURCENAME ?= linux-$(KERNELRELEASE)
>  KBUILD_PKG_ROOTCMD ?="fakeroot -u"
>  export KDEB_SOURCENAME
>  # Include only those top-level files that are needed by make, plus the GPL copy
> -TAR_CONTENT := $(KBUILD_ALLDIRS) .config .scmversion Makefile \
> +TAR_CONTENT := $(KBUILD_ALLDIRS) $(KCONFIG_CONFIG) .scmversion Makefile \
>                 Kbuild Kconfig COPYING $(wildcard localversion*)
>  MKSPEC     := $(srctree)/scripts/package/mkspec
>  
> @@ -44,6 +44,7 @@ if test "$(objtree)" != "$(srctree)"; then \
>  fi ; \
>  $(srctree)/scripts/setlocalversion --save-scmversion; \
  +cp $(KCONFIG_CONFIG) .tmp.config-tarball
>  tar -cz $(RCS_TAR_IGNORE) -f $(2).tar.gz \
  +	--absolute-names --transform 's:^.tmp.config-tarball$$:.config:S' \
>  	--transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3); \
  +rm -f .tmp.config-tarball
>  rm -f $(objtree)/.scmversion
>  
> -- 
> 2.20.1
> 

I'm not sure if this is strictly better.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v2 1/3] scripts/package/Makefile: put proper config in source tarball

[Masahiro Yamada]

On Sun, Mar 10, 2019 at 12:44 AM Arseny Maslennikov <ar@cs.msu.ru> wrote:
>
> It is widely known that one can build a kernel without a .config in the
> source tree by setting KCONFIG_CONFIG equal to the actual configuration
> file path.
>
> When making a *-pkg target, make(1) prepares a source tarball and tries
> to pack `.config' in there regardless of the value of KCONFIG_CONFIG,
> failing spectacularly if .config is absent and packing the wrong config
> if it exists.
> Let's fix that by including whatever KCONFIG_CONFIG points to in the
> archive under the traditional name `.config'.
>
> We have to pass --absolute-names to tar, since it seems to trim the
> paths of archive members first and only then consider --transform, which
> breaks KCONFIG_CONFIG=../../some.config, as well as absolute paths.
>
> Signed-off-by: Arseny Maslennikov <ar@cs.msu.ru>


rpm-pkg will work in a strange way.
(probably snap-pkg too).

rpm package is compiled in $(HOME)/rpmbuild/BUILD.
The source package will contain the .config in the standard location,
but KCONFIG_CONFIG will point to a different path.

Rather, I'd like to stop the build.


quiet_cmd_src_tar = TAR     $(2).tar.gz
      cmd_src_tar = \
if test "$(objtree)" != "$(srctree)"; then \
        echo >&2; \
        echo >&2 "  ERROR:"; \
        echo >&2 "  Building source tarball is not possible outside the"; \
        echo >&2 "  kernel source tree. Don't set KBUILD_OUTPUT, or use the"; \
        echo >&2 "  binrpm-pkg or bindeb-pkg target instead."; \
        echo >&2; \
        false; \
fi ; \
if test "$(KCONFIG_CONFIG)" != .config; then \
        echo >&2; \
        echo >&2 "  ERROR:"; \
        echo >&2 "  Please do not override KCONFIG_CONFIG"; \
        echo >&2 "  for source package build"; \
        echo >&2; \
        false; \
fi ; \
$(srctree)/scripts/setlocalversion --save-scmversion; \
tar -cz $(RCS_TAR_IGNORE) -f $(2).tar.gz \
--transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3); \
rm -f $(objtree)/.scmversion



I am included to do the opposite;
replace KCONFIG_CONFIG with the hard-coded .config

KCONFIG_CONFIG does not work.
It is half-baked at best.



> ---
>  scripts/package/Makefile | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/package/Makefile b/scripts/package/Makefile
> index 2c6de21e5152..134cefee068d 100644
> --- a/scripts/package/Makefile
> +++ b/scripts/package/Makefile
> @@ -27,7 +27,7 @@ KDEB_SOURCENAME ?= linux-$(KERNELRELEASE)
>  KBUILD_PKG_ROOTCMD ?="fakeroot -u"
>  export KDEB_SOURCENAME
>  # Include only those top-level files that are needed by make, plus the GPL copy
> -TAR_CONTENT := $(KBUILD_ALLDIRS) .config .scmversion Makefile \
> +TAR_CONTENT := $(KBUILD_ALLDIRS) $(KCONFIG_CONFIG) .scmversion Makefile \
>                 Kbuild Kconfig COPYING $(wildcard localversion*)
>  MKSPEC     := $(srctree)/scripts/package/mkspec
>
> @@ -44,6 +44,7 @@ if test "$(objtree)" != "$(srctree)"; then \
>  fi ; \
>  $(srctree)/scripts/setlocalversion --save-scmversion; \
>  tar -cz $(RCS_TAR_IGNORE) -f $(2).tar.gz \
> +       --absolute-names --transform 's:^$(KCONFIG_CONFIG)$$:.config:S' \
>         --transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3); \
>  rm -f $(objtree)/.scmversion
>
> --
> 2.20.1
>


-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2 2/3] scripts/package/mkdebian: avoid implicit effects

[Masahiro Yamada]

On Sun, Mar 10, 2019 at 12:44 AM Arseny Maslennikov <ar@cs.msu.ru> wrote:
>
> * The man page for dpkg-source(1) notes:
>
> >      -b, --build directory [format-specific-parameters]
> >             Build  a  source  package  (--build since dpkg 1.17.14).
> >             <...>
> >
> >             dpkg-source will build the source package with the first
> >             format found in this ordered list: the format  indicated
> >             with  the  --format  command  line  option,  the  format
> >             indicated in debian/source/format, “1.0”.  The  fallback
> >             to “1.0” is deprecated and will be removed at some point
> >             in the future, you should always  document  the  desired
> >             source   format  in  debian/source/format.  See  section
> >             SOURCE PACKAGE FORMATS for an extensive  description  of
> >             the various source package formats.
>
>   Thus it would be more foolproof to explicitly use 1.0 (as we always
>   did) than to rely on dpkg-source's defaults.
>
> * In a similar vein, debian/rules is not made executable by mkdebian,
>   and dpkg-source warns about that but still silently fixes the file.
>   Let's be explicit once again.
>
> Signed-off-by: Arseny Maslennikov <ar@cs.msu.ru>
> ---


Applied to linux-kbuild. Thanks.


>  scripts/package/mkdebian | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/package/mkdebian b/scripts/package/mkdebian
> index f030961c5165..d276eb671a27 100755
> --- a/scripts/package/mkdebian
> +++ b/scripts/package/mkdebian
> @@ -132,7 +132,9 @@ else
>          echo >&2 "Install lsb-release or set \$KDEB_CHANGELOG_DIST explicitly"
>  fi
>
> -mkdir -p debian/
> +mkdir -p debian/source/
> +echo "1.0" > debian/source/format
> +
>  echo $debarch > debian/arch
>
>  # Generate a simple changelog template
> @@ -221,5 +223,6 @@ clean:
>
>  binary: binary-arch
>  EOF
> +chmod +x debian/rules
>
>  exit 0
> --
> 2.20.1
>


-- 
Best Regards
Masahiro Yamada