[PATCH] kheaders: making headers archive reproducible

[PATCH] kheaders: making headers archive reproducible

[Dmitry Goldin]

From: Dmitry Goldin <dgoldin+lkml@protonmail.ch>

In commit 43d8ce9d65a5 ("Provide in-kernel headers to make
extending kernel easier") a new mechanism was introduced, for kernels
>=5.2, which embeds the kernel headers in the kernel image or a module
and exposed them in procfs for use by userland tools.

The archive containing the header files has nondeterminism through the
header files metadata. This patch normalizes the metadata and utilizes
KBUILD_BUILD_TIMESTAMP if provided and otherwise falls back to the
default behaviour.

In commit f7b101d33046 ("kheaders: Move from proc to sysfs") it was
modified to use sysfs and the script for generation of the archive was
renamed to what is being patched.

Signed-off-by: Dmitry Goldin <dgoldin+lkml@protonmail.ch>
---
 kernel/gen_kheaders.sh | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
index 9ff449888d9c..2e154741e3b2 100755
--- a/kernel/gen_kheaders.sh
+++ b/kernel/gen_kheaders.sh
@@ -71,7 +71,10 @@ done | cpio --quiet -pd $cpio_dir >/dev/null 2>&1
 find $cpio_dir -type f -print0 |
 	xargs -0 -P8 -n1 perl -pi -e 'BEGIN {undef $/;}; s/\/\*((?!SPDX).)*?\*\///smg;'

-tar -Jcf $tarfile -C $cpio_dir/ . > /dev/null
+# Create archive and try to normalized metadata for reproducibility
+tar "${KBUILD_BUILD_TIMESTAMP:+--mtime=$KBUILD_BUILD_TIMESTAMP}" \
+    --owner=0 --group=0 --sort=name --numeric-owner \
+    -Jcf $tarfile -C $cpio_dir/ . > /dev/null

 echo "$src_files_md5" >  kernel/kheaders.md5
 echo "$obj_files_md5" >> kernel/kheaders.md5
--
2.19.2

Re: [PATCH] kheaders: making headers archive reproducible

[Masahiro Yamada]

Hi Dmitry,


(+CC Ben Hutchings, who might be interested)


On Sun, Sep 22, 2019 at 10:38 PM Dmitry Goldin <dgoldin@protonmail.ch> wrote:
>
> From: Dmitry Goldin <dgoldin+lkml@protonmail.ch>
>
> In commit 43d8ce9d65a5 ("Provide in-kernel headers to make
> extending kernel easier") a new mechanism was introduced, for kernels
> >=5.2, which embeds the kernel headers in the kernel image or a module
> and exposed them in procfs for use by userland tools.
>
> The archive containing the header files has nondeterminism through the
> header files metadata. This patch normalizes the metadata and utilizes
> KBUILD_BUILD_TIMESTAMP if provided and otherwise falls back to the
> default behaviour.
>
> In commit f7b101d33046 ("kheaders: Move from proc to sysfs") it was
> modified to use sysfs and the script for generation of the archive was
> renamed to what is being patched.
>
> Signed-off-by: Dmitry Goldin <dgoldin+lkml@protonmail.ch>
> ---
>  kernel/gen_kheaders.sh | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)


Thanks, this produced the deterministic archive for me.


While you are here, could you also update the following hunk
in Documentation/kbuild/reproducible-builds.rst

---------->8---------------
The kernel embeds a timestamp in two places:

* The version string exposed by ``uname()`` and included in
  ``/proc/version``

* File timestamps in the embedded initramfs
---------->8---------------


With the documentation updated, I will pick it soon.

Thank you.




> diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
> index 9ff449888d9c..2e154741e3b2 100755
> --- a/kernel/gen_kheaders.sh
> +++ b/kernel/gen_kheaders.sh
> @@ -71,7 +71,10 @@ done | cpio --quiet -pd $cpio_dir >/dev/null 2>&1
>  find $cpio_dir -type f -print0 |
>         xargs -0 -P8 -n1 perl -pi -e 'BEGIN {undef $/;}; s/\/\*((?!SPDX).)*?\*\///smg;'
>
> -tar -Jcf $tarfile -C $cpio_dir/ . > /dev/null
> +# Create archive and try to normalized metadata for reproducibility
> +tar "${KBUILD_BUILD_TIMESTAMP:+--mtime=$KBUILD_BUILD_TIMESTAMP}" \
> +    --owner=0 --group=0 --sort=name --numeric-owner \
> +    -Jcf $tarfile -C $cpio_dir/ . > /dev/null
>
>  echo "$src_files_md5" >  kernel/kheaders.md5
>  echo "$obj_files_md5" >> kernel/kheaders.md5
> --
> 2.19.2
>
>
>


-- 
Best Regards
Masahiro Yamada

Re: [PATCH] kheaders: making headers archive reproducible

[Joel Fernandes]

On Thu, Oct 03, 2019 at 10:50:06AM +0900, Masahiro Yamada wrote:
> Hi Dmitry,
> 
> 
> (+CC Ben Hutchings, who might be interested)
> 
> 
> On Sun, Sep 22, 2019 at 10:38 PM Dmitry Goldin <dgoldin@protonmail.ch> wrote:
> >
> > From: Dmitry Goldin <dgoldin+lkml@protonmail.ch>
> >
> > In commit 43d8ce9d65a5 ("Provide in-kernel headers to make
> > extending kernel easier") a new mechanism was introduced, for kernels
> > >=5.2, which embeds the kernel headers in the kernel image or a module
> > and exposed them in procfs for use by userland tools.
> >
> > The archive containing the header files has nondeterminism through the
> > header files metadata. This patch normalizes the metadata and utilizes
> > KBUILD_BUILD_TIMESTAMP if provided and otherwise falls back to the
> > default behaviour.
> >
> > In commit f7b101d33046 ("kheaders: Move from proc to sysfs") it was
> > modified to use sysfs and the script for generation of the archive was
> > renamed to what is being patched.
> >
> > Signed-off-by: Dmitry Goldin <dgoldin+lkml@protonmail.ch>
> > ---
> >  kernel/gen_kheaders.sh | 5 ++++-
> >  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> 
> Thanks, this produced the deterministic archive for me.
> 

Reviewed-by: Joel Fernandes (Google) <joel@joelfernandes.org>

thanks,

 - Joel


> 
> While you are here, could you also update the following hunk
> in Documentation/kbuild/reproducible-builds.rst
> 
> ---------->8---------------
> The kernel embeds a timestamp in two places:
> 
> * The version string exposed by ``uname()`` and included in
>   ``/proc/version``
> 
> * File timestamps in the embedded initramfs
> ---------->8---------------
> 
> 
> With the documentation updated, I will pick it soon.
> 
> Thank you.
> 
> 
> 
> 
> > diff --git a/kernel/gen_kheaders.sh b/kernel/gen_kheaders.sh
> > index 9ff449888d9c..2e154741e3b2 100755
> > --- a/kernel/gen_kheaders.sh
> > +++ b/kernel/gen_kheaders.sh
> > @@ -71,7 +71,10 @@ done | cpio --quiet -pd $cpio_dir >/dev/null 2>&1
> >  find $cpio_dir -type f -print0 |
> >         xargs -0 -P8 -n1 perl -pi -e 'BEGIN {undef $/;}; s/\/\*((?!SPDX).)*?\*\///smg;'
> >
> > -tar -Jcf $tarfile -C $cpio_dir/ . > /dev/null
> > +# Create archive and try to normalized metadata for reproducibility
> > +tar "${KBUILD_BUILD_TIMESTAMP:+--mtime=$KBUILD_BUILD_TIMESTAMP}" \
> > +    --owner=0 --group=0 --sort=name --numeric-owner \
> > +    -Jcf $tarfile -C $cpio_dir/ . > /dev/null
> >
> >  echo "$src_files_md5" >  kernel/kheaders.md5
> >  echo "$obj_files_md5" >> kernel/kheaders.md5
> > --
> > 2.19.2
> >
> >
> >
> 
> 
> -- 
> Best Regards
> Masahiro Yamada