[PATCH] kernel.bbclass: improve reproducibility

[PATCH] kernel.bbclass: improve reproducibility

[Jonathan Liu]

Set KBUILD_BUILD_VERSION=1 to avoid build version being incremented on
every build. It is visible in the output of "cat /proc/version" after
the hash character.

Signed-off-by: Jonathan Liu <net147@gmail.com>
---
 meta/classes/kernel.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 4877965af0..78d6c30b07 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -151,6 +151,7 @@ PACKAGES_DYNAMIC += "^${KERNEL_PACKAGE_NAME}-firmware-.*"
 
 export OS = "${TARGET_OS}"
 export CROSS_COMPILE = "${TARGET_PREFIX}"
+export KBUILD_BUILD_VERSION = "1"
 export KBUILD_BUILD_USER = "oe-user"
 export KBUILD_BUILD_HOST = "oe-host"
 
-- 
2.16.3

Re: [PATCH] kernel.bbclass: improve reproducibility

[Bruce Ashfield]

On Tue, Apr 3, 2018 at 12:23 AM, Jonathan Liu <net147@gmail.com> wrote:
> Set KBUILD_BUILD_VERSION=1 to avoid build version being incremented on
> every build. It is visible in the output of "cat /proc/version" after
> the hash character.

It is possible that someone might actually want this feature, for debug or
different build variants.

So this would be better as a variable that can be overridden.

Bruce

>
> Signed-off-by: Jonathan Liu <net147@gmail.com>
> ---
>  meta/classes/kernel.bbclass | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
> index 4877965af0..78d6c30b07 100644
> --- a/meta/classes/kernel.bbclass
> +++ b/meta/classes/kernel.bbclass
> @@ -151,6 +151,7 @@ PACKAGES_DYNAMIC += "^${KERNEL_PACKAGE_NAME}-firmware-.*"
>
>  export OS = "${TARGET_OS}"
>  export CROSS_COMPILE = "${TARGET_PREFIX}"
> +export KBUILD_BUILD_VERSION = "1"
>  export KBUILD_BUILD_USER = "oe-user"
>  export KBUILD_BUILD_HOST = "oe-host"
>
> --
> 2.16.3
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: [PATCH] kernel.bbclass: improve reproducibility

[Richard Purdie]

On Tue, 2018-04-03 at 08:58 -0400, Bruce Ashfield wrote:
> On Tue, Apr 3, 2018 at 12:23 AM, Jonathan Liu <net147@gmail.com>
> wrote:
> > 
> > Set KBUILD_BUILD_VERSION=1 to avoid build version being incremented
> > on
> > every build. It is visible in the output of "cat /proc/version"
> > after
> > the hash character.
> It is possible that someone might actually want this feature, for
> debug or
> different build variants.
> 
> So this would be better as a variable that can be overridden.

Surely users can override it in this form (like the USER/HOST variants
already there)?

Cheers,

Richard

Re: [PATCH] kernel.bbclass: improve reproducibility

[Bruce Ashfield]

On Tue, Apr 3, 2018 at 9:59 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Tue, 2018-04-03 at 08:58 -0400, Bruce Ashfield wrote:
>> On Tue, Apr 3, 2018 at 12:23 AM, Jonathan Liu <net147@gmail.com>
>> wrote:
>> >
>> > Set KBUILD_BUILD_VERSION=1 to avoid build version being incremented
>> > on
>> > every build. It is visible in the output of "cat /proc/version"
>> > after
>> > the hash character.
>> It is possible that someone might actually want this feature, for
>> debug or
>> different build variants.
>>
>> So this would be better as a variable that can be overridden.
>
> Surely users can override it in this form (like the USER/HOST variants
> already there)?

I've never used the user/host variant, so I don't know.

How can you override an exported variable in a shell function in a
bbclass ? I've never tried.

I know for a fact, that I wouldn't want this on by default, hence my comment.

Bruce

>
> Cheers,
>
> Richard



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: [PATCH] kernel.bbclass: improve reproducibility

[Richard Purdie]

On Tue, 2018-04-03 at 10:11 -0400, Bruce Ashfield wrote:
> On Tue, Apr 3, 2018 at 9:59 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > 
> > On Tue, 2018-04-03 at 08:58 -0400, Bruce Ashfield wrote:
> > > 
> > > On Tue, Apr 3, 2018 at 12:23 AM, Jonathan Liu <net147@gmail.com>
> > > wrote:
> > > > 
> > > > 
> > > > Set KBUILD_BUILD_VERSION=1 to avoid build version being
> > > > incremented
> > > > on
> > > > every build. It is visible in the output of "cat /proc/version"
> > > > after
> > > > the hash character.
> > > It is possible that someone might actually want this feature, for
> > > debug or
> > > different build variants.
> > > 
> > > So this would be better as a variable that can be overridden.
> > Surely users can override it in this form (like the USER/HOST
> > variants
> > already there)?
> I've never used the user/host variant, so I don't know.
> 
> How can you override an exported variable in a shell function in a
> bbclass ? I've never tried.
> 
> I know for a fact, that I wouldn't want this on by default, hence my
> comment.

KBUILD_BUILD_VERSION[unexport] = "1"

will override the export flag...

Cheers,

Richard

Re: [PATCH] kernel.bbclass: improve reproducibility

[Trevor Woerner]

Should this be configurable?

Is there an assumption everyone who uses oe-core wants reproducible builds?

Could this affect one's ability to perform debugging?

Re: [PATCH] kernel.bbclass: improve reproducibility

[Bystricky, Juro]

[-- Attachment #1: Type: text/plain, Size: 936 bytes --]

Quite possibly. However, the biggest abuser of __FILE__ seems to be kernel and kernel_modules.

________________________________
From: Burton, Ross [ross.burton@intel.com]
Sent: Thursday, May 18, 2017 2:10 PM
To: Bystricky, Juro
Cc: OE-core; Purdie, Richard; Joshua G Lock; Juro Bystricky
Subject: Re: [PATCH] kernel.bbclass: improve reproducibility


On 18 May 2017 at 22:05, Juro Bystricky <juro.bystricky@intel.com<mailto:juro.bystricky@intel.com>> wrote:
Kernel and kernel modules contain hard coded paths referencing the host
build system. This is usually because the source code contains __FILE__
at some place. This prevents binary reproducibility. However, some compilers
allow remapping of the __FILE__ value. If we detect the compiler is capable
of doing this, we replace the source path $(S) part of __FILE__ by a string "/kernel-source".
For example:

Is this not something we should do globally?

Ross

[-- Attachment #2: Type: text/html, Size: 1979 bytes --]

Re: [PATCH] kernel.bbclass: improve reproducibility

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 564 bytes --]

On 18 May 2017 at 22:05, Juro Bystricky <juro.bystricky@intel.com> wrote:

> Kernel and kernel modules contain hard coded paths referencing the host
> build system. This is usually because the source code contains __FILE__
> at some place. This prevents binary reproducibility. However, some
> compilers
> allow remapping of the __FILE__ value. If we detect the compiler is capable
> of doing this, we replace the source path $(S) part of __FILE__ by a
> string "/kernel-source".
> For example:
>

Is this not something we should do globally?

Ross

[-- Attachment #2: Type: text/html, Size: 989 bytes --]

[PATCH] kernel.bbclass: improve reproducibility

[Juro Bystricky]

Kernel and kernel modules contain hard coded paths referencing the host
build system. This is usually because the source code contains __FILE__
at some place. This prevents binary reproducibility. However, some compilers
allow remapping of the __FILE__ value. If we detect the compiler is capable
of doing this, we replace the source path $(S) part of __FILE__ by a string "/kernel-source".
For example:

​/​data/​master/​build-​repro/​tmp/​work-​shared/​qemux86/​kernel-​source/​drivers/​media/​v4l2-​core/​videobuf2-​core.​c

will be replaced by a reproducible value:

/kernel-source/drivers/media/v4l2-core/videobuf2-core.c.

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
 meta/classes/kernel.bbclass | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index 8954b28..0a2cd3d 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -249,6 +249,18 @@ python do_devshell_prepend () {
 
 addtask bundle_initramfs after do_install before do_deploy
 
+get_cc_option () {
+		# Check if KERNEL_CC supports the option "file-prefix-map".
+		# This option allows us to build images with __FILE__ values that do not
+		# contain the host build path.
+		cc_option_supported=`${KERNEL_CC} -Q --help=joined | grep ffile-prefix-map`
+		cc_extra=""
+		if [ $cc_option_supported = "-ffile-prefix-map=<old=new>" ]; then
+			cc_extra=-ffile-prefix-map=${S}=/kernel-source/
+		fi
+		echo $cc_extra
+}
+
 kernel_do_compile() {
 	unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS MACHINE
 	# The $use_alternate_initrd is only set from
@@ -264,8 +276,9 @@ kernel_do_compile() {
 		copy_initramfs
 		use_alternate_initrd=CONFIG_INITRAMFS_SOURCE=${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio
 	fi
+	cc_extra=$(get_cc_option)
 	for typeformake in ${KERNEL_IMAGETYPE_FOR_MAKE} ; do
-		oe_runmake ${typeformake} CC="${KERNEL_CC}" LD="${KERNEL_LD}" ${KERNEL_EXTRA_ARGS} $use_alternate_initrd
+		oe_runmake ${typeformake} CC="${KERNEL_CC} $cc_extra " LD="${KERNEL_LD}" ${KERNEL_EXTRA_ARGS} $use_alternate_initrd
 		for type in ${KERNEL_IMAGETYPES} ; do
 			if test "${typeformake}.gz" = "${type}"; then
 				mkdir -p "${KERNEL_OUTPUT_DIR}"
@@ -279,7 +292,8 @@ kernel_do_compile() {
 do_compile_kernelmodules() {
 	unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS MACHINE
 	if (grep -q -i -e '^CONFIG_MODULES=y$' ${B}/.config); then
-		oe_runmake -C ${B} ${PARALLEL_MAKE} modules CC="${KERNEL_CC}" LD="${KERNEL_LD}" ${KERNEL_EXTRA_ARGS}
+		cc_extra=$(get_cc_option)
+		oe_runmake -C ${B} ${PARALLEL_MAKE} modules CC="${KERNEL_CC} $cc_extra " LD="${KERNEL_LD}" ${KERNEL_EXTRA_ARGS}
 
 		# Module.symvers gets updated during the 
 		# building of the kernel modules. We need to
-- 
2.7.4