[PATCH] make-mod-scripts: Fix a rare build race condition

[PATCH] make-mod-scripts: Fix a rare build race condition

[Khem Raj]

There is a build break which often happens whem using 4.1 kernel

/bin/sh: 1: scripts/basic/fixdep: Permission denied
scripts/Makefile.host:124: recipe for target 'scripts/dtc/srcpos.o' failed
make[3]: *** [scripts/dtc/srcpos.o] Error 126

this patch sequences the build targets so it can work reliably with
different kernel versions

Signed-off-by: Khem Raj <raj.khem@gmail.com>
Cc: Bruce Ashfield <bruce.ashfield@gmail.com>
---
 .../make-mod-scripts/make-mod-scripts_1.0.bb               | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
index 0be1422a24..c7edb20ee4 100644
--- a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
+++ b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
@@ -23,7 +23,8 @@ EXTRA_OEMAKE = " HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="
 #
 do_configure() {
 	unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
-	oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
-	           -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} scripts prepare
-
+	for t in prepare scripts_basic scripts; do
+		oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
+		-C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
+	done
 }
-- 
2.26.2

Re: [OE-core] [PATCH] make-mod-scripts: Fix a rare build race condition

[Denys Dmytriyenko]

On Wed, May 27, 2020 at 09:03:55AM -0700, Khem Raj wrote:
> There is a build break which often happens whem using 4.1 kernel
> 
> /bin/sh: 1: scripts/basic/fixdep: Permission denied
> scripts/Makefile.host:124: recipe for target 'scripts/dtc/srcpos.o' failed
> make[3]: *** [scripts/dtc/srcpos.o] Error 126
> 
> this patch sequences the build targets so it can work reliably with
> different kernel versions

Oh, I remember this failure from few years ago. Was it fixed upstream? Any 
specifics?


> Signed-off-by: Khem Raj <raj.khem@gmail.com>
> Cc: Bruce Ashfield <bruce.ashfield@gmail.com>
> ---
>  .../make-mod-scripts/make-mod-scripts_1.0.bb               | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> index 0be1422a24..c7edb20ee4 100644
> --- a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> +++ b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> @@ -23,7 +23,8 @@ EXTRA_OEMAKE = " HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="
>  #
>  do_configure() {
>  	unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
> -	oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
> -	           -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} scripts prepare
> -
> +	for t in prepare scripts_basic scripts; do
> +		oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
> +		-C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
> +	done
>  }
> -- 
> 2.26.2
> 

> 

Re: [PATCH] make-mod-scripts: Fix a rare build race condition

[Bruce Ashfield]

On Wed, May 27, 2020 at 12:03 PM Khem Raj <raj.khem@gmail.com> wrote:
>
> There is a build break which often happens whem using 4.1 kernel
>
> /bin/sh: 1: scripts/basic/fixdep: Permission denied
> scripts/Makefile.host:124: recipe for target 'scripts/dtc/srcpos.o' failed
> make[3]: *** [scripts/dtc/srcpos.o] Error 126
>
> this patch sequences the build targets so it can work reliably with
> different kernel versions
>
> Signed-off-by: Khem Raj <raj.khem@gmail.com>
> Cc: Bruce Ashfield <bruce.ashfield@gmail.com>
> ---
>  .../make-mod-scripts/make-mod-scripts_1.0.bb               | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> index 0be1422a24..c7edb20ee4 100644
> --- a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> +++ b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> @@ -23,7 +23,8 @@ EXTRA_OEMAKE = " HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="
>  #
>  do_configure() {
>         unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
> -       oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
> -                  -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} scripts prepare
> -
> +       for t in prepare scripts_basic scripts; do

Last I checked, we don't actually need scripts_basic.

Isn't just sequencing prepare, followed by 'scripts' enough ?

If not, we really should put in the long log why we need that new
target when sequencing, when we didn't need it before.

Bruce

> +               oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
> +               -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
> +       done
>  }
> --
> 2.26.2
>


-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [PATCH] make-mod-scripts: Fix a rare build race condition

[Khem Raj]

On Wed, May 27, 2020 at 9:57 AM Bruce Ashfield <bruce.ashfield@gmail.com> wrote:
>
> On Wed, May 27, 2020 at 12:03 PM Khem Raj <raj.khem@gmail.com> wrote:
> >
> > There is a build break which often happens whem using 4.1 kernel
> >
> > /bin/sh: 1: scripts/basic/fixdep: Permission denied
> > scripts/Makefile.host:124: recipe for target 'scripts/dtc/srcpos.o' failed
> > make[3]: *** [scripts/dtc/srcpos.o] Error 126
> >
> > this patch sequences the build targets so it can work reliably with
> > different kernel versions
> >
> > Signed-off-by: Khem Raj <raj.khem@gmail.com>
> > Cc: Bruce Ashfield <bruce.ashfield@gmail.com>
> > ---
> >  .../make-mod-scripts/make-mod-scripts_1.0.bb               | 7 ++++---
> >  1 file changed, 4 insertions(+), 3 deletions(-)
> >
> > diff --git a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> > index 0be1422a24..c7edb20ee4 100644
> > --- a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> > +++ b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> > @@ -23,7 +23,8 @@ EXTRA_OEMAKE = " HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="
> >  #
> >  do_configure() {
> >         unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
> > -       oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
> > -                  -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} scripts prepare
> > -
> > +       for t in prepare scripts_basic scripts; do
>
> Last I checked, we don't actually need scripts_basic.
>
> Isn't just sequencing prepare, followed by 'scripts' enough ?
>
> If not, we really should put in the long log why we need that new
> target when sequencing, when we didn't need it before.
>

in a way you are right that scripts should cover it, but calling it
out explicitly avoids any weird races if any and it does not
hurt until this target is removed. I really have not debugged kernel
build to see what is causing this, so there is not much
details to add other than this that we are handing some race
conditions which ideally should be in the makery

> Bruce
>
> > +               oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
> > +               -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
> > +       done
> >  }
> > --
> > 2.26.2
> >
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II

Re: [PATCH] make-mod-scripts: Fix a rare build race condition

[Bruce Ashfield]

On Wed, May 27, 2020 at 1:46 PM Khem Raj <raj.khem@gmail.com> wrote:
>
> On Wed, May 27, 2020 at 9:57 AM Bruce Ashfield <bruce.ashfield@gmail.com> wrote:
> >
> > On Wed, May 27, 2020 at 12:03 PM Khem Raj <raj.khem@gmail.com> wrote:
> > >
> > > There is a build break which often happens whem using 4.1 kernel
> > >
> > > /bin/sh: 1: scripts/basic/fixdep: Permission denied
> > > scripts/Makefile.host:124: recipe for target 'scripts/dtc/srcpos.o' failed
> > > make[3]: *** [scripts/dtc/srcpos.o] Error 126
> > >
> > > this patch sequences the build targets so it can work reliably with
> > > different kernel versions
> > >
> > > Signed-off-by: Khem Raj <raj.khem@gmail.com>
> > > Cc: Bruce Ashfield <bruce.ashfield@gmail.com>
> > > ---
> > >  .../make-mod-scripts/make-mod-scripts_1.0.bb               | 7 ++++---
> > >  1 file changed, 4 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> > > index 0be1422a24..c7edb20ee4 100644
> > > --- a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> > > +++ b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> > > @@ -23,7 +23,8 @@ EXTRA_OEMAKE = " HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="
> > >  #
> > >  do_configure() {
> > >         unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
> > > -       oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
> > > -                  -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} scripts prepare
> > > -
> > > +       for t in prepare scripts_basic scripts; do
> >
> > Last I checked, we don't actually need scripts_basic.
> >
> > Isn't just sequencing prepare, followed by 'scripts' enough ?
> >
> > If not, we really should put in the long log why we need that new
> > target when sequencing, when we didn't need it before.
> >
>
> in a way you are right that scripts should cover it, but calling it
> out explicitly avoids any weird races if any and it does not
> hurt until this target is removed. I really have not debugged kernel
> build to see what is causing this, so there is not much
> details to add other than this that we are handing some race
> conditions which ideally should be in the makery
>

But the race condition is handled with the two targets, no ? I realize
it doesn't hurt, but you are implying that you need all three to be
sure, but that really isn't the case from what I've seen.

I just think simpler is better, but don't take my comments as an objection.

Cheers,

Bruce

> > Bruce
> >
> > > +               oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
> > > +               -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
> > > +       done
> > >  }
> > > --
> > > 2.26.2
> > >
> >
> >
> > --
> > - Thou shalt not follow the NULL pointer, for chaos and madness await
> > thee at its end
> > - "Use the force Harry" - Gandalf, Star Trek II



-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [PATCH] make-mod-scripts: Fix a rare build race condition

[Khem Raj]

On Wed, May 27, 2020 at 11:05 AM Bruce Ashfield
<bruce.ashfield@gmail.com> wrote:
>
> On Wed, May 27, 2020 at 1:46 PM Khem Raj <raj.khem@gmail.com> wrote:
> >
> > On Wed, May 27, 2020 at 9:57 AM Bruce Ashfield <bruce.ashfield@gmail.com> wrote:
> > >
> > > On Wed, May 27, 2020 at 12:03 PM Khem Raj <raj.khem@gmail.com> wrote:
> > > >
> > > > There is a build break which often happens whem using 4.1 kernel
> > > >
> > > > /bin/sh: 1: scripts/basic/fixdep: Permission denied
> > > > scripts/Makefile.host:124: recipe for target 'scripts/dtc/srcpos.o' failed
> > > > make[3]: *** [scripts/dtc/srcpos.o] Error 126
> > > >
> > > > this patch sequences the build targets so it can work reliably with
> > > > different kernel versions
> > > >
> > > > Signed-off-by: Khem Raj <raj.khem@gmail.com>
> > > > Cc: Bruce Ashfield <bruce.ashfield@gmail.com>
> > > > ---
> > > >  .../make-mod-scripts/make-mod-scripts_1.0.bb               | 7 ++++---
> > > >  1 file changed, 4 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> > > > index 0be1422a24..c7edb20ee4 100644
> > > > --- a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> > > > +++ b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> > > > @@ -23,7 +23,8 @@ EXTRA_OEMAKE = " HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="
> > > >  #
> > > >  do_configure() {
> > > >         unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
> > > > -       oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
> > > > -                  -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} scripts prepare
> > > > -
> > > > +       for t in prepare scripts_basic scripts; do
> > >
> > > Last I checked, we don't actually need scripts_basic.
> > >
> > > Isn't just sequencing prepare, followed by 'scripts' enough ?
> > >
> > > If not, we really should put in the long log why we need that new
> > > target when sequencing, when we didn't need it before.
> > >
> >
> > in a way you are right that scripts should cover it, but calling it
> > out explicitly avoids any weird races if any and it does not
> > hurt until this target is removed. I really have not debugged kernel
> > build to see what is causing this, so there is not much
> > details to add other than this that we are handing some race
> > conditions which ideally should be in the makery
> >
>
> But the race condition is handled with the two targets, no ? I realize
> it doesn't hurt, but you are implying that you need all three to be
> sure, but that really isn't the case from what I've seen.
>
> I just think simpler is better, but don't take my comments as an objection.

Since it is a random failure that means validating any changes to fix
it is as hard as reproducing it.
This solution has worked over a number of builds, testing anything
else would take as much long and I am reluctant to do that
without a potential gain.

>
> Cheers,
>
> Bruce
>
> > > Bruce
> > >
> > > > +               oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
> > > > +               -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
> > > > +       done
> > > >  }
> > > > --
> > > > 2.26.2
> > > >
> > >
> > >
> > > --
> > > - Thou shalt not follow the NULL pointer, for chaos and madness await
> > > thee at its end
> > > - "Use the force Harry" - Gandalf, Star Trek II
>
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II

Re: [PATCH] make-mod-scripts: Fix a rare build race condition

[Bruce Ashfield]

On Wed, May 27, 2020 at 3:23 PM Khem Raj <raj.khem@gmail.com> wrote:
>
> On Wed, May 27, 2020 at 11:05 AM Bruce Ashfield
> <bruce.ashfield@gmail.com> wrote:
> >
> > On Wed, May 27, 2020 at 1:46 PM Khem Raj <raj.khem@gmail.com> wrote:
> > >
> > > On Wed, May 27, 2020 at 9:57 AM Bruce Ashfield <bruce.ashfield@gmail.com> wrote:
> > > >
> > > > On Wed, May 27, 2020 at 12:03 PM Khem Raj <raj.khem@gmail.com> wrote:
> > > > >
> > > > > There is a build break which often happens whem using 4.1 kernel
> > > > >
> > > > > /bin/sh: 1: scripts/basic/fixdep: Permission denied
> > > > > scripts/Makefile.host:124: recipe for target 'scripts/dtc/srcpos.o' failed
> > > > > make[3]: *** [scripts/dtc/srcpos.o] Error 126
> > > > >
> > > > > this patch sequences the build targets so it can work reliably with
> > > > > different kernel versions
> > > > >
> > > > > Signed-off-by: Khem Raj <raj.khem@gmail.com>
> > > > > Cc: Bruce Ashfield <bruce.ashfield@gmail.com>
> > > > > ---
> > > > >  .../make-mod-scripts/make-mod-scripts_1.0.bb               | 7 ++++---
> > > > >  1 file changed, 4 insertions(+), 3 deletions(-)
> > > > >
> > > > > diff --git a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> > > > > index 0be1422a24..c7edb20ee4 100644
> > > > > --- a/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> > > > > +++ b/meta/recipes-kernel/make-mod-scripts/make-mod-scripts_1.0.bb
> > > > > @@ -23,7 +23,8 @@ EXTRA_OEMAKE = " HOSTCC="${BUILD_CC} ${BUILD_CFLAGS} ${BUILD_LDFLAGS}" HOSTCPP="
> > > > >  #
> > > > >  do_configure() {
> > > > >         unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
> > > > > -       oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
> > > > > -                  -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} scripts prepare
> > > > > -
> > > > > +       for t in prepare scripts_basic scripts; do
> > > >
> > > > Last I checked, we don't actually need scripts_basic.
> > > >
> > > > Isn't just sequencing prepare, followed by 'scripts' enough ?
> > > >
> > > > If not, we really should put in the long log why we need that new
> > > > target when sequencing, when we didn't need it before.
> > > >
> > >
> > > in a way you are right that scripts should cover it, but calling it
> > > out explicitly avoids any weird races if any and it does not
> > > hurt until this target is removed. I really have not debugged kernel
> > > build to see what is causing this, so there is not much
> > > details to add other than this that we are handing some race
> > > conditions which ideally should be in the makery
> > >
> >
> > But the race condition is handled with the two targets, no ? I realize
> > it doesn't hurt, but you are implying that you need all three to be
> > sure, but that really isn't the case from what I've seen.
> >
> > I just think simpler is better, but don't take my comments as an objection.
>
> Since it is a random failure that means validating any changes to fix
> it is as hard as reproducing it.
> This solution has worked over a number of builds, testing anything
> else would take as much long and I am reluctant to do that
> without a potential gain.

Minimizing the amount of change is definitely consider a gain by many
(me included).

But I'm happy enough if you can just amend the message to indicate
that it isn't strictly necessary and was simply what was used for
testing.

Since I guarantee, someday, I'll be staring at that change wondering
why it was added :D

Bruce

>
> >
> > Cheers,
> >
> > Bruce
> >
> > > > Bruce
> > > >
> > > > > +               oe_runmake CC="${KERNEL_CC}" LD="${KERNEL_LD}" AR="${KERNEL_AR}" \
> > > > > +               -C ${STAGING_KERNEL_DIR} O=${STAGING_KERNEL_BUILDDIR} $t
> > > > > +       done
> > > > >  }
> > > > > --
> > > > > 2.26.2
> > > > >
> > > >
> > > >
> > > > --
> > > > - Thou shalt not follow the NULL pointer, for chaos and madness await
> > > > thee at its end
> > > > - "Use the force Harry" - Gandalf, Star Trek II
> >
> >
> >
> > --
> > - Thou shalt not follow the NULL pointer, for chaos and madness await
> > thee at its end
> > - "Use the force Harry" - Gandalf, Star Trek II



-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II