[Buildroot] [PATCH 1/1] package/bearssl: disable parallel build

[Buildroot] [PATCH 1/1] package/bearssl: disable parallel build

[James Hilliard]

The bearssl package is not compatible with parallel builds.

Fixes:
 - http://autobuild.buildroot.net/results/5e1/5e1b72db206b2d17fa5ce378a70fdc2f3f5bf773

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
---
 package/bearssl/bearssl.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/bearssl/bearssl.mk b/package/bearssl/bearssl.mk
index ff5f4e87b7..c2a5760af5 100644
--- a/package/bearssl/bearssl.mk
+++ b/package/bearssl/bearssl.mk
@@ -24,7 +24,7 @@ BEARSSL_TARGETS += lib
 endif
 
 define BEARSSL_BUILD_CMDS
-	$(TARGET_MAKE_ENV) $(MAKE) $(BEARSSL_MAKE_OPTS) -C $(@D) \
+	$(TARGET_MAKE_ENV) $(MAKE1) $(BEARSSL_MAKE_OPTS) -C $(@D) \
 		$(BEARSSL_TARGETS)
 endef
 
-- 
2.34.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/bearssl: disable parallel build

[Fabrice Fontaine]

[-- Attachment #1.1: Type: text/plain, Size: 1179 bytes --]

Hi James,

Le sam. 15 oct. 2022 à 02:58, James Hilliard <james.hilliard1@gmail.com> a
écrit :

> The bearssl package is not compatible with parallel builds.
>

bearssl was building perfectly fine up until now and it seems there are a
lot of packages that are affected by parallel build issues (e.g. dmraid,
musepack, slang, etc).
Did we change something recently on the autobuilders?


>
> Fixes:
>  -
> http://autobuild.buildroot.net/results/5e1/5e1b72db206b2d17fa5ce378a70fdc2f3f5bf773
>
> Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
> ---
>  package/bearssl/bearssl.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/package/bearssl/bearssl.mk b/package/bearssl/bearssl.mk
> index ff5f4e87b7..c2a5760af5 100644
> --- a/package/bearssl/bearssl.mk
> +++ b/package/bearssl/bearssl.mk
> @@ -24,7 +24,7 @@ BEARSSL_TARGETS += lib
>  endif
>
>  define BEARSSL_BUILD_CMDS
> -       $(TARGET_MAKE_ENV) $(MAKE) $(BEARSSL_MAKE_OPTS) -C $(@D) \
> +       $(TARGET_MAKE_ENV) $(MAKE1) $(BEARSSL_MAKE_OPTS) -C $(@D) \
>                 $(BEARSSL_TARGETS)
>  endef
>
> --
> 2.34.1
>
> Best Regards,

Fabrice

[-- Attachment #1.2: Type: text/html, Size: 2293 bytes --]

[-- Attachment #2: Type: text/plain, Size: 150 bytes --]

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/bearssl: disable parallel build

[James Hilliard]

On Sat, Oct 15, 2022 at 6:14 PM Fabrice Fontaine
<fontaine.fabrice@gmail.com> wrote:
>
> Hi James,
>
> Le sam. 15 oct. 2022 à 02:58, James Hilliard <james.hilliard1@gmail.com> a écrit :
>>
>> The bearssl package is not compatible with parallel builds.
>
>
> bearssl was building perfectly fine up until now and it seems there are a lot of packages that are affected by parallel build issues (e.g. dmraid, musepack, slang, etc).

Some parallel build bugs don't reproduce reliably due to timing, I made a change
to my autobuilders which massively increases the chance of triggering
build failures
in packages which have dependency graph bugs. This triggers build build failures
in -j1 builds which would otherwise only appear when running parallel builds(my
autobuilders are generally running with -j1 to make the logs more readable).

> Did we change something recently on the autobuilders?

Yes, all my autobuilders were recently changed to use make master(will be
make version 4.4 once released) with top level --shuffle=random.

Details:
https://github.com/mirror/make/blob/3e20e376b7aa26fa43f81f23323c717c4cae1bfb/NEWS#L123-L128
https://trofi.github.io/posts/238-new-make-shuffle-mode.html
http://trofi.github.io/posts/249-an-update-on-make-shuffle.html

Note that MAKE1 is currently broken itself without this when using shuffle mode:
https://patchwork.ozlabs.org/project/buildroot/patch/20221015005611.4054933-1-james.hilliard1@gmail.com/

>
>>
>>
>> Fixes:
>>  - http://autobuild.buildroot.net/results/5e1/5e1b72db206b2d17fa5ce378a70fdc2f3f5bf773
>>
>> Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
>> ---
>>  package/bearssl/bearssl.mk | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/package/bearssl/bearssl.mk b/package/bearssl/bearssl.mk
>> index ff5f4e87b7..c2a5760af5 100644
>> --- a/package/bearssl/bearssl.mk
>> +++ b/package/bearssl/bearssl.mk
>> @@ -24,7 +24,7 @@ BEARSSL_TARGETS += lib
>>  endif
>>
>>  define BEARSSL_BUILD_CMDS
>> -       $(TARGET_MAKE_ENV) $(MAKE) $(BEARSSL_MAKE_OPTS) -C $(@D) \
>> +       $(TARGET_MAKE_ENV) $(MAKE1) $(BEARSSL_MAKE_OPTS) -C $(@D) \
>>                 $(BEARSSL_TARGETS)
>>  endef
>>
>> --
>> 2.34.1
>>
> Best Regards,
>
> Fabrice
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/bearssl: disable parallel build

[Yann E. MORIN]

James, All,

On 2022-10-15 19:21 -0400, James Hilliard spake thusly:
> On Sat, Oct 15, 2022 at 6:14 PM Fabrice Fontaine
> <fontaine.fabrice@gmail.com> wrote:
> > Le sam. 15 oct. 2022 à 02:58, James Hilliard <james.hilliard1@gmail.com> a écrit :
> >> The bearssl package is not compatible with parallel builds.
> > bearssl was building perfectly fine up until now and it seems there
> > are a lot of packages that are affected by parallel build issues
> > (e.g. dmraid, musepack, slang, etc).
> 
> Some parallel build bugs don't reproduce reliably due to timing, I made a change
> to my autobuilders which massively increases the chance of triggering
> build failures
> in packages which have dependency graph bugs. This triggers build build failures
> in -j1 builds which would otherwise only appear when running parallel builds(my
> autobuilders are generally running with -j1 to make the logs more readable).
> 
> > Did we change something recently on the autobuilders?
> 
> Yes, all my autobuilders were recently changed to use make master(will be
> make version 4.4 once released) with top level --shuffle=random.

By disabling parallel build because you are using ana as yet unreleased
version, we remove the parallel build for everyone, which is not very
nice at all.

Instead, we should really try and fix the packages rather than papering
over the issue.

Notes:
  - it *is* interesting to find those bugs;
  - it is sad that the default mode for --shuffle will be random, not
    none.

Regards,
Yann E. MORIN.

> Details:
> https://github.com/mirror/make/blob/3e20e376b7aa26fa43f81f23323c717c4cae1bfb/NEWS#L123-L128
> https://trofi.github.io/posts/238-new-make-shuffle-mode.html
> http://trofi.github.io/posts/249-an-update-on-make-shuffle.html
> 
> Note that MAKE1 is currently broken itself without this when using shuffle mode:
> https://patchwork.ozlabs.org/project/buildroot/patch/20221015005611.4054933-1-james.hilliard1@gmail.com/
> 
> >
> >>
> >>
> >> Fixes:
> >>  - http://autobuild.buildroot.net/results/5e1/5e1b72db206b2d17fa5ce378a70fdc2f3f5bf773
> >>
> >> Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
> >> ---
> >>  package/bearssl/bearssl.mk | 2 +-
> >>  1 file changed, 1 insertion(+), 1 deletion(-)
> >>
> >> diff --git a/package/bearssl/bearssl.mk b/package/bearssl/bearssl.mk
> >> index ff5f4e87b7..c2a5760af5 100644
> >> --- a/package/bearssl/bearssl.mk
> >> +++ b/package/bearssl/bearssl.mk
> >> @@ -24,7 +24,7 @@ BEARSSL_TARGETS += lib
> >>  endif
> >>
> >>  define BEARSSL_BUILD_CMDS
> >> -       $(TARGET_MAKE_ENV) $(MAKE) $(BEARSSL_MAKE_OPTS) -C $(@D) \
> >> +       $(TARGET_MAKE_ENV) $(MAKE1) $(BEARSSL_MAKE_OPTS) -C $(@D) \
> >>                 $(BEARSSL_TARGETS)
> >>  endef
> >>
> >> --
> >> 2.34.1
> >>
> > Best Regards,
> >
> > Fabrice
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/bearssl: disable parallel build

[James Hilliard]

On Sun, Oct 16, 2022 at 3:51 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> James, All,
>
> On 2022-10-15 19:21 -0400, James Hilliard spake thusly:
> > On Sat, Oct 15, 2022 at 6:14 PM Fabrice Fontaine
> > <fontaine.fabrice@gmail.com> wrote:
> > > Le sam. 15 oct. 2022 à 02:58, James Hilliard <james.hilliard1@gmail.com> a écrit :
> > >> The bearssl package is not compatible with parallel builds.
> > > bearssl was building perfectly fine up until now and it seems there
> > > are a lot of packages that are affected by parallel build issues
> > > (e.g. dmraid, musepack, slang, etc).
> >
> > Some parallel build bugs don't reproduce reliably due to timing, I made a change
> > to my autobuilders which massively increases the chance of triggering
> > build failures
> > in packages which have dependency graph bugs. This triggers build build failures
> > in -j1 builds which would otherwise only appear when running parallel builds(my
> > autobuilders are generally running with -j1 to make the logs more readable).
> >
> > > Did we change something recently on the autobuilders?
> >
> > Yes, all my autobuilders were recently changed to use make master(will be
> > make version 4.4 once released) with top level --shuffle=random.
>
> By disabling parallel build because you are using ana as yet unreleased
> version, we remove the parallel build for everyone, which is not very
> nice at all.

Failures with shuffle mode indicate that the package is not parallel compatible
on existing make releases as well, it just makes those bugs more visible.

I'm running shuffle with -j1 because that way it's obvious in the logs where the
failure is getting hit, otherwise these bugs tend to be transient and
difficult to
trace as they otherwise are timing dependent.

>
> Instead, we should really try and fix the packages rather than papering
> over the issue.
>
> Notes:
>   - it *is* interesting to find those bugs;
>   - it is sad that the default mode for --shuffle will be random, not
>     none.

Default is equivalent to none, I modified my autobuilder runner to pass
--shuffle=random.

>
> Regards,
> Yann E. MORIN.
>
> > Details:
> > https://github.com/mirror/make/blob/3e20e376b7aa26fa43f81f23323c717c4cae1bfb/NEWS#L123-L128
> > https://trofi.github.io/posts/238-new-make-shuffle-mode.html
> > http://trofi.github.io/posts/249-an-update-on-make-shuffle.html
> >
> > Note that MAKE1 is currently broken itself without this when using shuffle mode:
> > https://patchwork.ozlabs.org/project/buildroot/patch/20221015005611.4054933-1-james.hilliard1@gmail.com/
> >
> > >
> > >>
> > >>
> > >> Fixes:
> > >>  - http://autobuild.buildroot.net/results/5e1/5e1b72db206b2d17fa5ce378a70fdc2f3f5bf773
> > >>
> > >> Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
> > >> ---
> > >>  package/bearssl/bearssl.mk | 2 +-
> > >>  1 file changed, 1 insertion(+), 1 deletion(-)
> > >>
> > >> diff --git a/package/bearssl/bearssl.mk b/package/bearssl/bearssl.mk
> > >> index ff5f4e87b7..c2a5760af5 100644
> > >> --- a/package/bearssl/bearssl.mk
> > >> +++ b/package/bearssl/bearssl.mk
> > >> @@ -24,7 +24,7 @@ BEARSSL_TARGETS += lib
> > >>  endif
> > >>
> > >>  define BEARSSL_BUILD_CMDS
> > >> -       $(TARGET_MAKE_ENV) $(MAKE) $(BEARSSL_MAKE_OPTS) -C $(@D) \
> > >> +       $(TARGET_MAKE_ENV) $(MAKE1) $(BEARSSL_MAKE_OPTS) -C $(@D) \
> > >>                 $(BEARSSL_TARGETS)
> > >>  endef
> > >>
> > >> --
> > >> 2.34.1
> > >>
> > > Best Regards,
> > >
> > > Fabrice
> > _______________________________________________
> > buildroot mailing list
> > buildroot@buildroot.org
> > https://lists.buildroot.org/mailman/listinfo/buildroot
>
> --
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/bearssl: disable parallel build

[Yann E. MORIN]

James, All,

+Thomas, Peter, Arnout

On 2022-10-16 11:44 -0400, James Hilliard spake thusly:
> On Sun, Oct 16, 2022 at 3:51 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> > On 2022-10-15 19:21 -0400, James Hilliard spake thusly:
[--SNIP--]
> > > Yes, all my autobuilders were recently changed to use make master(will be
> > > make version 4.4 once released) with top level --shuffle=random.
> > By disabling parallel build because you are using ana as yet unreleased
> > version, we remove the parallel build for everyone, which is not very
> > nice at all.
> Failures with shuffle mode indicate that the package is not parallel compatible
> on existing make releases as well, it just makes those bugs more visible.

I never said they were not, and I even said (further on) that it *is*
interesting to find such bugs.

However, the behaviour so far has been ordering of prerequisites (and
goals!), and for some packages, that ordering was enough to work in a
parallel make.

For example, the bearsll parallel issue is caused by the ordering so
the mkdir is very early. Yes, it is incorrect, but it happens to always
work in practice. And yes it is easy to fix, here's a patch:

    diff -durN a/mk/mkrules.sh b/mk/mkrules.sh
    --- a/mk/mkrules.sh	2018-08-14 22:41:54.000000000 +0200
    +++ b/mk/mkrules.sh	2022-10-16 17:54:22.282069851 +0200
    @@ -531,23 +531,23 @@
     (for f in $coresrc ; do
     	b="$(basename "$f" .c)\$O"
     	g="$(escsep "$f")"
    -	printf '\n$(OBJDIR)$P%s: %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
    +	printf '\n$(OBJDIR)$P%s: $(OBJDIR) %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
     done
     
     for f in $toolssrc ; do
     	b="$(basename "$f" .c)\$O"
     	g="$(escsep "$f")"
    -	printf '\n$(OBJDIR)$P%s: %s $(HEADERSTOOLS)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
    +	printf '\n$(OBJDIR)$P%s: $(OBJDIR) %s $(HEADERSTOOLS)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
     done
     
     for f in $testcryptosrc $testspeedsrc ; do
     	b="$(basename "$f" .c)\$O"
     	g="$(escsep "$f")"
    -	printf '\n$(OBJDIR)$P%s: %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
    +	printf '\n$(OBJDIR)$P%s: $(OBJDIR) %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
     done
     
     for f in $testx509src ; do
     	b="$(basename "$f" .c)\$O"
     	g="$(escsep "$f")"
    -	printf '\n$(OBJDIR)$P%s: %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) -DSRCDIRNAME=".." $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
    +	printf '\n$(OBJDIR)$P%s: $(OBJDIR) %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) -DSRCDIRNAME=".." $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
     done) >> Rules.mk
    Binary files a/mk/.mkrules.sh.swp and b/mk/.mkrules.sh.swp differ

And then adding a CONFIGURE_CMDS that runs mk/mkrules.sh to regenerate
the Makefile.

> I'm running shuffle with -j1 because that way it's obvious in the logs where the
> failure is getting hit, otherwise these bugs tend to be transient and
> difficult to
> trace as they otherwise are timing dependent.

Yes, this is very interesting to find such bugs, but disabling parallel
build in packages is not nice when they can be (easily) fixed.

> > Notes:
> >   - it *is* interesting to find those bugs;
> >   - it is sad that the default mode for --shuffle will be random, not
> >     none.
> Default is equivalent to none,

Yeah, I noticed the hard way after having my patch rejected in makem as
you may have noticed. :-/

> I modified my autobuilder runner to pass
> --shuffle=random.

Again, that is very interesting, but please, please, conordinate with
the rest of us when you do such changes, because that is very weird, as
Fabrice noted, to suddenly get build failures out of the blue when
nothing seemed to have otherwise changed.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/bearssl: disable parallel build

[James Hilliard]

On Sun, Oct 16, 2022 at 12:05 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> James, All,
>
> +Thomas, Peter, Arnout
>
> On 2022-10-16 11:44 -0400, James Hilliard spake thusly:
> > On Sun, Oct 16, 2022 at 3:51 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> > > On 2022-10-15 19:21 -0400, James Hilliard spake thusly:
> [--SNIP--]
> > > > Yes, all my autobuilders were recently changed to use make master(will be
> > > > make version 4.4 once released) with top level --shuffle=random.
> > > By disabling parallel build because you are using ana as yet unreleased
> > > version, we remove the parallel build for everyone, which is not very
> > > nice at all.
> > Failures with shuffle mode indicate that the package is not parallel compatible
> > on existing make releases as well, it just makes those bugs more visible.
>
> I never said they were not, and I even said (further on) that it *is*
> interesting to find such bugs.
>
> However, the behaviour so far has been ordering of prerequisites (and
> goals!), and for some packages, that ordering was enough to work in a
> parallel make.
>
> For example, the bearsll parallel issue is caused by the ordering so
> the mkdir is very early. Yes, it is incorrect, but it happens to always
> work in practice. And yes it is easy to fix, here's a patch:

Yeah, I think a lot of these are bugs that usually don't appear most of the
time but may randomly pop up under high load highly parallel conditions.

Randomly failing parallel builds has been an issue I've seen pop up a good
bit for my project builds so I wanted to try and find a way to make them
more visible.

>
>     diff -durN a/mk/mkrules.sh b/mk/mkrules.sh
>     --- a/mk/mkrules.sh 2018-08-14 22:41:54.000000000 +0200
>     +++ b/mk/mkrules.sh 2022-10-16 17:54:22.282069851 +0200
>     @@ -531,23 +531,23 @@
>      (for f in $coresrc ; do
>         b="$(basename "$f" .c)\$O"
>         g="$(escsep "$f")"
>     -   printf '\n$(OBJDIR)$P%s: %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
>     +   printf '\n$(OBJDIR)$P%s: $(OBJDIR) %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
>      done
>
>      for f in $toolssrc ; do
>         b="$(basename "$f" .c)\$O"
>         g="$(escsep "$f")"
>     -   printf '\n$(OBJDIR)$P%s: %s $(HEADERSTOOLS)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
>     +   printf '\n$(OBJDIR)$P%s: $(OBJDIR) %s $(HEADERSTOOLS)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
>      done
>
>      for f in $testcryptosrc $testspeedsrc ; do
>         b="$(basename "$f" .c)\$O"
>         g="$(escsep "$f")"
>     -   printf '\n$(OBJDIR)$P%s: %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
>     +   printf '\n$(OBJDIR)$P%s: $(OBJDIR) %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
>      done
>
>      for f in $testx509src ; do
>         b="$(basename "$f" .c)\$O"
>         g="$(escsep "$f")"
>     -   printf '\n$(OBJDIR)$P%s: %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) -DSRCDIRNAME=".." $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
>     +   printf '\n$(OBJDIR)$P%s: $(OBJDIR) %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) -DSRCDIRNAME=".." $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
>      done) >> Rules.mk
>     Binary files a/mk/.mkrules.sh.swp and b/mk/.mkrules.sh.swp differ
>
> And then adding a CONFIGURE_CMDS that runs mk/mkrules.sh to regenerate
> the Makefile.
>
> > I'm running shuffle with -j1 because that way it's obvious in the logs where the
> > failure is getting hit, otherwise these bugs tend to be transient and
> > difficult to
> > trace as they otherwise are timing dependent.
>
> Yes, this is very interesting to find such bugs, but disabling parallel
> build in packages is not nice when they can be (easily) fixed.

Yeah, I hadn't investigated the root cause in detail as bearssl didn't
appear to be very widely used. At least with per-package directories
disabling parallel build for an individual package shouldn't meaningfully
slow down the build in most cases AFAIU.

I agree fixing the root cause is ideal, I thought our typical approach was
to just mark parallel incompatible stuff with MAKE1 and then fix things
later on as needed. If a lot of packages have dependency ordering issues
I assume we would want to prioritise fixing popular packages that have
long build times first.

What do you think the best strategy there would be?

>
> > > Notes:
> > >   - it *is* interesting to find those bugs;
> > >   - it is sad that the default mode for --shuffle will be random, not
> > >     none.
> > Default is equivalent to none,
>
> Yeah, I noticed the hard way after having my patch rejected in makem as
> you may have noticed. :-/
>
> > I modified my autobuilder runner to pass
> > --shuffle=random.
>
> Again, that is very interesting, but please, please, conordinate with
> the rest of us when you do such changes, because that is very weird, as
> Fabrice noted, to suddenly get build failures out of the blue when
> nothing seemed to have otherwise changed.

I've been experimenting a bit in general with autobuilder settings to try
and fix some issues with poor coverage.

One other reason for testing with shuffle=random is to make higher kconfig
probability values more useful(as otherwise build failures will be concentrated
in packages early in the dependency ordering) :
https://patchwork.ozlabs.org/project/buildroot/patch/20220514214612.3221647-1-james.hilliard1@gmail.com/

>
> Regards,
> Yann E. MORIN.
>
> --
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/bearssl: disable parallel build

[Thomas Petazzoni via buildroot]

Hello James,

On Sun, 16 Oct 2022 12:45:06 -0400
James Hilliard <james.hilliard1@gmail.com> wrote:

> > Again, that is very interesting, but please, please, conordinate with
> > the rest of us when you do such changes, because that is very weird, as
> > Fabrice noted, to suddenly get build failures out of the blue when
> > nothing seemed to have otherwise changed.  
> 
> I've been experimenting a bit in general with autobuilder settings to try
> and fix some issues with poor coverage.
> 
> One other reason for testing with shuffle=random is to make higher kconfig
> probability values more useful(as otherwise build failures will be concentrated
> in packages early in the dependency ordering) :
> https://patchwork.ozlabs.org/project/buildroot/patch/20220514214612.3221647-1-james.hilliard1@gmail.com/

Thanks a lot for all this work, which I find really useful. However,
like Yann said I believe we need to coordinate better. I think we would
prefer to see build results submitted on autobuild.buildroot.org to
only be generated by the official/upstream autobuild-run script.
Otherwise, it's super confusing as we get results that corresponding to
variations of build configurations/build logic that we are not even
aware they exist.

Another concern is that the autobuilder results are already extremely
bad these days, and therefore the priority is more on addressing/fixing
the existing issues rather than adding more. It's a lot
easier/practical to fix new problems when they are not mixed together
with zillions of old problems.

Again, thanks a lot for your work on this!

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/bearssl: disable parallel build

[James Hilliard]

On Mon, Oct 17, 2022 at 3:16 AM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello James,
>
> On Sun, 16 Oct 2022 12:45:06 -0400
> James Hilliard <james.hilliard1@gmail.com> wrote:
>
> > > Again, that is very interesting, but please, please, conordinate with
> > > the rest of us when you do such changes, because that is very weird, as
> > > Fabrice noted, to suddenly get build failures out of the blue when
> > > nothing seemed to have otherwise changed.
> >
> > I've been experimenting a bit in general with autobuilder settings to try
> > and fix some issues with poor coverage.
> >
> > One other reason for testing with shuffle=random is to make higher kconfig
> > probability values more useful(as otherwise build failures will be concentrated
> > in packages early in the dependency ordering) :
> > https://patchwork.ozlabs.org/project/buildroot/patch/20220514214612.3221647-1-james.hilliard1@gmail.com/
>
> Thanks a lot for all this work, which I find really useful. However,
> like Yann said I believe we need to coordinate better. I think we would
> prefer to see build results submitted on autobuild.buildroot.org to
> only be generated by the official/upstream autobuild-run script.
> Otherwise, it's super confusing as we get results that corresponding to
> variations of build configurations/build logic that we are not even
> aware they exist.

The shuffle=random changes don't require modifying autobuild-run, I'm
just tweaking the path to make in my autobuild service environment and
then passing an additional config flag to autobuild-run like this:
--make-opts="--shuffle=random"

>
> Another concern is that the autobuilder results are already extremely
> bad these days, and therefore the priority is more on addressing/fixing
> the existing issues rather than adding more. It's a lot
> easier/practical to fix new problems when they are not mixed together
> with zillions of old problems.

Well builds were mostly failing in the same place due to deterministic
build ordering, I was trying to make it so that we would see failures
in different places as that's more useful compared with always seeing
the same errors.

To me seeing the same errors most of the time indicates there's some
major coverage issues with the autobuilders which need fixing.

>
> Again, thanks a lot for your work on this!
>
> Thomas
> --
> Thomas Petazzoni, co-owner and CEO, Bootlin
> Embedded Linux and Kernel engineering and training
> https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/bearssl: disable parallel build

[Thomas Petazzoni via buildroot]

On Mon, 17 Oct 2022 07:29:52 -0400
James Hilliard <james.hilliard1@gmail.com> wrote:

> The shuffle=random changes don't require modifying autobuild-run, I'm
> just tweaking the path to make in my autobuild service environment and
> then passing an additional config flag to autobuild-run like this:
> --make-opts="--shuffle=random"

Right, but my point stands: it's custom usage/configuration of
autobuild-run that we would prefer to discuss on the mailing list first
before deploying/enabling.

> Well builds were mostly failing in the same place due to deterministic
> build ordering, I was trying to make it so that we would see failures
> in different places as that's more useful compared with always seeing
> the same errors.
> 
> To me seeing the same errors most of the time indicates there's some
> major coverage issues with the autobuilders which need fixing.

I think you're missing a point here: many of the builds today are
failing due to the glibc 2.36 bump. So pretty much every single build
that is glibc based will indeed fail with some error related to this
glibc bump.

Your additional randomization will not change anything to this. It's
just that we now have the glibc bump issues + the newly discovered
issues related to the extra randomization + the usual amount of build
issues.

Best regards,

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/bearssl: disable parallel build

[James Hilliard]

On Mon, Oct 17, 2022 at 8:21 AM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> On Mon, 17 Oct 2022 07:29:52 -0400
> James Hilliard <james.hilliard1@gmail.com> wrote:
>
> > The shuffle=random changes don't require modifying autobuild-run, I'm
> > just tweaking the path to make in my autobuild service environment and
> > then passing an additional config flag to autobuild-run like this:
> > --make-opts="--shuffle=random"
>
> Right, but my point stands: it's custom usage/configuration of
> autobuild-run that we would prefer to discuss on the mailing list first
> before deploying/enabling.

I've generally been deploying autobuild-run tweaks after initial local testing.

Typically I send them to the mailing list after running them deployed for
a little to clean them up and make sure they work as expected and verify
the builds that are failing are not due to bugs in the autobuild-run tweaks
themselves.

They don't usually get looked at for a while in patchwork so I haven't really
been bringing them up on the mailing list until after I've tested/deployed
them myself for a little, I have a couple pending in patchwork at the moment
for autobuild-run for example:
https://patchwork.ozlabs.org/project/buildroot/patch/20220607231211.533086-1-james.hilliard1@gmail.com/
https://patchwork.ozlabs.org/project/buildroot/patch/20220718043901.873466-1-james.hilliard1@gmail.com/

Sometimes I override genrandconfig as well to test patches such as:
https://patchwork.ozlabs.org/project/buildroot/patch/20220514214612.3221647-1-james.hilliard1@gmail.com/

I have a local patch to autobuild-run that redirects to an out of tree
genrandconfig so that I can test tweaks there. I also have it validating
that the in tree version of genrandconfig hasn't changed by comparing
against a known sha256 hash before using the out of tree tweaked version
to ensure that autobuild-run automatically falls back to using the in-tree
genrandconfig if genrandconfig gets changed/fixes in upstream buildroot.

For autobuild-run changes that increase error rates significantly due to
higher rates of bugs which need fixes in buildroot I sometimes have held off
on sending them as patches until the rates of errors triggered get low enough
that it would make sense for others to use those changes as well.

In this case I was also holding off on a patch automatically enabling shuffle
mode to autobuild-run as using that feature currently requires running a yet
to be released version of make and was triggering a few issues in buildroot
that needed fixing on the buildroot side.

I also figured that by testing master make using the autobuilders we can
get make issues fixed upstream before the next make release, such as
this shuffle mode bug so far:
https://savannah.gnu.org/bugs/index.php?63215

What sort of autobuilder changes would you like me to discuss on the mailing
list before deploying? I tend to experiment a bit but am usually pretty careful
to make sure that my autobuilder tweaks only cause build failures due to real
bugs that need fixing in buildroot/buildroot packages.

>
> > Well builds were mostly failing in the same place due to deterministic
> > build ordering, I was trying to make it so that we would see failures
> > in different places as that's more useful compared with always seeing
> > the same errors.
> >
> > To me seeing the same errors most of the time indicates there's some
> > major coverage issues with the autobuilders which need fixing.
>
> I think you're missing a point here: many of the builds today are
> failing due to the glibc 2.36 bump. So pretty much every single build
> that is glibc based will indeed fail with some error related to this
> glibc bump.

Probably a lot were failing for a couple days due to this glibc bug:
https://github.com/buildroot/buildroot/commit/b749d6dd7b5fc6cc8b85a316878721d42654759c

There's probably some failures still due to this issue but most of the others
seem to be dependency bugs in package makefiles:
https://patchwork.ozlabs.org/project/buildroot/patch/20221016193014.3384022-1-james.hilliard1@gmail.com/

However I'm actually referring to a separate issue with test coverage
that has always been present.

>
> Your additional randomization will not change anything to this. It's
> just that we now have the glibc bump issues + the newly discovered
> issues related to the extra randomization + the usual amount of build
> issues.

When I brought patch up this on irc:
https://patchwork.ozlabs.org/project/buildroot/patch/20220514214612.3221647-1-james.hilliard1@gmail.com/

arnout mentioned:
2022-07-25.log-[03:28:02] <arnout> Lightsword: about the build time: I
think the issue is that if you increase the number of selected
packages, you also reduce the number of successful builds (because it
fails as soon as one failing package is selected) - and that way,
packages that get built late (i.e. that come late in the alphabet and
have no reverse dependencies) don't get built.

This is indeed the case when I tested increasing the probability further,
however by randomizing the dependency ordering using make 4.4's new
shuffle mode we should be able to mitigate the alphabetical ordering bias
issue and thus get better test coverage of packages with larger dependency
trees by increasing the number of selected packages while using shuffle mode.

Looking at the current autobuilder failure logs I'm seeing that most
autobuilders
other than my own have much lower build failure rates and not much variety
in terms of failure reasons due to poor coverage of packages with larger
dependency trees.

For my own autobuilders running with shuffle mode I'm seeing much higher
failure rates for different reasons so these changes do appear to be
increasing out test coverage overall, there's a lot of issues with makefile
dependency ordering that are new and I suspect once those are fixed
we'll be able to get much better overall coverage of more common issues,
especially issues hit when running parallel builds.

>
> Best regards,
>
> Thomas
> --
> Thomas Petazzoni, co-owner and CEO, Bootlin
> Embedded Linux and Kernel engineering and training
> https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/bearssl: disable parallel build

[Thomas Petazzoni via buildroot]

Hello James,

On Fri, 14 Oct 2022 18:58:13 -0600
James Hilliard <james.hilliard1@gmail.com> wrote:

> The bearssl package is not compatible with parallel builds.
> 
> Fixes:
>  - http://autobuild.buildroot.net/results/5e1/5e1b72db206b2d17fa5ce378a70fdc2f3f5bf773
> 
> Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
> ---
>  package/bearssl/bearssl.mk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Based on the feedback which said that fixing bearssl was preferred over
using $(MAKE1), I have marked this patch as Changes Requested. Could
you see at fixing the bearssl build system, submit the fix upstream,
and provide the fix for Buildroot?

Thanks a lot!

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/bearssl: disable parallel build

[James Hilliard]

On Sun, Oct 30, 2022 at 4:29 PM Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello James,
>
> On Fri, 14 Oct 2022 18:58:13 -0600
> James Hilliard <james.hilliard1@gmail.com> wrote:
>
> > The bearssl package is not compatible with parallel builds.
> >
> > Fixes:
> >  - http://autobuild.buildroot.net/results/5e1/5e1b72db206b2d17fa5ce378a70fdc2f3f5bf773
> >
> > Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
> > ---
> >  package/bearssl/bearssl.mk | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
>
> Based on the feedback which said that fixing bearssl was preferred over
> using $(MAKE1), I have marked this patch as Changes Requested. Could
> you see at fixing the bearssl build system, submit the fix upstream,
> and provide the fix for Buildroot?

I contacted the maintainer as described here https://bearssl.org/contrib.html
with the suggested change, so far I haven't received a response back.

>
> Thanks a lot!
>
> Thomas
> --
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot