Re: [Buildroot] [PATCH 1/1] package/bearssl: disable parallel build

["Yann E. MORIN" <yann.morin.1998@free.fr>]

James, All,

+Thomas, Peter, Arnout

On 2022-10-16 11:44 -0400, James Hilliard spake thusly:
> On Sun, Oct 16, 2022 at 3:51 AM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> > On 2022-10-15 19:21 -0400, James Hilliard spake thusly:
[--SNIP--]
> > > Yes, all my autobuilders were recently changed to use make master(will be
> > > make version 4.4 once released) with top level --shuffle=random.
> > By disabling parallel build because you are using ana as yet unreleased
> > version, we remove the parallel build for everyone, which is not very
> > nice at all.
> Failures with shuffle mode indicate that the package is not parallel compatible
> on existing make releases as well, it just makes those bugs more visible.

I never said they were not, and I even said (further on) that it *is*
interesting to find such bugs.

However, the behaviour so far has been ordering of prerequisites (and
goals!), and for some packages, that ordering was enough to work in a
parallel make.

For example, the bearsll parallel issue is caused by the ordering so
the mkdir is very early. Yes, it is incorrect, but it happens to always
work in practice. And yes it is easy to fix, here's a patch:

    diff -durN a/mk/mkrules.sh b/mk/mkrules.sh
    --- a/mk/mkrules.sh	2018-08-14 22:41:54.000000000 +0200
    +++ b/mk/mkrules.sh	2022-10-16 17:54:22.282069851 +0200
    @@ -531,23 +531,23 @@
     (for f in $coresrc ; do
     	b="$(basename "$f" .c)\$O"
     	g="$(escsep "$f")"
    -	printf '\n$(OBJDIR)$P%s: %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
    +	printf '\n$(OBJDIR)$P%s: $(OBJDIR) %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
     done
     
     for f in $toolssrc ; do
     	b="$(basename "$f" .c)\$O"
     	g="$(escsep "$f")"
    -	printf '\n$(OBJDIR)$P%s: %s $(HEADERSTOOLS)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
    +	printf '\n$(OBJDIR)$P%s: $(OBJDIR) %s $(HEADERSTOOLS)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
     done
     
     for f in $testcryptosrc $testspeedsrc ; do
     	b="$(basename "$f" .c)\$O"
     	g="$(escsep "$f")"
    -	printf '\n$(OBJDIR)$P%s: %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
    +	printf '\n$(OBJDIR)$P%s: $(OBJDIR) %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
     done
     
     for f in $testx509src ; do
     	b="$(basename "$f" .c)\$O"
     	g="$(escsep "$f")"
    -	printf '\n$(OBJDIR)$P%s: %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) -DSRCDIRNAME=".." $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
    +	printf '\n$(OBJDIR)$P%s: $(OBJDIR) %s $(HEADERSPRIV)\n\t$(CC) $(CFLAGS) $(INCFLAGS) -DSRCDIRNAME=".." $(CCOUT)$(OBJDIR)$P%s %s\n' "$b" "$g" "$b" "$g"
     done) >> Rules.mk
    Binary files a/mk/.mkrules.sh.swp and b/mk/.mkrules.sh.swp differ

And then adding a CONFIGURE_CMDS that runs mk/mkrules.sh to regenerate
the Makefile.

> I'm running shuffle with -j1 because that way it's obvious in the logs where the
> failure is getting hit, otherwise these bugs tend to be transient and
> difficult to
> trace as they otherwise are timing dependent.

Yes, this is very interesting to find such bugs, but disabling parallel
build in packages is not nice when they can be (easily) fixed.

> > Notes:
> >   - it *is* interesting to find those bugs;
> >   - it is sad that the default mode for --shuffle will be random, not
> >     none.
> Default is equivalent to none,

Yeah, I noticed the hard way after having my patch rejected in makem as
you may have noticed. :-/

> I modified my autobuilder runner to pass
> --shuffle=random.

Again, that is very interesting, but please, please, conordinate with
the rest of us when you do such changes, because that is very weird, as
Fabrice noted, to suddenly get build failures out of the blue when
nothing seemed to have otherwise changed.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot