perl makefile race - any make experts who can help?

perl makefile race - any make experts who can help?

[Richard Purdie]

Hi,

Occasionally we see perl do_install failures on the autobuilder. It
looks like some kind of make race. With the last failure I saved off
the build directory and have spent a lot of time staring at it but I
don't understand how what the logs say happened actually happened.

The build failure is here:

https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5315/steps/11/logs/stdio

and you can see it failed in do_install. The first error is;

| Couldn't copy cpan/podlators/blib/script/pod2text to /home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/core2-64-poky-linux/perl/5.34.1-r0/image/usr/bin/pod2text: No such file or directory
| Couldn't chmod 755 /home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/core2-64-poky-linux/perl/5.34.1-r0/image/usr/bin/pod2text: No such file or directory

then

installman: Can't open cpan/podlators/blib/script/pod2text: No such file or directory
| ABORTED

Looking at the build directory, cpan/podlators/blib/script/pod2text
isn't there, cpan/podlators/script/pod2text is. I went digging in the
compile log which didn't fail but is interesting in what it doesn't say
nd the ordering. The full log:

https://autobuilder.yocto.io/pub/failed-builds-data/perl-race/log.do_compile.25823

or my edited down version:

https://autobuilder.yocto.io/pub/failed-builds-data/perl-race/compile-cutdown.log

and note it never actually builds cpan/podlators/blib/script/pod2text.

If I go into the failed build dir, it does build it:

https://autobuilder.yocto.io/pub/failed-builds-data/perl-race/manual-compile.log

I tried a few variations of deleting and rebuilding files there just to show behaviour.

The makefiles from the podlators directory:

https://autobuilder.yocto.io/pub/failed-builds-data/perl-race/Makefile
https://autobuilder.yocto.io/pub/failed-builds-data/perl-race/Makefile.PL


What really puzzles me is the "Manifying 2 pod documents" message
*before* it then generates scripts/pod2text from scripts/pod2text.PL in
the do_compile log. It appears to run the scripts/pod2text late and 
never runs the $(INST_SCRIPT)/pod2text at all (hence no cp message and
the missing file).

The Makefile has default targets of pure_all and manifypods, that
message comes from manifypods but manifypods depends on pure_all.

pure_all is a double colon rule and I'm on less certain ground with how
those behave.

Are there any make experts out there who can spot the race in this
makefile?

Cheers,

Richard

Re: [OE-core] perl makefile race - any make experts who can help?

[Alexander Kanavin]

Here's something I didn't think of before. Has this occurred anywhere
else except Ubuntu 18.04?

Alex

On Sat, 4 Jun 2022 at 13:07, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> Hi,
>
> Occasionally we see perl do_install failures on the autobuilder. It
> looks like some kind of make race. With the last failure I saved off
> the build directory and have spent a lot of time staring at it but I
> don't understand how what the logs say happened actually happened.
>
> The build failure is here:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/5315/steps/11/logs/stdio
>
> and you can see it failed in do_install. The first error is;
>
> | Couldn't copy cpan/podlators/blib/script/pod2text to /home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/core2-64-poky-linux/perl/5.34.1-r0/image/usr/bin/pod2text: No such file or directory
> | Couldn't chmod 755 /home/pokybuild/yocto-worker/genericx86-64/build/build/tmp/work/core2-64-poky-linux/perl/5.34.1-r0/image/usr/bin/pod2text: No such file or directory
>
> then
>
> installman: Can't open cpan/podlators/blib/script/pod2text: No such file or directory
> | ABORTED
>
> Looking at the build directory, cpan/podlators/blib/script/pod2text
> isn't there, cpan/podlators/script/pod2text is. I went digging in the
> compile log which didn't fail but is interesting in what it doesn't say
> nd the ordering. The full log:
>
> https://autobuilder.yocto.io/pub/failed-builds-data/perl-race/log.do_compile.25823
>
> or my edited down version:
>
> https://autobuilder.yocto.io/pub/failed-builds-data/perl-race/compile-cutdown.log
>
> and note it never actually builds cpan/podlators/blib/script/pod2text.
>
> If I go into the failed build dir, it does build it:
>
> https://autobuilder.yocto.io/pub/failed-builds-data/perl-race/manual-compile.log
>
> I tried a few variations of deleting and rebuilding files there just to show behaviour.
>
> The makefiles from the podlators directory:
>
> https://autobuilder.yocto.io/pub/failed-builds-data/perl-race/Makefile
> https://autobuilder.yocto.io/pub/failed-builds-data/perl-race/Makefile.PL
>
>
> What really puzzles me is the "Manifying 2 pod documents" message
> *before* it then generates scripts/pod2text from scripts/pod2text.PL in
> the do_compile log. It appears to run the scripts/pod2text late and
> never runs the $(INST_SCRIPT)/pod2text at all (hence no cp message and
> the missing file).
>
> The Makefile has default targets of pure_all and manifypods, that
> message comes from manifypods but manifypods depends on pure_all.
>
> pure_all is a double colon rule and I'm on less certain ground with how
> those behave.
>
> Are there any make experts out there who can spot the race in this
> makefile?
>
> Cheers,
>
> Richard
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#166564): https://lists.openembedded.org/g/openembedded-core/message/166564
> Mute This Topic: https://lists.openembedded.org/mt/91538663/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] perl makefile race - any make experts who can help?

[Richard Purdie]

On Sat, 2022-06-04 at 13:51 +0200, Alexander Kanavin wrote:
> Here's something I didn't think of before. Has this occurred anywhere
> else except Ubuntu 18.04?

https://bugzilla.yoctoproject.org/show_bug.cgi?id=14096

I'm struggling to get the data out from the old builds, one mentions
ubuntu1604, there is an ubuntu1804 on both x86 and arm hosts.

It is possible this is an ubuntu specific make issue or a make bug.

Cheers,

Richard

Re: [swat] [OE-core] perl makefile race - any make experts who can help?

[Richard Purdie]

On Sat, 2022-06-04 at 13:36 +0100, Richard Purdie via
lists.yoctoproject.org wrote:
> On Sat, 2022-06-04 at 13:51 +0200, Alexander Kanavin wrote:
> > Here's something I didn't think of before. Has this occurred anywhere
> > else except Ubuntu 18.04?
> 
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=14096
> 
> I'm struggling to get the data out from the old builds, one mentions
> ubuntu1604, there is an ubuntu1804 on both x86 and arm hosts.
> 
> It is possible this is an ubuntu specific make issue or a make bug.

Ubuntu 18.04 uses make 4.1 which is old (Oct 2014).

I noticed these patches from 2016:

https://git.savannah.gnu.org/cgit/make.git/commit/?id=9bb994e8319c2b153cd3d6d61e2c2882895e7c3a
https://git.savannah.gnu.org/cgit/make.git/commit/?id=4762480ae9cb8df4878286411f178d32db14eff0

I think we may want to mandate a modern make for both this class of
issues and also perhaps for better loadavg support to keep load under
control on the autobuilders.

I'm torn, on the one hand we need to test the distros people use, on
the other we do need to remove sources of intermittent issues. I think
this bug must be some issue with make itself.

Adding a make-native dependency to perl would "hurt" people on modern
distros...

Cheers,

Richard

Re: [swat] [OE-core] perl makefile race - any make experts who can help?

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 2018 bytes --]

On Sat, Jun 4, 2022 at 6:23 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Sat, 2022-06-04 at 13:36 +0100, Richard Purdie via
> lists.yoctoproject.org wrote:
> > On Sat, 2022-06-04 at 13:51 +0200, Alexander Kanavin wrote:
> > > Here's something I didn't think of before. Has this occurred anywhere
> > > else except Ubuntu 18.04?
> >
> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=14096
> >
> > I'm struggling to get the data out from the old builds, one mentions
> > ubuntu1604, there is an ubuntu1804 on both x86 and arm hosts.
> >
> > It is possible this is an ubuntu specific make issue or a make bug.
>
> Ubuntu 18.04 uses make 4.1 which is old (Oct 2014).
>
> I noticed these patches from 2016:
>
>
> https://git.savannah.gnu.org/cgit/make.git/commit/?id=9bb994e8319c2b153cd3d6d61e2c2882895e7c3a
>
> https://git.savannah.gnu.org/cgit/make.git/commit/?id=4762480ae9cb8df4878286411f178d32db14eff0
>
> I think we may want to mandate a modern make for both this class of
> issues and also perhaps for better loadavg support to keep load under
> control on the autobuilders.
>
> I'm torn, on the one hand we need to test the distros people use, on
> the other we do need to remove sources of intermittent issues. I think
> this bug must be some issue with make itself.
>
> Adding a make-native dependency to perl would "hurt" people on modern
> distros...


Make perhaps does not have many complex dependency needs so it might not be
as bad

Second option is to mandate buildtools on this distro
And add make to tarball

>
>
> Cheers,
>
> Richard
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#166567):
> https://lists.openembedded.org/g/openembedded-core/message/166567
> Mute This Topic: https://lists.openembedded.org/mt/91540379/1997914
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 3658 bytes --]

Re: [swat] [OE-core] perl makefile race - any make experts who can help?

[Jacob Kroon]

On 6/4/22 16:55, Khem Raj wrote:
> 
> 
> On Sat, Jun 4, 2022 at 6:23 AM Richard Purdie 
> <richard.purdie@linuxfoundation.org 
> <mailto:richard.purdie@linuxfoundation.org>> wrote:
> 
>     On Sat, 2022-06-04 at 13:36 +0100, Richard Purdie via
>     lists.yoctoproject.org <http://lists.yoctoproject.org> wrote:
>      > On Sat, 2022-06-04 at 13:51 +0200, Alexander Kanavin wrote:
>      > > Here's something I didn't think of before. Has this occurred
>     anywhere
>      > > else except Ubuntu 18.04?
>      >
>      > https://bugzilla.yoctoproject.org/show_bug.cgi?id=14096
>     <https://bugzilla.yoctoproject.org/show_bug.cgi?id=14096>
>      >
>      > I'm struggling to get the data out from the old builds, one mentions
>      > ubuntu1604, there is an ubuntu1804 on both x86 and arm hosts.
>      >
>      > It is possible this is an ubuntu specific make issue or a make bug.
> 
>     Ubuntu 18.04 uses make 4.1 which is old (Oct 2014).
> 
>     I noticed these patches from 2016:
> 
>     https://git.savannah.gnu.org/cgit/make.git/commit/?id=9bb994e8319c2b153cd3d6d61e2c2882895e7c3a
>     <https://git.savannah.gnu.org/cgit/make.git/commit/?id=9bb994e8319c2b153cd3d6d61e2c2882895e7c3a>
>     https://git.savannah.gnu.org/cgit/make.git/commit/?id=4762480ae9cb8df4878286411f178d32db14eff0
>     <https://git.savannah.gnu.org/cgit/make.git/commit/?id=4762480ae9cb8df4878286411f178d32db14eff0>
> 
>     I think we may want to mandate a modern make for both this class of
>     issues and also perhaps for better loadavg support to keep load under
>     control on the autobuilders.
> 
>     I'm torn, on the one hand we need to test the distros people use, on
>     the other we do need to remove sources of intermittent issues. I think
>     this bug must be some issue with make itself.
> 
>     Adding a make-native dependency to perl would "hurt" people on modern
>     distros...
> 
> 
> Make perhaps does not have many complex dependency needs so it might not 
> be as bad
> 

My master build is already building make-native due to a dependency from 
glibc, since 2018:

https://git.openembedded.org/openembedded-core/commit/?id=0cd89e4af625941f8ab8c033f72f900a2979b304

Don't know if that dependency is still valid though.

Jacob

Re: [swat] [OE-core] perl makefile race - any make experts who can help?

[Richard Purdie]

On Sat, 2022-06-04 at 17:12 +0200, Jacob Kroon wrote:
> On 6/4/22 16:55, Khem Raj wrote:
> > 
> > 
> > On Sat, Jun 4, 2022 at 6:23 AM Richard Purdie 
> > <richard.purdie@linuxfoundation.org 
> > <mailto:richard.purdie@linuxfoundation.org>> wrote:
> > 
> >     On Sat, 2022-06-04 at 13:36 +0100, Richard Purdie via
> >     lists.yoctoproject.org <http://lists.yoctoproject.org> wrote:
> >      > On Sat, 2022-06-04 at 13:51 +0200, Alexander Kanavin wrote:
> >      > > Here's something I didn't think of before. Has this occurred
> >     anywhere
> >      > > else except Ubuntu 18.04?
> >      >
> >      > https://bugzilla.yoctoproject.org/show_bug.cgi?id=14096
> >     <https://bugzilla.yoctoproject.org/show_bug.cgi?id=14096>
> >      >
> >      > I'm struggling to get the data out from the old builds, one mentions
> >      > ubuntu1604, there is an ubuntu1804 on both x86 and arm hosts.
> >      >
> >      > It is possible this is an ubuntu specific make issue or a make bug.
> > 
> >     Ubuntu 18.04 uses make 4.1 which is old (Oct 2014).
> > 
> >     I noticed these patches from 2016:
> > 
> >     https://git.savannah.gnu.org/cgit/make.git/commit/?id=9bb994e8319c2b153cd3d6d61e2c2882895e7c3a
> >     <https://git.savannah.gnu.org/cgit/make.git/commit/?id=9bb994e8319c2b153cd3d6d61e2c2882895e7c3a>
> >     https://git.savannah.gnu.org/cgit/make.git/commit/?id=4762480ae9cb8df4878286411f178d32db14eff0
> >     <https://git.savannah.gnu.org/cgit/make.git/commit/?id=4762480ae9cb8df4878286411f178d32db14eff0>
> > 
> >     I think we may want to mandate a modern make for both this class of
> >     issues and also perhaps for better loadavg support to keep load under
> >     control on the autobuilders.
> > 
> >     I'm torn, on the one hand we need to test the distros people use, on
> >     the other we do need to remove sources of intermittent issues. I think
> >     this bug must be some issue with make itself.
> > 
> >     Adding a make-native dependency to perl would "hurt" people on modern
> >     distros...
> > 
> > 
> > Make perhaps does not have many complex dependency needs so it might not 
> > be as bad
> > 
> 
> My master build is already building make-native due to a dependency from 
> glibc, since 2018:
> 
> https://git.openembedded.org/openembedded-core/commit/?id=0cd89e4af625941f8ab8c033f72f900a2979b304
> 
> Don't know if that dependency is still valid though.

It is a fair point. We may as well add it to perl/perl-native. Centos7
still has make 3.82 but I think we now already require buildtools
tarball there so we could probably drop the glibc dependency on make-
native now.

Cheers,

Richard

Re: [swat] [OE-core] perl makefile race - any make experts who can help?

[Jacob Kroon]

[-- Attachment #1: Type: text/plain, Size: 3005 bytes --]

On Sat, 4 Jun 2022, 19:40 Richard Purdie, <
richard.purdie@linuxfoundation.org> wrote:

> On Sat, 2022-06-04 at 17:12 +0200, Jacob Kroon wrote:
> > On 6/4/22 16:55, Khem Raj wrote:
> > >
> > >
> > > On Sat, Jun 4, 2022 at 6:23 AM Richard Purdie
> > > <richard.purdie@linuxfoundation.org
> > > <mailto:richard.purdie@linuxfoundation.org>> wrote:
> > >
> > >     On Sat, 2022-06-04 at 13:36 +0100, Richard Purdie via
> > >     lists.yoctoproject.org <http://lists.yoctoproject.org> wrote:
> > >      > On Sat, 2022-06-04 at 13:51 +0200, Alexander Kanavin wrote:
> > >      > > Here's something I didn't think of before. Has this occurred
> > >     anywhere
> > >      > > else except Ubuntu 18.04?
> > >      >
> > >      > https://bugzilla.yoctoproject.org/show_bug.cgi?id=14096
> > >     <https://bugzilla.yoctoproject.org/show_bug.cgi?id=14096>
> > >      >
> > >      > I'm struggling to get the data out from the old builds, one
> mentions
> > >      > ubuntu1604, there is an ubuntu1804 on both x86 and arm hosts.
> > >      >
> > >      > It is possible this is an ubuntu specific make issue or a make
> bug.
> > >
> > >     Ubuntu 18.04 uses make 4.1 which is old (Oct 2014).
> > >
> > >     I noticed these patches from 2016:
> > >
> > >
> https://git.savannah.gnu.org/cgit/make.git/commit/?id=9bb994e8319c2b153cd3d6d61e2c2882895e7c3a
> > >     <
> https://git.savannah.gnu.org/cgit/make.git/commit/?id=9bb994e8319c2b153cd3d6d61e2c2882895e7c3a
> >
> > >
> https://git.savannah.gnu.org/cgit/make.git/commit/?id=4762480ae9cb8df4878286411f178d32db14eff0
> > >     <
> https://git.savannah.gnu.org/cgit/make.git/commit/?id=4762480ae9cb8df4878286411f178d32db14eff0
> >
> > >
> > >     I think we may want to mandate a modern make for both this class of
> > >     issues and also perhaps for better loadavg support to keep load
> under
> > >     control on the autobuilders.
> > >
> > >     I'm torn, on the one hand we need to test the distros people use,
> on
> > >     the other we do need to remove sources of intermittent issues. I
> think
> > >     this bug must be some issue with make itself.
> > >
> > >     Adding a make-native dependency to perl would "hurt" people on
> modern
> > >     distros...
> > >
> > >
> > > Make perhaps does not have many complex dependency needs so it might
> not
> > > be as bad
> > >
> >
> > My master build is already building make-native due to a dependency from
> > glibc, since 2018:
> >
> >
> https://git.openembedded.org/openembedded-core/commit/?id=0cd89e4af625941f8ab8c033f72f900a2979b304
> >
> > Don't know if that dependency is still valid though.
>
> It is a fair point. We may as well add it to perl/perl-native. Centos7
> still has make 3.82 but I think we now already require buildtools
> tarball there so we could probably drop the glibc dependency on make-
> native now.
>

Would it be a bad idea to add make-native to DEPENDS depending on whether
the host version of make is new enough or not ? Would it break sstate cache
reuse in some way ?

>

[-- Attachment #2: Type: text/html, Size: 5672 bytes --]

Re: [swat] [OE-core] perl makefile race - any make experts who can help?

[Richard Purdie]

On Sun, 2022-06-05 at 04:26 +0200, Jacob Kroon wrote:
> On Sat, 4 Jun 2022, 19:40 Richard Purdie,
> <richard.purdie@linuxfoundation.org> wrote:
> > On Sat, 2022-06-04 at 17:12 +0200, Jacob Kroon wrote:
> > > On 6/4/22 16:55, Khem Raj wrote:
> > > > 
> > > > 
> > > > On Sat, Jun 4, 2022 at 6:23 AM Richard Purdie 
> > > > <richard.purdie@linuxfoundation.org 
> > > > <mailto:richard.purdie@linuxfoundation.org>> wrote:
> > > > 
> > > >      On Sat, 2022-06-04 at 13:36 +0100, Richard Purdie via
> > > >      lists.yoctoproject.org <http://lists.yoctoproject.org>
> > > > wrote:
> > > >       > On Sat, 2022-06-04 at 13:51 +0200, Alexander Kanavin
> > > > wrote:
> > > >       > > Here's something I didn't think of before. Has this
> > > > occurred
> > > >      anywhere
> > > >       > > else except Ubuntu 18.04?
> > > >       >
> > > >       > https://bugzilla.yoctoproject.org/show_bug.cgi?id=14096
> > > >      <https://bugzilla.yoctoproject.org/show_bug.cgi?id=14096>
> > > >       >
> > > >       > I'm struggling to get the data out from the old builds,
> > > > one mentions
> > > >       > ubuntu1604, there is an ubuntu1804 on both x86 and arm
> > > > hosts.
> > > >       >
> > > >       > It is possible this is an ubuntu specific make issue or
> > > > a make bug.
> > > > 
> > > >      Ubuntu 18.04 uses make 4.1 which is old (Oct 2014).
> > > > 
> > > >      I noticed these patches from 2016:
> > > > 
> > > >    
> > > >  https://git.savannah.gnu.org/cgit/make.git/commit/?id=9bb994e8
> > > > 319c2b153cd3d6d61e2c2882895e7c3a
> > > >    
> > > >  <https://git.savannah.gnu.org/cgit/make.git/commit/?id=9bb994e
> > > > 8319c2b153cd3d6d61e2c2882895e7c3a>
> > > >    
> > > >  https://git.savannah.gnu.org/cgit/make.git/commit/?id=4762480a
> > > > e9cb8df4878286411f178d32db14eff0
> > > >    
> > > >  <https://git.savannah.gnu.org/cgit/make.git/commit/?id=4762480
> > > > ae9cb8df4878286411f178d32db14eff0>
> > > > 
> > > >      I think we may want to mandate a modern make for both this
> > > > class of
> > > >      issues and also perhaps for better loadavg support to keep
> > > > load under
> > > >      control on the autobuilders.
> > > > 
> > > >      I'm torn, on the one hand we need to test the distros
> > > > people use, on
> > > >      the other we do need to remove sources of intermittent
> > > > issues. I think
> > > >      this bug must be some issue with make itself.
> > > > 
> > > >      Adding a make-native dependency to perl would "hurt"
> > > > people on modern
> > > >      distros...
> > > > 
> > > > 
> > > > Make perhaps does not have many complex dependency needs so it
> > > > might not 
> > > > be as bad
> > > > 
> > > 
> > > My master build is already building make-native due to a
> > > dependency from 
> > > glibc, since 2018:
> > > 
> > > https://git.openembedded.org/openembedded-core/commit/?id=0cd89e4af625941f8ab8c033f72f900a2979b304
> > > 
> > > Don't know if that dependency is still valid though.
> > 
> > It is a fair point. We may as well add it to perl/perl-native.
> > Centos7
> > still has make 3.82 but I think we now already require buildtools
> > tarball there so we could probably drop the glibc dependency on
> > make-
> > native now.
> 
> 
> Would it be a bad idea to add make-native to DEPENDS depending on
> whether the host version of make is new enough or not ? Would it
> break sstate cache reuse in some way ? 

We can't have a conditional dependency like that, the task checksums as
implemented today wouldn't work and it would break ssttate reuse.

Cheers,

Richard