The coreutils-native race...

The coreutils-native race...

[Tom Rini]

Hey all.  I thought I would try and explain what Chris has been up to
with at least some of the base.bbclass changes (the ones related to
md5sum and cp).

Right now, with a big enough BB_NUM_THREADS we can get into a race where
coreutils-native is installing programs and elsewhere we are in a
do_fetch and either trying to use 'cp' or 'md5sum', and blam, we try and
invoke the program while it's being installed (and see things like
sh: /path/to/staging/i686-linux/usr/bin/cp: Textfile is busy).

There's a few ways out of this:
1) Don't rely on 'cp' and 'md5sum' anymore but use python for it.
2) Make an oe_cp and oe_md5sum to go with oe_sha256sum
3) IIRC, the big part of coreutils-native was a fully functional,
always, 'install'.  We could just copy the install we build or provide
an install wrapper (oe_install) or so
4) ???

Thoughts? Comments?

-- 
Tom Rini <tom_rini@mentor.com>
Mentor Graphics Corporation

Re: The coreutils-native race...

[Phil Blundell]

On Sun, 2010-04-25 at 10:48 -0700, Tom Rini wrote:
> Hey all.  I thought I would try and explain what Chris has been up to
> There's a few ways out of this:
> 1) Don't rely on 'cp' and 'md5sum' anymore but use python for it.
> 2) Make an oe_cp and oe_md5sum to go with oe_sha256sum
> 3) IIRC, the big part of coreutils-native was a fully functional,
> always, 'install'.  We could just copy the install we build or provide
> an install wrapper (oe_install) or so
> 4) ???

How about making the coreutils-native staging process be atomic, i.e.
install binaries as cp.new (or whatever) and then do "mv cp.new cp"?
Or, more generally I guess, arrange for new-style staging to do that for
all binaries it installs.

If (3) is sufficient then it seems like we might just as well exhume the
old install-native package and ditch coreutils-native altogether.  That
would avoid the race since it doesn't install anything other than
install.

p.

Re: The coreutils-native race...

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 25-04-10 19:48, Tom Rini wrote:
> Hey all.  I thought I would try and explain what Chris has been up to
> with at least some of the base.bbclass changes (the ones related to
> md5sum and cp).
> 
> Right now, with a big enough BB_NUM_THREADS we can get into a race where
> coreutils-native is installing programs and elsewhere we are in a
> do_fetch and either trying to use 'cp' or 'md5sum', and blam, we try and
> invoke the program while it's being installed (and see things like
> sh: /path/to/staging/i686-linux/usr/bin/cp: Textfile is busy).
> 
> There's a few ways out of this:
> 1) Don't rely on 'cp' and 'md5sum' anymore but use python for it.
> 2) Make an oe_cp and oe_md5sum to go with oe_sha256sum
> 3) IIRC, the big part of coreutils-native was a fully functional,
> always, 'install'.  We could just copy the install we build or provide
> an install wrapper (oe_install) or so
> 4) ???

Even thought I loathe python, option 1 sounds like a nice way to go
since it doesn't involve butchering recipes.

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFL1JfMMkyGM64RGpERAncaAJ41plrV7jSmlXFUa5AY1VXzIulpggCfeWsr
+7/jI/XS67azRPdGlJJLmYo=
=vq5J
-----END PGP SIGNATURE-----

Re: The coreutils-native race...

[Tom Rini]

On Sun, 2010-04-25 at 21:28 +0200, Koen Kooi wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 25-04-10 19:48, Tom Rini wrote:
> > Hey all.  I thought I would try and explain what Chris has been up to
> > with at least some of the base.bbclass changes (the ones related to
> > md5sum and cp).
> > 
> > Right now, with a big enough BB_NUM_THREADS we can get into a race where
> > coreutils-native is installing programs and elsewhere we are in a
> > do_fetch and either trying to use 'cp' or 'md5sum', and blam, we try and
> > invoke the program while it's being installed (and see things like
> > sh: /path/to/staging/i686-linux/usr/bin/cp: Textfile is busy).
> > 
> > There's a few ways out of this:
> > 1) Don't rely on 'cp' and 'md5sum' anymore but use python for it.
> > 2) Make an oe_cp and oe_md5sum to go with oe_sha256sum
> > 3) IIRC, the big part of coreutils-native was a fully functional,
> > always, 'install'.  We could just copy the install we build or provide
> > an install wrapper (oe_install) or so
> > 4) ???
> 
> Even thought I loathe python, option 1 sounds like a nice way to go
> since it doesn't involve butchering recipes.

I think we're OK, except in the case of when we can't rely on a built-in
md5sum (and then the race comes back) and just need to figure out if the
libpam-base-files recipe does things (a) optimally and (b) if that's a
common thing.

Currently, with a quick glance at the recipe, I don't see any
distro-specific overrides, so I'd be inclined to call it lazy recipe
writing, until someone points out history of why that's a good thing..

-- 
Tom Rini <tom_rini@mentor.com>
Mentor Graphics Corporation

Re: The coreutils-native race...

[Roman I Khimov]

[-- Attachment #1: Type: Text/Plain, Size: 2327 bytes --]

В сообщении от Воскресенье 25 апреля 2010 21:48:59 автор Tom Rini написал:
> Right now, with a big enough BB_NUM_THREADS we can get into a race where
> coreutils-native is installing programs and elsewhere we are in a
> do_fetch and either trying to use 'cp' or 'md5sum', and blam, we try and
> invoke the program while it's being installed (and see things like
> sh: /path/to/staging/i686-linux/usr/bin/cp: Textfile is busy).
> 
> There's a few ways out of this:
> 1) Don't rely on 'cp' and 'md5sum' anymore but use python for it.

I guess it couldn't really be done, since those could be invoked from any 
shell script, right? Although it only is a concern for things before 
do_configure, so there shouldn't be many scripts there.

> 2) Make an oe_cp and oe_md5sum to go with oe_sha256sum
> 3) IIRC, the big part of coreutils-native was a fully functional,
> always, 'install'.  We could just copy the install we build or provide
> an install wrapper (oe_install) or so

We've had install-native, but that was some broken dusty script that act like 
a detonator - you build some package having it in DEPENDS (not many had, but I 
was lucky enough) and boom! random packages brake randomly in do_install 
because of disfunctional install.

I've proposed back then to just drop it. But there've been a concern that for 
reliable builds it's better to be sure that we have known proper install and 
that is provided by coreutils along with many other things, which are also 
good to have in some known good state for an OS-independent build system.

This is the way we've got to the point of having coreutils-native in all of 
our builds.

Although, as I'm personally building my things on sane GNU hosts with sane 
coreutils with known nice install, I've always had coreutils-native in 
ASSUME_PROVIDED. Thus, not seeing any failures with usual 4 bb threads.

I guess, if we can solve it with python, it's the best option. If not, then we 
can try to remove coreutils-native from base dependencies and see how much 
breakage it introduces. Shouldn't be a problem for most of Linux distros, but 
for other platforms it may be.

-- 
 http://roman.khimov.ru
mailto: roman@khimov.ru
gpg --keyserver hkp://subkeys.pgp.net --recv-keys 0xE5E055C3

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: The coreutils-native race...

[Tom Rini]

On Mon, 2010-04-26 at 00:42 +0400, Roman I Khimov wrote:
> В сообщении от Воскресенье 25 апреля 2010 21:48:59 автор Tom Rini написал:
> > Right now, with a big enough BB_NUM_THREADS we can get into a race where
> > coreutils-native is installing programs and elsewhere we are in a
> > do_fetch and either trying to use 'cp' or 'md5sum', and blam, we try and
> > invoke the program while it's being installed (and see things like
> > sh: /path/to/staging/i686-linux/usr/bin/cp: Textfile is busy).
> > 
> > There's a few ways out of this:
> > 1) Don't rely on 'cp' and 'md5sum' anymore but use python for it.
> 
> I guess it couldn't really be done, since those could be invoked from any 
> shell script, right? Although it only is a concern for things before 
> do_configure, so there shouldn't be many scripts there.

We control (as it's in bbclasses) what's invoked in these tasks that are
allowed to run before do_configure is invoked (the current choke point
the world has on coreutils-native being complete).

> > 2) Make an oe_cp and oe_md5sum to go with oe_sha256sum
> > 3) IIRC, the big part of coreutils-native was a fully functional,
> > always, 'install'.  We could just copy the install we build or provide
> > an install wrapper (oe_install) or so
> 
> We've had install-native, but that was some broken dusty script that act like 
> a detonator - you build some package having it in DEPENDS (not many had, but I 
> was lucky enough) and boom! random packages brake randomly in do_install 
> because of disfunctional install.

True, but we could just as easily replace coreutils-native with
oeinstall-native in base.bbclass.

> I've proposed back then to just drop it. But there've been a concern that for 
> reliable builds it's better to be sure that we have known proper install and 
> that is provided by coreutils along with many other things, which are also 
> good to have in some known good state for an OS-independent build system.
> 
> This is the way we've got to the point of having coreutils-native in all of 
> our builds.

I think I forgot #5
5) Make coreutils-native:do_populate_sysroot (or whatever fully staged
means at this point) before anything else is really allowed to go on.

> Although, as I'm personally building my things on sane GNU hosts with sane 
> coreutils with known nice install, I've always had coreutils-native in 
> ASSUME_PROVIDED. Thus, not seeing any failures with usual 4 bb threads.

This rarely happens with 4 threads here and is much more often at 6 or 8
threads.

-- 
Tom Rini <tom_rini@mentor.com>
Mentor Graphics Corporation

Re: The coreutils-native race...

[Yuri Bushmelev]

В сообщении от Понедельник 26 апреля 2010 Roman I Khimov написал(a):
> В сообщении от Воскресенье 25 апреля 2010 21:48:59 автор Tom Rini написал:

> I guess, if we can solve it with python, it's the best option. If not,
> then we can try to remove coreutils-native from base dependencies and
> see how much breakage it introduces. Shouldn't be a problem for most of
> Linux distros, but for other platforms it may be.

FreeBSD will be a first candidate :)

-- 
Yuri Bushmelev

Re: The coreutils-native race...

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 25-04-10 21:57, Tom Rini wrote:
> On Sun, 2010-04-25 at 21:28 +0200, Koen Kooi wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> On 25-04-10 19:48, Tom Rini wrote:
>>> Hey all.  I thought I would try and explain what Chris has been up to
>>> with at least some of the base.bbclass changes (the ones related to
>>> md5sum and cp).
>>>
>>> Right now, with a big enough BB_NUM_THREADS we can get into a race where
>>> coreutils-native is installing programs and elsewhere we are in a
>>> do_fetch and either trying to use 'cp' or 'md5sum', and blam, we try and
>>> invoke the program while it's being installed (and see things like
>>> sh: /path/to/staging/i686-linux/usr/bin/cp: Textfile is busy).
>>>
>>> There's a few ways out of this:
>>> 1) Don't rely on 'cp' and 'md5sum' anymore but use python for it.
>>> 2) Make an oe_cp and oe_md5sum to go with oe_sha256sum
>>> 3) IIRC, the big part of coreutils-native was a fully functional,
>>> always, 'install'.  We could just copy the install we build or provide
>>> an install wrapper (oe_install) or so
>>> 4) ???
>>
>> Even thought I loathe python, option 1 sounds like a nice way to go
>> since it doesn't involve butchering recipes.
> 
> I think we're OK, except in the case of when we can't rely on a built-in
> md5sum (and then the race comes back) and just need to figure out if the
> libpam-base-files recipe does things (a) optimally and (b) if that's a
> common thing.
> 
> Currently, with a quick glance at the recipe, I don't see any
> distro-specific overrides, so I'd be inclined to call it lazy recipe
> writing, until someone points out history of why that's a good thing..

I have no problem with changing the libpam-base-files SRC_URI to

file://pam.d/

or

file://foo.pam \
file://bar.pam \
...
etc

But I would like to see it documented in the OE manual that file://dir/*
can't be used, and why.

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFL1TMeMkyGM64RGpERApkGAJ97rdsQSb+HvBKP2xKUdyAvqyfqPgCfUEBG
h/0qC3TUbMrfO0oOvZv9mmk=
=cb0z
-----END PGP SIGNATURE-----

Re: The coreutils-native race...

[Chris Larson]

On Sun, Apr 25, 2010 at 11:30 PM, Koen Kooi <k.kooi@student.utwente.nl>wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 25-04-10 21:57, Tom Rini wrote:
> > On Sun, 2010-04-25 at 21:28 +0200, Koen Kooi wrote:
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA1
> >>
> >> On 25-04-10 19:48, Tom Rini wrote:
> >>> Hey all.  I thought I would try and explain what Chris has been up to
> >>> with at least some of the base.bbclass changes (the ones related to
> >>> md5sum and cp).
> >>>
> >>> Right now, with a big enough BB_NUM_THREADS we can get into a race
> where
> >>> coreutils-native is installing programs and elsewhere we are in a
> >>> do_fetch and either trying to use 'cp' or 'md5sum', and blam, we try
> and
> >>> invoke the program while it's being installed (and see things like
> >>> sh: /path/to/staging/i686-linux/usr/bin/cp: Textfile is busy).
> >>>
> >>> There's a few ways out of this:
> >>> 1) Don't rely on 'cp' and 'md5sum' anymore but use python for it.
> >>> 2) Make an oe_cp and oe_md5sum to go with oe_sha256sum
> >>> 3) IIRC, the big part of coreutils-native was a fully functional,
> >>> always, 'install'.  We could just copy the install we build or provide
> >>> an install wrapper (oe_install) or so
> >>> 4) ???
> >>
> >> Even thought I loathe python, option 1 sounds like a nice way to go
> >> since it doesn't involve butchering recipes.
> >
> > I think we're OK, except in the case of when we can't rely on a built-in
> > md5sum (and then the race comes back) and just need to figure out if the
> > libpam-base-files recipe does things (a) optimally and (b) if that's a
> > common thing.
> >
> > Currently, with a quick glance at the recipe, I don't see any
> > distro-specific overrides, so I'd be inclined to call it lazy recipe
> > writing, until someone points out history of why that's a good thing..
>
> I have no problem with changing the libpam-base-files SRC_URI to
>
> file://pam.d/
>
> or
>
> file://foo.pam \
> file://bar.pam \
> ...
> etc
>
> But I would like to see it documented in the OE manual that file://dir/*
> can't be used, and why.
>

Agreed.  In this particular case, it was never, as far as I'm aware,
*directly* supported, but was one of many cases where we depend on
implementation details - it happens that the argument was directly shoved
onto a 'cp' command line, and that was executed via a shell, so the shell
could expand the glob.  I'll take on adjusting such cases and updating the
docs.

The issue I see with trying to support a globbing mechanism for this is that
the semantics of such a thing are not obvious.  Does it grab everything in
every 'dir' that exists in filespath, or only the first?  It's unclear what
the user intends by that, whereas a 'file://dir/' is rather more clear.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics