Re: [OE-core] [PATCH] rootfs_ipk: allow do_populate_sdk in parallel to do_rootfs

["Paul Barker" <pbarker@konsulko.com>]

On Tue, 12 Jan 2021 at 09:21, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Mon, 2021-01-11 at 13:45 +0100, Michael Ho wrote:
> > From: Michael Ho <Michael.Ho@bmw.de>
> >
> > Switch do_populate_sdk for the ipk package manager to use a separate target
> > opkg config file and separate the lockfiles restricting do_rootfs and
> > do_populate_sdk from running in parallel.
> >
> > This way if an image recipe includes a dependency to do_populate_sdk by
> > default then it will run in parallel to do_rootfs saving time compared to the
> > sequential execution.
> >
> > Signed-off-by: Michael Ho <Michael.Ho@bmw.de>
> > ---
> >  meta/classes/package_ipk.bbclass       | 1 +
> >  meta/classes/rootfs_ipk.bbclass        | 4 ++--
> >  meta/lib/oe/package_manager/ipk/sdk.py | 6 ++++++
> >  3 files changed, 9 insertions(+), 2 deletions(-)
>
> I have to admit I'm very nervous about this change. The races we've
> seen betweem rootfs and sdk can be quite unusual.
>
> I did put this in for testing and we saw:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/2938
>
> which in the pseudo.log shows:
>
> path mismatch [2 links]: ino 372706913 db '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato/1.0-r0/tmp-wic' req '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato/1.0-r0/sdk/image/opt/poky/3.2+snapshot/sysroots/cortexa8hf-neon-poky-linux-gnueabi/usr/src/debug/puzzles'.
> path mismatch [2 links]: ino 372706913 db '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato/1.0-r0/tmp-wic' req '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato/1.0-r0/sdk/image/opt/poky/3.2+snapshot/sysroots/cortexa8hf-neon-poky-linux-gnueabi/usr/src/debug/puzzles'.
>
> its hard to know if this is due to this patch or possibly Paul's wic
> changes as both were in this test series. I suspect it won't reproduce
> every time since its a race.

Those path mismatches do seem to show that the `tmp-wic` directory was
removed outside of a pseudo context. That would point the finger at my
patches.

wic does delete its temporary working directory when exiting
(http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/scripts/lib/wic/plugins/imager/direct.py?h=master-next&id=b86e5b32f767b69bb770f6060cb784b15b147f22#n268)
but that should be ok as wic itself is running under pseudo when
invoked by bitbake. So when do_image_wic finishes, the path to
`tmp-wic` should no longer be present in the pseudo db. The error seen
here may be due to a race condition but the underlying question of
whether `tmp-wic` is removed outside a pseudo context (leaving the
path in the db) should hopefully be deterministic. Is there a way to
query the pseudo db to see if a path is present or not? I'd like to
try that after running do_image_wic.

Thanks,

-- 
Paul Barker
Konsulko Group