Re: [PATCH 1/1] rm_work.bbclass: reserve 'image' folder when remove WORKDIR

Re: [PATCH 1/1] rm_work.bbclass: reserve 'image' folder when remove WORKDIR

[Koen Kooi]

Op 26 jan 2011, om 07:30 heeft poky-bounces@yoctoproject.org het volgende geschreven:

> From: Dongxiao Xu <dongxiao.xu@intel.com>
> 
> After the implementation of per machine sysroot, do_populate_sysroot
> and do_package need to be re-run if we build two machines of one
> architecture (in case that sstate prebuilt result could not be reused
> for certain recipes). But if removing all the WORKDIR contents after
> the first macine build, the second machine's do_populate_sysroot and
> do_package will fail due to the missing of "image" directory.
> 
> Thus reserve 'image' folder when doing rm_work.

Wouldn't adding do_install to sstate be a better idea? OE was designed that it wouldn't depend on WORKDIR after building, so this patch is a step in the wrong direction.

regards,

Koen

Re: [PATCH 1/1] rm_work.bbclass: reserve 'image' folder when remove WORKDIR

[Xu, Dongxiao]

Koen Kooi wrote:
> Op 26 jan 2011, om 07:30 heeft poky-bounces@yoctoproject.org het
> volgende geschreven: 
> 
>> From: Dongxiao Xu <dongxiao.xu@intel.com>
>> 
>> After the implementation of per machine sysroot, do_populate_sysroot
>> and do_package need to be re-run if we build two machines of one
>> architecture (in case that sstate prebuilt result could not be reused
>> for certain recipes). But if removing all the WORKDIR contents after
>> the first macine build, the second machine's do_populate_sysroot and
>> do_package will fail due to the missing of "image" directory.
>> 
>> Thus reserve 'image' folder when doing rm_work.
> 
> Wouldn't adding do_install to sstate be a better idea? OE was
> designed that it wouldn't depend on WORKDIR after building, so this
> patch is a step in the wrong direction.  

Hi Koen,

Thanks for your suggestions, however this may not help to solve this issue.

The senario of the above issue is, build one machine firstly, doing rm_work, and then try to build the second machine of the same architecture. Some certain recipe's prebuilt result could not be shared between machines.

After the first machine finishes the build, most contents in ${WORKDIR} has been removed by "rm_work". Prebuilt result could not be shared for certain recipes. For those recipes, the second machine finds that most of the recipe stamp files are already there (do_fetch, do_unpack, do_patch, ...) except do_populate_sysroot and do_package, which contains ${MACHINE} tag, thus it will try to re-run do_populate_sysroot and do_package, which relies on the existence of "image" directory.

So even we add the do_install in sstate, the above issue could not be solved because it is triggered when certain recipe's sstate prebuilt result could *not* be shared.

Thanks,
Dongxiao

> 
> regards,
> 
> Koen
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: [PATCH 1/1] rm_work.bbclass: reserve 'image' folder when remove WORKDIR

[Richard Purdie]

On Wed, 2011-01-26 at 14:30 +0800, poky-bounces@yoctoproject.org wrote:
> From: Dongxiao Xu <dongxiao.xu@intel.com>
> 
> After the implementation of per machine sysroot, do_populate_sysroot
> and do_package need to be re-run if we build two machines of one
> architecture (in case that sstate prebuilt result could not be reused
> for certain recipes). But if removing all the WORKDIR contents after
> the first macine build, the second machine's do_populate_sysroot and
> do_package will fail due to the missing of "image" directory.
> 
> Thus reserve 'image' folder when doing rm_work.

Whilst I understand the intent, this is just putting wallpaper over
cracks, sooner or later the crack will reappear, possibly worse than
before.

For example if you run "bitbake xxx -c install -f" for a target xxx
which has had rm_work run against it, the result will be a nasty error
in Poky.

OE's rm_work.bbclass is a little cleverer than ours and has this:

addhandler rmwork_stampfixing_eventhandler
python rmwork_stampfixing_eventhandler() {
    if bb.event.getName(e) == "StampUpdate":
        for (fn, task) in e.targets:
            if task == 'rm_work_all':
                continue
            stamp = "%s.do_rm_work" % e.stampPrefix[fn]
            if os.path.exists(stamp):
                dir = "%s.*" % e.stampPrefix[fn]
                bb.note("Removing stamps: " + dir)
                os.system('rm -f '+ dir)
}

which is intended to fix my specific example above. It won't fix the new
corner cases we've introduced with machine specific sysroots though. So
what do we do to fix this?

I'm actually tempted to tie together rm_work, sstate and the scenequeue.
When rm_work wipes out the workdir, it also migrates the stamps into
_setscene stamps as if the package had been installed by sstate. We'd
then remove the unpack, fetch, patch and so on stamps just leaving the
setscene varients. This would mean the above wouldn't be needed and most
things should "just work".

So to be specific, we'd take stamps like:

xxx-1.4.7-r3.do_compile
xxx-1.4.7-r3.do_configure
xxx-1.4.7-r3.do_fetch
xxx-1.4.7-r3.do_generate_toolchain_file
xxx-1.4.7-r3.do_install
xxx-1.4.7-r3.do_package.emenlow
xxx-1.4.7-r3.do_package_write
xxx-1.4.7-r3.do_package_write_ipk
xxx-1.4.7-r3.do_package_write_rpm
xxx-1.4.7-r3.do_patch
xxx-1.4.7-r3.do_populate_sysroot.emenlow
xxx-1.4.7-r3.do_setscene
xxx-1.4.7-r3.do_unpack

and end up with:

xxx-1.4.7-r3.do_package_setscene.emenlow
xxx-1.4.7-r3.do_package_write_ipk_setscene
xxx-1.4.7-r3.do_package_write_rpm_setscene
xxx-1.4.7-r3.do_populate_sysroot_setscene.emenlow

Does that sound reasonable?

Then both codepaths use the same underlying code and everything should
be fine apart from some hardcoded assumptions about stamp filenames
which we'll probably just have to live with for now. The function will
need to handle stamps including the taskhash too, e.g.:

xxx-1.4.7-r3.do_unpack.HASHHERE

Cheers,

Richard

Re: [PATCH 1/1] rm_work.bbclass: reserve 'image' folder when remove WORKDIR

[Xu, Dongxiao]

Richard Purdie wrote:
> On Wed, 2011-01-26 at 14:30 +0800, poky-bounces@yoctoproject.org
> wrote: 
>> From: Dongxiao Xu <dongxiao.xu@intel.com>
>> 
>> After the implementation of per machine sysroot, do_populate_sysroot
>> and do_package need to be re-run if we build two machines of one
>> architecture (in case that sstate prebuilt result could not be reused
>> for certain recipes). But if removing all the WORKDIR contents after
>> the first macine build, the second machine's do_populate_sysroot and
>> do_package will fail due to the missing of "image" directory.
>> 
>> Thus reserve 'image' folder when doing rm_work.
> 
> Whilst I understand the intent, this is just putting wallpaper over
> cracks, sooner or later the crack will reappear, possibly worse than
> before.  
> 
> For example if you run "bitbake xxx -c install -f" for a target xxx
> which has had rm_work run against it, the result will be a nasty
> error in Poky.  
> 
> OE's rm_work.bbclass is a little cleverer than ours and has this:
> 
> addhandler rmwork_stampfixing_eventhandler python
>     rmwork_stampfixing_eventhandler() { if bb.event.getName(e) ==
>         "StampUpdate": for (fn, task) in e.targets:
>             if task == 'rm_work_all':
>                 continue
>             stamp = "%s.do_rm_work" % e.stampPrefix[fn]
>             if os.path.exists(stamp):
>                 dir = "%s.*" % e.stampPrefix[fn]
>                 bb.note("Removing stamps: " + dir)
>                 os.system('rm -f '+ dir) }
> 
> which is intended to fix my specific example above. It won't fix the
> new corner cases we've introduced with machine specific sysroots
> though. So what do we do to fix this?  
> 
> I'm actually tempted to tie together rm_work, sstate and the
> scenequeue. 
> When rm_work wipes out the workdir, it also migrates the stamps into
> _setscene stamps as if the package had been installed by sstate. We'd
> then remove the unpack, fetch, patch and so on stamps just leaving
> the setscene varients. This would mean the above wouldn't be needed
> and most things should "just work".    
> 
> So to be specific, we'd take stamps like:
> 
> xxx-1.4.7-r3.do_compile
> xxx-1.4.7-r3.do_configure
> xxx-1.4.7-r3.do_fetch
> xxx-1.4.7-r3.do_generate_toolchain_file
> xxx-1.4.7-r3.do_install
> xxx-1.4.7-r3.do_package.emenlow
> xxx-1.4.7-r3.do_package_write
> xxx-1.4.7-r3.do_package_write_ipk
> xxx-1.4.7-r3.do_package_write_rpm
> xxx-1.4.7-r3.do_patch
> xxx-1.4.7-r3.do_populate_sysroot.emenlow
> xxx-1.4.7-r3.do_setscene
> xxx-1.4.7-r3.do_unpack
> 
> and end up with:
> 
> xxx-1.4.7-r3.do_package_setscene.emenlow
> xxx-1.4.7-r3.do_package_write_ipk_setscene
> xxx-1.4.7-r3.do_package_write_rpm_setscene
> xxx-1.4.7-r3.do_populate_sysroot_setscene.emenlow
> 
> Does that sound reasonable?
> 
> Then both codepaths use the same underlying code and everything
> should be fine apart from some hardcoded assumptions about stamp
> filenames which we'll probably just have to live with for now. The
> function will need to handle stamps including the taskhash too, e.g.:
> 
> xxx-1.4.7-r3.do_unpack.HASHHERE

Hi Richard, 

I think this way is workable to track down the several rm_work issues.

Could you help to clarify what are "hardcoded assumptions about stamp filenames"?

Also you mentioned we need to handle the HASH stamp, do you mean we need to reserve those HASH stamps and they should not be removed while rm_work?

I just wrote a patch that implement the above senario and now is doing test for it.

Please help to review it to see if I understand your point correctly.

http://git.pokylinux.org/cgit/cgit.cgi/poky-contrib/log/?h=dxu4/distro

There are two patches in the branch, the first one is a fix patch that add "${MACHINE}" tag into stamps of do_populate_sysroot_setscene and do_package_setscene, and the second one is to handle rm_work issue.

Thanks,
Dongxiao

> 
> Cheers,
> 
> Richard