[PATCH 0/3][PULL] several machine specific sysroots fixes

[PATCH 0/3][PULL] several machine specific sysroots fixes

[Dongxiao Xu]

From: Dongxiao Xu <dongxiao.xu@intel.com>

Hi Richard,

Here are several fixes for machine specific sysroots, please help to review and pull.

The first patch is to add ${MACHINE} tag on stamps for do_populate_sysroot_setscene and do_package_setscene.
The second patch is to handle the rm_work issues.
The third patch is to remove ${MACHINE} in compile and package functions in perl bb files.

Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: dxu4/distro
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dxu4/distro

Thanks,
    Dongxiao Xu <dongxiao.xu@intel.com>
---


Dongxiao Xu (3):
  bitbake: add ${MACHINE} for do_populate_sysroot_setscene and
    do_package_setscene
  rm_work.bbclass: handle stamp files while doing rm_work
  perl: Remove ${MACHINE} dependency in compile and package.

 meta/classes/cross-canadian.bbclass       |    2 +
 meta/classes/crosssdk.bbclass             |    2 +
 meta/classes/native.bbclass               |    2 +
 meta/classes/nativesdk.bbclass            |    2 +
 meta/classes/package.bbclass              |    1 +
 meta/classes/rm_work.bbclass              |   17 +++++++++++
 meta/classes/staging.bbclass              |    1 +
 meta/recipes-devtools/perl/perl_5.12.2.bb |   42 +++++++++++++----------------
 8 files changed, 46 insertions(+), 23 deletions(-)

[PATCH 1/3] bitbake: add ${MACHINE} for do_populate_sysroot_setscene and do_package_setscene

[Dongxiao Xu]

From: Dongxiao Xu <dongxiao.xu@intel.com>

${MACHINE} information is also needed to be tagged on stamps for
do_populate_sysroot_setscene and do_package_setscene, otherwise if
build two machines both from sstate, the second machine's sysroot will
not be populated correctly.

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
---
 meta/classes/cross-canadian.bbclass |    2 ++
 meta/classes/crosssdk.bbclass       |    2 ++
 meta/classes/native.bbclass         |    2 ++
 meta/classes/nativesdk.bbclass      |    2 ++
 meta/classes/package.bbclass        |    1 +
 meta/classes/staging.bbclass        |    1 +
 6 files changed, 10 insertions(+), 0 deletions(-)

diff --git a/meta/classes/cross-canadian.bbclass b/meta/classes/cross-canadian.bbclass
index 993f831..db4c356 100644
--- a/meta/classes/cross-canadian.bbclass
+++ b/meta/classes/cross-canadian.bbclass
@@ -82,4 +82,6 @@ export PKG_CONFIG_SYSROOT_DIR = "${STAGING_DIR_HOST}"
 SHLIBSDIR = "${STAGING_DIR}/${SDK_ARCH}-nativesdk${SDK_VENDOR}-${BUILD_OS}/shlibs"
 
 do_populate_sysroot[stamp-extra-info] = ""
+do_populate_sysroot_setscene[stamp-extra-info] = ""
 do_package[stamp-extra-info] = ""
+do_package_setscene[stamp-extra-info] = ""
diff --git a/meta/classes/crosssdk.bbclass b/meta/classes/crosssdk.bbclass
index 75fcfef..98c0204 100644
--- a/meta/classes/crosssdk.bbclass
+++ b/meta/classes/crosssdk.bbclass
@@ -17,4 +17,6 @@ target_prefix = "${SDKPATHNATIVE}${prefix_nativesdk}"
 target_exec_prefix = "${SDKPATHNATIVE}${exec_prefix_nativesdk}"
 
 do_populate_sysroot[stamp-extra-info] = ""
+do_populate_sysroot_setscene[stamp-extra-info] = ""
 do_package[stamp-extra-info] = ""
+do_package_setscene[stamp-extra-info] = ""
diff --git a/meta/classes/native.bbclass b/meta/classes/native.bbclass
index 545a73e..789a163 100644
--- a/meta/classes/native.bbclass
+++ b/meta/classes/native.bbclass
@@ -121,4 +121,6 @@ do_package_write_deb[noexec] = "1"
 do_package_write_rpm[noexec] = "1"
 
 do_populate_sysroot[stamp-extra-info] = ""
+do_populate_sysroot_setscene[stamp-extra-info] = ""
 do_package[stamp-extra-info] = ""
+do_package_setscene[stamp-extra-info] = ""
diff --git a/meta/classes/nativesdk.bbclass b/meta/classes/nativesdk.bbclass
index 587a907..cd642c6 100644
--- a/meta/classes/nativesdk.bbclass
+++ b/meta/classes/nativesdk.bbclass
@@ -82,4 +82,6 @@ python __anonymous () {
 }
 
 do_populate_sysroot[stamp-extra-info] = ""
+do_populate_sysroot_setscene[stamp-extra-info] = ""
 do_package[stamp-extra-info] = ""
+do_package_setscene[stamp-extra-info] = ""
diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 1b950bb..c4e4276 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1095,6 +1095,7 @@ do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
 do_package[sstate-lockfile] = "${PACKAGELOCK}"
 do_package[stamp-extra-info] = "${MACHINE}"
+do_package_setscene[stamp-extra-info] = "${MACHINE}"
 do_package_setscene[dirs] = "${STAGING_DIR}"
 
 python do_package_setscene () {
diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
index 718b2f1..e2333ca 100644
--- a/meta/classes/staging.bbclass
+++ b/meta/classes/staging.bbclass
@@ -86,6 +86,7 @@ do_populate_sysroot[sstate-name] = "populate-sysroot"
 do_populate_sysroot[sstate-inputdirs] = "${SYSROOT_DESTDIR}"
 do_populate_sysroot[sstate-outputdirs] = "${STAGING_DIR_HOST}/"
 do_populate_sysroot[stamp-extra-info] = "${MACHINE}"
+do_populate_sysroot_setscene[stamp-extra-info] = "${MACHINE}"
 
 python do_populate_sysroot_setscene () {
 	sstate_setscene(d)
-- 
1.6.3.3

[PATCH 2/3] rm_work.bbclass: handle stamp files while doing rm_work

[Dongxiao Xu]

From: Dongxiao Xu <dongxiao.xu@intel.com>

If rm_work is scheduled, we need to remove the stamps of fetch, unpack,
patch, etc, while just leaving the setscene varients.

For example, we'd task stamps like:

xxx-1.4.7-r3.do_compile
xxx-1.4.7-r3.do_configure
xxx-1.4.7-r3.do_fetch
xxx-1.4.7-r3.do_generate_toolchain_file
xxx-1.4.7-r3.do_install
xxx-1.4.7-r3.do_package.emenlow
xxx-1.4.7-r3.do_package_write
xxx-1.4.7-r3.do_package_write_ipk
xxx-1.4.7-r3.do_package_write_rpm
xxx-1.4.7-r3.do_patch
xxx-1.4.7-r3.do_populate_sysroot.emenlow
xxx-1.4.7-r3.do_setscene
xxx-1.4.7-r3.do_unpack

and after rm_work, we have stamps of:

xxx-1.4.7-r3.do_package_setscene.emenlow
xxx-1.4.7-r3.do_package_write_ipk_setscene
xxx-1.4.7-r3.do_package_write_rpm_setscene
xxx-1.4.7-r3.do_populate_sysroot_setscene.emenlow

This can solve the issue we meet if we force to schedule a task after
contents in ${WORKDIR} are removed.

Also this can solve the rm_work issue we met after machine specific
sysroot is implemented.

Thanks for Richard's suggestion to tracking these issues down.

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
---
 meta/classes/rm_work.bbclass |   17 +++++++++++++++++
 1 files changed, 17 insertions(+), 0 deletions(-)

diff --git a/meta/classes/rm_work.bbclass b/meta/classes/rm_work.bbclass
index 260ecb0..a151f4c 100644
--- a/meta/classes/rm_work.bbclass
+++ b/meta/classes/rm_work.bbclass
@@ -26,6 +26,23 @@ do_rm_work () {
     # Need to add pseudo back or subsqeuent work in this workdir
     # might fail since setscene may not rerun to recreate it
     mkdir ${WORKDIR}/pseudo/
+
+    rm -rf `ls ${STAMP}* | grep -v sigdata | grep -v do_populate_sysroot | grep -v do_package` ${STAMP}.do_package_write
+
+    for stamp in `ls ${STAMP}* | grep do_package_write_* | grep -v setscene`
+    do
+        mv $stamp $stamp\_setscene
+    done
+
+    for task in do_populate_sysroot do_package
+    do
+        if [ -e ${STAMP}.$task ]; then
+            mv ${STAMP}.$task ${STAMP}.$task\_setscene
+        elif [ -e ${STAMP}.$task.${MACHINE} ]; then
+            mv ${STAMP}.$task.${MACHINE} ${STAMP}.$task\_setscene.${MACHINE}
+        fi
+    done
+
 }
 addtask rm_work after do_${RMWORK_ORIG_TASK}
 
-- 
1.6.3.3

[PATCH 3/3] perl: Remove ${MACHINE} dependency in compile and package.

[Dongxiao Xu]

From: Dongxiao Xu <dongxiao.xu@intel.com>

The judgement on ${MACHINE} is meaningless, remove it in order to avoid
being rebuild on second machine.

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
---
 meta/recipes-devtools/perl/perl_5.12.2.bb |   42 +++++++++++++----------------
 1 files changed, 19 insertions(+), 23 deletions(-)

diff --git a/meta/recipes-devtools/perl/perl_5.12.2.bb b/meta/recipes-devtools/perl/perl_5.12.2.bb
index e7e1cb4..82801e8 100644
--- a/meta/recipes-devtools/perl/perl_5.12.2.bb
+++ b/meta/recipes-devtools/perl/perl_5.12.2.bb
@@ -151,12 +151,10 @@ do_configure() {
 }
 
 do_compile() {
-        if test "${MACHINE}" != "native"; then
-            sed -i -e 's|/usr/include|${STAGING_INCDIR}|g' ext/Errno/Errno_pm.PL
-            sed -i -e 's|/usr/include|${STAGING_INCDIR}|g' cpan/Compress-Raw-Zlib/config.in
-            sed -i -e 's|/usr/lib|${STAGING_LIBDIR}|g' cpan/Compress-Raw-Zlib/config.in
+        sed -i -e 's|/usr/include|${STAGING_INCDIR}|g' ext/Errno/Errno_pm.PL
+        sed -i -e 's|/usr/include|${STAGING_INCDIR}|g' cpan/Compress-Raw-Zlib/config.in
+        sed -i -e 's|/usr/lib|${STAGING_LIBDIR}|g' cpan/Compress-Raw-Zlib/config.in
 
-        fi
         cd Cross
         oe_runmake perl LD="${CCLD}"
 }
@@ -186,24 +184,22 @@ PACKAGE_PREPROCESS_FUNCS += "perl_package_preprocess"
 
 perl_package_preprocess () {
         # Fix up installed configuration
-        if test "${MACHINE}" != "native"; then
-            sed -i -e "s,${D},,g" \
-                   -e "s,-isystem${STAGING_INCDIR} ,,g" \
-                   -e "s,${STAGING_LIBDIR},${libdir},g" \
-                   -e "s,${STAGING_BINDIR},${bindir},g" \
-                   -e "s,${STAGING_INCDIR},${includedir},g" \
-                   -e "s,${STAGING_BINDIR_NATIVE}/,,g" \
-                ${PKGD}${bindir}/h2xs \
-                ${PKGD}${bindir}/h2ph \
-                ${PKGD}${libdir}/perl/${PV}/pod/*.pod \
-                ${PKGD}${libdir}/perl/${PV}/cacheout.pl \
-                ${PKGD}${libdir}/perl/${PV}/FileCache.pm \
-                ${PKGD}${libdir}/perl/config.sh \
-                ${PKGD}${libdir}/perl/${PV}/Config.pm \
-                ${PKGD}${libdir}/perl/${PV}/Config_heavy.pl \
-                ${PKGD}${libdir}/perl/${PV}/CORE/perl.h \
-                ${PKGD}${libdir}/perl/${PV}/CORE/pp.h
-        fi
+        sed -i -e "s,${D},,g" \
+               -e "s,-isystem${STAGING_INCDIR} ,,g" \
+               -e "s,${STAGING_LIBDIR},${libdir},g" \
+               -e "s,${STAGING_BINDIR},${bindir},g" \
+               -e "s,${STAGING_INCDIR},${includedir},g" \
+               -e "s,${STAGING_BINDIR_NATIVE}/,,g" \
+            ${PKGD}${bindir}/h2xs \
+            ${PKGD}${bindir}/h2ph \
+            ${PKGD}${libdir}/perl/${PV}/pod/*.pod \
+            ${PKGD}${libdir}/perl/${PV}/cacheout.pl \
+            ${PKGD}${libdir}/perl/${PV}/FileCache.pm \
+            ${PKGD}${libdir}/perl/config.sh \
+            ${PKGD}${libdir}/perl/${PV}/Config.pm \
+            ${PKGD}${libdir}/perl/${PV}/Config_heavy.pl \
+            ${PKGD}${libdir}/perl/${PV}/CORE/perl.h \
+            ${PKGD}${libdir}/perl/${PV}/CORE/pp.h
 }
 
 PACKAGES = "perl-dbg perl perl-misc perl-dev perl-pod perl-doc perl-lib \
-- 
1.6.3.3

Re: [PATCH 2/3] rm_work.bbclass: handle stamp files while doing rm_work

[Richard Purdie]

On Thu, 2011-01-27 at 16:04 +0800, Dongxiao Xu wrote:
> diff --git a/meta/classes/rm_work.bbclass b/meta/classes/rm_work.bbclass
> index 260ecb0..a151f4c 100644
> --- a/meta/classes/rm_work.bbclass
> +++ b/meta/classes/rm_work.bbclass
> @@ -26,6 +26,23 @@ do_rm_work () {
>      # Need to add pseudo back or subsqeuent work in this workdir
>      # might fail since setscene may not rerun to recreate it
>      mkdir ${WORKDIR}/pseudo/
> +
> +    rm -rf `ls ${STAMP}* | grep -v sigdata | grep -v do_populate_sysroot | grep -v do_package` ${STAMP}.do_package_write
> +
> +    for stamp in `ls ${STAMP}* | grep do_package_write_* | grep -v setscene`
> +    do
> +        mv $stamp $stamp\_setscene
> +    done
> +
> +    for task in do_populate_sysroot do_package
> +    do
> +        if [ -e ${STAMP}.$task ]; then
> +            mv ${STAMP}.$task ${STAMP}.$task\_setscene
> +        elif [ -e ${STAMP}.$task.${MACHINE} ]; then
> +            mv ${STAMP}.$task.${MACHINE} ${STAMP}.$task\_setscene.${MACHINE}
> +        fi
> +    done
> +
>  }
>  addtask rm_work after do_${RMWORK_ORIG_TASK}

This is the right approach, however, if I set BB_SIGNATURE_HANDLER =
"basichash" it will break since the stamps then look like:

ncurses-5.7-r0.do_populate_sysroot.59a5b1c1ce0a1bab23be44c27ba308a5.qemux86

and also, if we add new tasks to sstate, this code would then need to be
updated with the new tasknames.

I ended up writing http://git.pokylinux.org/cgit.cgi/poky/commit/?id=e8e9f56057f3732d2550375a23573ab7231eac22
which I merged into master to address the problem since its breaking the
autobuilder and people's builds out there. I'd have thought there would
be a neater way to do this but I can't find one at the moment. Any
improvements anyone can see are welcome.

Cheers,

Richard

Re: [PATCH 1/3] bitbake: add ${MACHINE} for do_populate_sysroot_setscene and do_package_setscene

[Richard Purdie]

On Thu, 2011-01-27 at 16:04 +0800, Dongxiao Xu wrote:
> From: Dongxiao Xu <dongxiao.xu@intel.com>
> 
> ${MACHINE} information is also needed to be tagged on stamps for
> do_populate_sysroot_setscene and do_package_setscene, otherwise if
> build two machines both from sstate, the second machine's sysroot will
> not be populated correctly.
> 
> Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>

The _setscene stuff is really an internal implementation issue within
bitbake. I don't like the idea that the metadata has to get this right
and I'd prefer it if bitbake always did the right thing. In this case
that means looking up the extra stamp info against the parent task, not
the setscene one.

I therefore merged:

http://git.pokylinux.org/cgit.cgi/poky/commit/?id=c34913ea5a4f20ce4bca3c68a65e9f64f1e3910c

which should obsolete this patch?

Good catch on finding the problem though!

Cheers,

Richard

Re: [PATCH 3/3] perl: Remove ${MACHINE} dependency in compile and package.

[Richard Purdie]

On Thu, 2011-01-27 at 16:04 +0800, Dongxiao Xu wrote:
> From: Dongxiao Xu <dongxiao.xu@intel.com>
> 
> The judgement on ${MACHINE} is meaningless, remove it in order to avoid
> being rebuild on second machine.
> 
> Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>

I've merged this into master, thanks!

Richard

Re: [PATCH 2/3] rm_work.bbclass: handle stamp files while doing rm_work

[Xu, Dongxiao]

Richard Purdie wrote:
> On Thu, 2011-01-27 at 16:04 +0800, Dongxiao Xu wrote:
>> diff --git a/meta/classes/rm_work.bbclass
>> b/meta/classes/rm_work.bbclass index 260ecb0..a151f4c 100644
>> --- a/meta/classes/rm_work.bbclass
>> +++ b/meta/classes/rm_work.bbclass
>> @@ -26,6 +26,23 @@ do_rm_work () {
>>      # Need to add pseudo back or subsqeuent work in this workdir
>>      # might fail since setscene may not rerun to recreate it     
>> mkdir ${WORKDIR}/pseudo/ +
>> +    rm -rf `ls ${STAMP}* | grep -v sigdata | grep -v
>> + do_populate_sysroot | grep -v do_package`
>> ${STAMP}.do_package_write + +    for stamp in `ls ${STAMP}* | grep
>> do_package_write_* | grep -v setscene` +    do +        mv $stamp
>> $stamp\_setscene +    done
>> +
>> +    for task in do_populate_sysroot do_package
>> +    do
>> +        if [ -e ${STAMP}.$task ]; then
>> +            mv ${STAMP}.$task ${STAMP}.$task\_setscene
>> +        elif [ -e ${STAMP}.$task.${MACHINE} ]; then
>> +            mv ${STAMP}.$task.${MACHINE}
>> ${STAMP}.$task\_setscene.${MACHINE} +        fi +    done
>> +
>>  }
>>  addtask rm_work after do_${RMWORK_ORIG_TASK}
> 
> This is the right approach, however, if I set BB_SIGNATURE_HANDLER =
> "basichash" it will break since the stamps then look like: 
> 
> ncurses-5.7-r0.do_populate_sysroot.59a5b1c1ce0a1bab23be44c27ba308a5.qemux86
> 
> and also, if we add new tasks to sstate, this code would then need to
> be updated with the new tasknames. 
> 
> I ended up writing
> http://git.pokylinux.org/cgit.cgi/poky/commit/?id=e8e9f56057f3732d2550375a23573ab7231eac22
> which I merged into master to address the problem since its breaking
> the autobuilder and people's builds out there. I'd have thought there
> would be a neater way to do this but I can't find one at the moment.
> Any improvements anyone can see are welcome.

Thanks for your fix, I am not aware of such stamp format, thus it's not covered in my previous fix.

Thanks,
Dongxiao
   
> 
> Cheers,
> 
> Richard

Re: [PATCH 1/3] bitbake: add ${MACHINE} for do_populate_sysroot_setscene and do_package_setscene

[Xu, Dongxiao]

Richard Purdie wrote:
> On Thu, 2011-01-27 at 16:04 +0800, Dongxiao Xu wrote:
>> From: Dongxiao Xu <dongxiao.xu@intel.com>
>> 
>> ${MACHINE} information is also needed to be tagged on stamps for
>> do_populate_sysroot_setscene and do_package_setscene, otherwise if
>> build two machines both from sstate, the second machine's sysroot
>> will not be populated correctly. 
>> 
>> Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
> 
> The _setscene stuff is really an internal implementation issue within
> bitbake. I don't like the idea that the metadata has to get this
> right and I'd prefer it if bitbake always did the right thing. In
> this case that means looking up the extra stamp info against the
> parent task, not the setscene one.    
> 
> I therefore merged:
> 
> http://git.pokylinux.org/cgit.cgi/poky/commit/?id=c34913ea5a4f20ce4bca3c68a65e9f64f1e3910c

Yes, this approach is better then specifically handling *_setscene stamp-extra-info in meta files. Thanks!

-- Dongxiao

> 
> which should obsolete this patch?
> 
> Good catch on finding the problem though!
> 
> Cheers,
> 
> Richard