All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 1/3] Make package_update_index_ipk proper task with locking
@ 2009-06-17 19:41 Roman I Khimov
  2009-06-17 19:41 ` [PATCH 2/3] package_ipk: optimize do_package_update_index_ipk Roman I Khimov
                   ` (2 more replies)
  0 siblings, 3 replies; 16+ messages in thread
From: Roman I Khimov @ 2009-06-17 19:41 UTC (permalink / raw)
  To: openembedded-devel

Parallel image builds race in package_update_index_ipk, so locking is
needed and for a proper lock package_update_index_ipk needs to be a
proper task.
---
 classes/package_ipk.bbclass    |    9 ++++++++-
 classes/rootfs_ipk.bbclass     |    4 +---
 recipes/meta/canadian-sdk.bb   |    3 +--
 recipes/meta/meta-toolchain.bb |    3 +--
 recipes/meta/package-index.bb  |    6 +-----
 5 files changed, 12 insertions(+), 13 deletions(-)

diff --git a/classes/package_ipk.bbclass b/classes/package_ipk.bbclass
index d4cb3b6..1f1b5c9 100644
--- a/classes/package_ipk.bbclass
+++ b/classes/package_ipk.bbclass
@@ -67,10 +67,16 @@ python package_ipk_install () {
 		raise bb.build.FuncFailed
 }
 
+do_package_update_index_ipk[lockfiles] = "${DEPLOY_DIR_IPK}.lock"
+do_package_update_index_ipk[nostamp] = "1"
+do_package_update_index_ipk[recrdeptask] += "do_package_write_ipk"
+do_package_update_index_ipk[recrdeptask] += "do_package_write_ipk"
+do_package_update_index_ipk[depends] += "ipkg-utils-native:do_populate_staging"
+
 #
 # Update the Packages index files in ${DEPLOY_DIR_IPK}
 #
-package_update_index_ipk () {
+do_package_update_index_ipk () {
 	set -x
 
 	ipkgarchs="${PACKAGE_ARCHS}"
@@ -322,3 +328,4 @@ python do_package_write_ipk () {
 }
 do_package_write_ipk[dirs] = "${D}"
 addtask package_write_ipk before do_package_write after do_package
+addtask package_update_index_ipk before do_rootfs
diff --git a/classes/rootfs_ipk.bbclass b/classes/rootfs_ipk.bbclass
index 99a8ed2..16dd511 100644
--- a/classes/rootfs_ipk.bbclass
+++ b/classes/rootfs_ipk.bbclass
@@ -5,8 +5,7 @@
 # See image.bbclass for a usage of this.
 #
 
-do_rootfs[depends] += "opkg-native:do_populate_staging ipkg-utils-native:do_populate_staging"
-do_rootfs[recrdeptask] += "do_package_write_ipk"
+do_rootfs[depends] += "opkg-native:do_populate_staging"
 
 IPKG_ARGS = "-f ${IPKGCONF_TARGET} -o ${IMAGE_ROOTFS} ${@base_conditional("PACKAGE_INSTALL_NO_DEPS", "1", "-nodeps", "", d)}"
 
@@ -29,7 +28,6 @@ PACKAGE_INSTALL_append = " ${@base_conditional("ONLINE_PACKAGE_MANAGEMENT", "non
 fakeroot rootfs_ipk_do_rootfs () {
 	set -x
 
-	package_update_index_ipk
 	package_generate_ipkg_conf
 
 	mkdir -p ${T}
diff --git a/recipes/meta/canadian-sdk.bb b/recipes/meta/canadian-sdk.bb
index 809173b..86b645b 100644
--- a/recipes/meta/canadian-sdk.bb
+++ b/recipes/meta/canadian-sdk.bb
@@ -56,7 +56,6 @@ do_populate_sdk() {
 	rm -rf ${SDK_OUTPUT}
 	mkdir -p ${SDK_OUTPUT}
 
-	package_update_index_ipk
 	package_generate_ipkg_conf
 
 	for arch in ${PACKAGE_ARCHS}; do
@@ -170,5 +169,5 @@ do_populate_sdk() {
 }
 
 do_populate_sdk[nostamp] = "1"
-do_populate_sdk[recrdeptask] = "do_package_write"
+addtask package_update_index_ipk before do_populate_sdk
 addtask populate_sdk before do_build after do_install
diff --git a/recipes/meta/meta-toolchain.bb b/recipes/meta/meta-toolchain.bb
index b641be1..b8ed801 100644
--- a/recipes/meta/meta-toolchain.bb
+++ b/recipes/meta/meta-toolchain.bb
@@ -56,7 +56,6 @@ do_populate_sdk() {
 	rm -rf ${SDK_OUTPUT}
 	mkdir -p ${SDK_OUTPUT}
 
-	package_update_index_ipk
 	package_generate_ipkg_conf
 
 	for arch in ${PACKAGE_ARCHS}; do
@@ -170,5 +169,5 @@ do_populate_sdk() {
 }
 
 do_populate_sdk[nostamp] = "1"
-do_populate_sdk[recrdeptask] = "do_package_write"
+addtask package_update_index_ipk before do_populate_sdk
 addtask populate_sdk before do_build after do_install
diff --git a/recipes/meta/package-index.bb b/recipes/meta/package-index.bb
index f52fb6c..ce8b999 100644
--- a/recipes/meta/package-index.bb
+++ b/recipes/meta/package-index.bb
@@ -23,8 +23,4 @@ do_stage() {
 
 do_build[nostamp] = "1"
 do_build[dirs] = "${DEPLOY_DIR_IPK}"
-do_build() {
-	set -ex
-	package_update_index_ipk
-	set +ex
-}
+addtask package_update_index_ipk before do_build
-- 
1.6.3.1




^ permalink raw reply related	[flat|nested] 16+ messages in thread

* [PATCH 2/3] package_ipk: optimize do_package_update_index_ipk
  2009-06-17 19:41 [PATCH 1/3] Make package_update_index_ipk proper task with locking Roman I Khimov
@ 2009-06-17 19:41 ` Roman I Khimov
  2009-06-17 19:42   ` [PATCH 3/3] package_ipk: fix race in opkg.conf creation Roman I Khimov
                     ` (3 more replies)
  2009-06-17 21:04 ` [PATCH 1/3] Make package_update_index_ipk proper task with locking Tom Rini
  2009-06-18 13:05 ` Richard Purdie
  2 siblings, 4 replies; 16+ messages in thread
From: Roman I Khimov @ 2009-06-17 19:41 UTC (permalink / raw)
  To: openembedded-devel

If there are no package updates since last index update, don't re-create
indexes. Really helps when building something like 10 images.

Remove code duplication along the way.
---
 classes/package_ipk.bbclass |   19 +++++++------------
 1 files changed, 7 insertions(+), 12 deletions(-)

diff --git a/classes/package_ipk.bbclass b/classes/package_ipk.bbclass
index 1f1b5c9..d804c78 100644
--- a/classes/package_ipk.bbclass
+++ b/classes/package_ipk.bbclass
@@ -89,18 +89,13 @@ do_package_update_index_ipk () {
 	ipkg-make-index -r ${DEPLOY_DIR_IPK}/Packages -p ${DEPLOY_DIR_IPK}/Packages -l ${DEPLOY_DIR_IPK}/Packages.filelist -m ${DEPLOY_DIR_IPK}
 
 	for arch in $ipkgarchs; do
-		if [ -e ${DEPLOY_DIR_IPK}/$arch/ ] ; then 
-			touch ${DEPLOY_DIR_IPK}/$arch/Packages
-			ipkg-make-index -r ${DEPLOY_DIR_IPK}/$arch/Packages -p ${DEPLOY_DIR_IPK}/$arch/Packages -l ${DEPLOY_DIR_IPK}/$arch/Packages.filelist -m ${DEPLOY_DIR_IPK}/$arch/
-		fi
-		if [ -e ${DEPLOY_DIR_IPK}/${BUILD_ARCH}-$arch-sdk/ ] ; then 
-			touch ${DEPLOY_DIR_IPK}/${BUILD_ARCH}-$arch-sdk/Packages
-			ipkg-make-index -r ${DEPLOY_DIR_IPK}/${BUILD_ARCH}-$arch-sdk/Packages -p ${DEPLOY_DIR_IPK}/${BUILD_ARCH}-$arch-sdk/Packages -l ${DEPLOY_DIR_IPK}/${BUILD_ARCH}-$arch-sdk/Packages.filelist -m ${DEPLOY_DIR_IPK}/${BUILD_ARCH}-$arch-sdk/
-		fi
-		if [ -e ${DEPLOY_DIR_IPK}/${SDK_SYS}-sdk-$arch/ ] ; then
-			touch ${DEPLOY_DIR_IPK}/${SDK_SYS}-sdk-$arch/Packages
-			ipkg-make-index -r ${DEPLOY_DIR_IPK}/${SDK_SYS}-sdk-$arch/Packages -p ${DEPLOY_DIR_IPK}/${SDK_SYS}-sdk-$arch/Packages -l ${DEPLOY_DIR_IPK}/${SDK_SYS}-sdk-$arch/Packages.filelist -m ${DEPLOY_DIR_IPK}/${SDK_SYS}-sdk-$arch/
-		fi
+		for ipk_path in $arch ${BUILD_ARCH}-$arch-sdk ${SDK_SYS}-sdk-$arch; do
+			PACK_COUNT=`ls -t ${DEPLOY_DIR_IPK}/$ipk_path/ | head -n 4 | grep Packages | wc -l`
+			if [ -e ${DEPLOY_DIR_IPK}/$ipk_path/ -a "$PACK_COUNT" != "4" ] ; then 
+				touch ${DEPLOY_DIR_IPK}/$ipk_path/Packages
+				ipkg-make-index -r ${DEPLOY_DIR_IPK}/$ipk_path/Packages -p ${DEPLOY_DIR_IPK}/$ipk_path/Packages -l ${DEPLOY_DIR_IPK}/$ipk_path/Packages.filelist -m ${DEPLOY_DIR_IPK}/$ipk_path/
+			fi
+		done
 	done
 }
 
-- 
1.6.3.1




^ permalink raw reply related	[flat|nested] 16+ messages in thread

* [PATCH 3/3] package_ipk: fix race in opkg.conf creation
  2009-06-17 19:41 ` [PATCH 2/3] package_ipk: optimize do_package_update_index_ipk Roman I Khimov
@ 2009-06-17 19:42   ` Roman I Khimov
  2009-06-17 21:08     ` Tom Rini
  2009-06-18 12:47     ` Richard Purdie
  2009-06-17 21:06   ` [PATCH 2/3] package_ipk: optimize do_package_update_index_ipk Tom Rini
                     ` (2 subsequent siblings)
  3 siblings, 2 replies; 16+ messages in thread
From: Roman I Khimov @ 2009-06-17 19:42 UTC (permalink / raw)
  To: openembedded-devel

It's a very little race, but one can hit it anyway on concurrent image builds.
---
 classes/package_ipk.bbclass |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/classes/package_ipk.bbclass b/classes/package_ipk.bbclass
index d804c78..a4e6918 100644
--- a/classes/package_ipk.bbclass
+++ b/classes/package_ipk.bbclass
@@ -3,7 +3,7 @@ inherit package
 BOOTSTRAP_EXTRA_RDEPENDS += "opkg-collateral opkg"
 IMAGE_PKGTYPE ?= "ipk"
 
-IPKGCONF_TARGET = "${STAGING_ETCDIR_NATIVE}/opkg.conf"
+IPKGCONF_TARGET = "${STAGING_ETCDIR_NATIVE}/${PN}-opkg.conf"
 IPKGCONF_SDK =  "${STAGING_ETCDIR_NATIVE}/opkg-sdk.conf"
 IPKGCONF_CANSDK =  "${STAGING_ETCDIR_NATIVE}/opkg-canadian-sdk.conf"
 
-- 
1.6.3.1




^ permalink raw reply related	[flat|nested] 16+ messages in thread

* Re: [PATCH 1/3] Make package_update_index_ipk proper task with locking
  2009-06-17 19:41 [PATCH 1/3] Make package_update_index_ipk proper task with locking Roman I Khimov
  2009-06-17 19:41 ` [PATCH 2/3] package_ipk: optimize do_package_update_index_ipk Roman I Khimov
@ 2009-06-17 21:04 ` Tom Rini
  2009-06-18 13:05 ` Richard Purdie
  2 siblings, 0 replies; 16+ messages in thread
From: Tom Rini @ 2009-06-17 21:04 UTC (permalink / raw)
  To: openembedded-devel

On Wed, Jun 17, 2009 at 11:41:58PM +0400, Roman I Khimov wrote:

> Parallel image builds race in package_update_index_ipk, so locking is
> needed and for a proper lock package_update_index_ipk needs to be a
> proper task.

Acked-by: Tom Rini <trini@embeddedalley.com>

-- 
Tom Rini



^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH 2/3] package_ipk: optimize do_package_update_index_ipk
  2009-06-17 19:41 ` [PATCH 2/3] package_ipk: optimize do_package_update_index_ipk Roman I Khimov
  2009-06-17 19:42   ` [PATCH 3/3] package_ipk: fix race in opkg.conf creation Roman I Khimov
@ 2009-06-17 21:06   ` Tom Rini
  2009-06-18  9:17   ` Phil Blundell
  2009-06-18 13:00   ` Richard Purdie
  3 siblings, 0 replies; 16+ messages in thread
From: Tom Rini @ 2009-06-17 21:06 UTC (permalink / raw)
  To: openembedded-devel

On Wed, Jun 17, 2009 at 11:41:59PM +0400, Roman I Khimov wrote:

> If there are no package updates since last index update, don't re-create
> indexes. Really helps when building something like 10 images.
> 
> Remove code duplication along the way.

Acked-by: Tom Rini <trini@embeddedalley.com>

-- 
Tom Rini



^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH 3/3] package_ipk: fix race in opkg.conf creation
  2009-06-17 19:42   ` [PATCH 3/3] package_ipk: fix race in opkg.conf creation Roman I Khimov
@ 2009-06-17 21:08     ` Tom Rini
  2009-06-18  8:49       ` Roman I Khimov
  2009-06-18 12:47     ` Richard Purdie
  1 sibling, 1 reply; 16+ messages in thread
From: Tom Rini @ 2009-06-17 21:08 UTC (permalink / raw)
  To: openembedded-devel

On Wed, Jun 17, 2009 at 11:42:00PM +0400, Roman I Khimov wrote:

> It's a very little race, but one can hit it anyway on concurrent image builds.

So, the way this works is we do cp template ${IPKGCONF_TARGET} and use
it, or so?  And IPKGCONF_SDK / IPKGCONF_CANSDK won't race because we're
cp'ing them to a final destination ?

-- 
Tom Rini



^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH 3/3] package_ipk: fix race in opkg.conf creation
  2009-06-17 21:08     ` Tom Rini
@ 2009-06-18  8:49       ` Roman I Khimov
  0 siblings, 0 replies; 16+ messages in thread
From: Roman I Khimov @ 2009-06-18  8:49 UTC (permalink / raw)
  To: openembedded-devel

On Четверг 18 июня 2009 01:08:05 Tom Rini wrote:
> On Wed, Jun 17, 2009 at 11:42:00PM +0400, Roman I Khimov wrote:
> > It's a very little race, but one can hit it anyway on concurrent image
> > builds.
>
> So, the way this works is we do cp template ${IPKGCONF_TARGET} and use
> it, or so?  

See package_generate_ipkg_conf in package_ipk.bbclass, it's echo-ing things in 
destination files sequentially.

> And IPKGCONF_SDK / IPKGCONF_CANSDK won't race because we're
> cp'ing them to a final destination ?

I've thought that we're not doing multi-sdk, so it shouldn't be a problem. 
Although if some build would need to do some images plus an SDK it might be 
invoking package_generate_ipkg_conf from several places.

Maybe we can merge it with package_update_index_ipk (or just invoke 
package_generate_ipkg_conf from there leaving the function itself)? Everything 
except package-index meta-recipe does

package_update_index_ipk
package_generate_ipkg_conf

anyway.



^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH 2/3] package_ipk: optimize do_package_update_index_ipk
  2009-06-17 19:41 ` [PATCH 2/3] package_ipk: optimize do_package_update_index_ipk Roman I Khimov
  2009-06-17 19:42   ` [PATCH 3/3] package_ipk: fix race in opkg.conf creation Roman I Khimov
  2009-06-17 21:06   ` [PATCH 2/3] package_ipk: optimize do_package_update_index_ipk Tom Rini
@ 2009-06-18  9:17   ` Phil Blundell
  2009-06-18 10:00     ` Roman I Khimov
  2009-06-18 10:11     ` Roman I Khimov
  2009-06-18 13:00   ` Richard Purdie
  3 siblings, 2 replies; 16+ messages in thread
From: Phil Blundell @ 2009-06-18  9:17 UTC (permalink / raw)
  To: openembedded-devel

On Wed, 2009-06-17 at 23:41 +0400, Roman I Khimov wrote:
> +			PACK_COUNT=`ls -t ${DEPLOY_DIR_IPK}/$ipk_path/ | head -n 4 | grep Packages | wc -l`
> +			if [ -e ${DEPLOY_DIR_IPK}/$ipk_path/ -a "$PACK_COUNT" != "4" ] ; then 

This test seems a little bit fragile.  How about something like:

NEWEST_PACKAGE=`ls -t ${DEPLOY_DIR_IPK}/$ipk_path/*.ipk | head -n 1`
if [ -n "$NEWEST_PACKAGE" ] && [ "$NEWEST_PACKAGE" -nt
"${DEPLOY_DIR_IPK}/$ipk_path/Packages" ]; then
 ...

instead?  Or, perhaps even better, teach ipkg-make-index to do this
checking for itself and just return immediately if there's no work for
it to do.

p.





^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH 2/3] package_ipk: optimize do_package_update_index_ipk
  2009-06-18  9:17   ` Phil Blundell
@ 2009-06-18 10:00     ` Roman I Khimov
  2009-06-18 10:11     ` Roman I Khimov
  1 sibling, 0 replies; 16+ messages in thread
From: Roman I Khimov @ 2009-06-18 10:00 UTC (permalink / raw)
  To: openembedded-devel

On Четверг 18 июня 2009 13:17:58 Phil Blundell wrote:
> On Wed, 2009-06-17 at 23:41 +0400, Roman I Khimov wrote:
> > +			PACK_COUNT=`ls -t ${DEPLOY_DIR_IPK}/$ipk_path/ | head -n 4 | grep
> > Packages | wc -l` +			if [ -e ${DEPLOY_DIR_IPK}/$ipk_path/ -a
> > "$PACK_COUNT" != "4" ] ; then
>
> This test seems a little bit fragile.  How about something like:
>
> NEWEST_PACKAGE=`ls -t ${DEPLOY_DIR_IPK}/$ipk_path/*.ipk | head -n 1`
> if [ -n "$NEWEST_PACKAGE" ] && [ "$NEWEST_PACKAGE" -nt
> "${DEPLOY_DIR_IPK}/$ipk_path/Packages" ]; then
>  ...
>
> instead?  

Works good here, updated patch to follow.



^ permalink raw reply	[flat|nested] 16+ messages in thread

* [PATCH 2/3] package_ipk: optimize do_package_update_index_ipk
  2009-06-18  9:17   ` Phil Blundell
  2009-06-18 10:00     ` Roman I Khimov
@ 2009-06-18 10:11     ` Roman I Khimov
  2009-06-18 10:57       ` Phil Blundell
  1 sibling, 1 reply; 16+ messages in thread
From: Roman I Khimov @ 2009-06-18 10:11 UTC (permalink / raw)
  To: openembedded-devel

If there are no package updates since last index update, don't re-create
indexes. Really helps when building something like 10 images.

Remove code duplication along the way.
---
 classes/package_ipk.bbclass |   19 +++++++------------
 1 files changed, 7 insertions(+), 12 deletions(-)

diff --git a/classes/package_ipk.bbclass b/classes/package_ipk.bbclass
index 1f1b5c9..ad84e6e 100644
--- a/classes/package_ipk.bbclass
+++ b/classes/package_ipk.bbclass
@@ -89,18 +89,13 @@ do_package_update_index_ipk () {
 	ipkg-make-index -r ${DEPLOY_DIR_IPK}/Packages -p ${DEPLOY_DIR_IPK}/Packages -l ${DEPLOY_DIR_IPK}/Packages.filelist -m ${DEPLOY_DIR_IPK}
 
 	for arch in $ipkgarchs; do
-		if [ -e ${DEPLOY_DIR_IPK}/$arch/ ] ; then 
-			touch ${DEPLOY_DIR_IPK}/$arch/Packages
-			ipkg-make-index -r ${DEPLOY_DIR_IPK}/$arch/Packages -p ${DEPLOY_DIR_IPK}/$arch/Packages -l ${DEPLOY_DIR_IPK}/$arch/Packages.filelist -m ${DEPLOY_DIR_IPK}/$arch/
-		fi
-		if [ -e ${DEPLOY_DIR_IPK}/${BUILD_ARCH}-$arch-sdk/ ] ; then 
-			touch ${DEPLOY_DIR_IPK}/${BUILD_ARCH}-$arch-sdk/Packages
-			ipkg-make-index -r ${DEPLOY_DIR_IPK}/${BUILD_ARCH}-$arch-sdk/Packages -p ${DEPLOY_DIR_IPK}/${BUILD_ARCH}-$arch-sdk/Packages -l ${DEPLOY_DIR_IPK}/${BUILD_ARCH}-$arch-sdk/Packages.filelist -m ${DEPLOY_DIR_IPK}/${BUILD_ARCH}-$arch-sdk/
-		fi
-		if [ -e ${DEPLOY_DIR_IPK}/${SDK_SYS}-sdk-$arch/ ] ; then
-			touch ${DEPLOY_DIR_IPK}/${SDK_SYS}-sdk-$arch/Packages
-			ipkg-make-index -r ${DEPLOY_DIR_IPK}/${SDK_SYS}-sdk-$arch/Packages -p ${DEPLOY_DIR_IPK}/${SDK_SYS}-sdk-$arch/Packages -l ${DEPLOY_DIR_IPK}/${SDK_SYS}-sdk-$arch/Packages.filelist -m ${DEPLOY_DIR_IPK}/${SDK_SYS}-sdk-$arch/
-		fi
+		for ipk_path in $arch ${BUILD_ARCH}-$arch-sdk ${SDK_SYS}-sdk-$arch; do
+			NEWEST_PACKAGE=`ls -t ${DEPLOY_DIR_IPK}/$ipk_path/*.ipk | head -n 1`
+			if [ -n "$NEWEST_PACKAGE" ] && [ "$NEWEST_PACKAGE" -nt "${DEPLOY_DIR_IPK}/$ipk_path/Packages" ]; then
+				touch ${DEPLOY_DIR_IPK}/$ipk_path/Packages
+				ipkg-make-index -r ${DEPLOY_DIR_IPK}/$ipk_path/Packages -p ${DEPLOY_DIR_IPK}/$ipk_path/Packages -l ${DEPLOY_DIR_IPK}/$ipk_path/Packages.filelist -m ${DEPLOY_DIR_IPK}/$ipk_path/
+			fi
+		done
 	done
 }
 
-- 
1.6.3.1




^ permalink raw reply related	[flat|nested] 16+ messages in thread

* Re: [PATCH 2/3] package_ipk: optimize do_package_update_index_ipk
  2009-06-18 10:11     ` Roman I Khimov
@ 2009-06-18 10:57       ` Phil Blundell
  0 siblings, 0 replies; 16+ messages in thread
From: Phil Blundell @ 2009-06-18 10:57 UTC (permalink / raw)
  To: openembedded-devel

On Thu, 2009-06-18 at 14:11 +0400, Roman I Khimov wrote:
> +		for ipk_path in $arch ${BUILD_ARCH}-$arch-sdk ${SDK_SYS}-sdk-$arch; do
> +			NEWEST_PACKAGE=`ls -t ${DEPLOY_DIR_IPK}/$ipk_path/*.ipk | head -n 1`
> +			if [ -n "$NEWEST_PACKAGE" ] && [ "$NEWEST_PACKAGE" -nt "${DEPLOY_DIR_IPK}/$ipk_path/Packages" ]; then
> +				touch ${DEPLOY_DIR_IPK}/$ipk_path/Packages
> +				ipkg-make-index -r ${DEPLOY_DIR_IPK}/$ipk_path/Packages -p ${DEPLOY_DIR_IPK}/$ipk_path/Packages -l ${DEPLOY_DIR_IPK}/$ipk_path/Packages.filelist -m ${DEPLOY_DIR_IPK}/$ipk_path/
> +			fi
> +		done

Thanks for the patch.  This looks good to me.

p.





^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH 3/3] package_ipk: fix race in opkg.conf creation
  2009-06-17 19:42   ` [PATCH 3/3] package_ipk: fix race in opkg.conf creation Roman I Khimov
  2009-06-17 21:08     ` Tom Rini
@ 2009-06-18 12:47     ` Richard Purdie
  1 sibling, 0 replies; 16+ messages in thread
From: Richard Purdie @ 2009-06-18 12:47 UTC (permalink / raw)
  To: openembedded-devel

On Wed, 2009-06-17 at 23:42 +0400, Roman I Khimov wrote:
> It's a very little race, but one can hit it anyway on concurrent image builds.
> ---
>  classes/package_ipk.bbclass |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/classes/package_ipk.bbclass b/classes/package_ipk.bbclass
> index d804c78..a4e6918 100644
> --- a/classes/package_ipk.bbclass
> +++ b/classes/package_ipk.bbclass
> @@ -3,7 +3,7 @@ inherit package
>  BOOTSTRAP_EXTRA_RDEPENDS += "opkg-collateral opkg"
>  IMAGE_PKGTYPE ?= "ipk"
>  
> -IPKGCONF_TARGET = "${STAGING_ETCDIR_NATIVE}/opkg.conf"
> +IPKGCONF_TARGET = "${STAGING_ETCDIR_NATIVE}/${PN}-opkg.conf"
>  IPKGCONF_SDK =  "${STAGING_ETCDIR_NATIVE}/opkg-sdk.conf"
>  IPKGCONF_CANSDK =  "${STAGING_ETCDIR_NATIVE}/opkg-canadian-sdk.conf"

Please don't do this, it will just create a ton of files all with the
same content. If you're really worried about racing, add proper locking
instead.

Cheers,

Richard





^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH 2/3] package_ipk: optimize do_package_update_index_ipk
  2009-06-17 19:41 ` [PATCH 2/3] package_ipk: optimize do_package_update_index_ipk Roman I Khimov
                     ` (2 preceding siblings ...)
  2009-06-18  9:17   ` Phil Blundell
@ 2009-06-18 13:00   ` Richard Purdie
  3 siblings, 0 replies; 16+ messages in thread
From: Richard Purdie @ 2009-06-18 13:00 UTC (permalink / raw)
  To: openembedded-devel

On Wed, 2009-06-17 at 23:41 +0400, Roman I Khimov wrote:
> If there are no package updates since last index update, don't re-create
> indexes. Really helps when building something like 10 images.

ipkg-make-index should have caching internally such that it only
rebuilds the segments of packages that have changed. If this patch makes
a significant difference, we should fix ipkg-make-index.

Also, just because the number of .ipk files matches the number of files
in the index it doesn't mean nothing changed (think about a package
being rebuilt and having just its dependencies changes).

Finally, using date stamps also doesn't work with packaged staging.

So a definite NAK I'm afraid, please fix ipkg-make-index.

Richard





^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH 1/3] Make package_update_index_ipk proper task with locking
  2009-06-17 19:41 [PATCH 1/3] Make package_update_index_ipk proper task with locking Roman I Khimov
  2009-06-17 19:41 ` [PATCH 2/3] package_ipk: optimize do_package_update_index_ipk Roman I Khimov
  2009-06-17 21:04 ` [PATCH 1/3] Make package_update_index_ipk proper task with locking Tom Rini
@ 2009-06-18 13:05 ` Richard Purdie
  2009-06-18 13:16   ` Phil Blundell
  2 siblings, 1 reply; 16+ messages in thread
From: Richard Purdie @ 2009-06-18 13:05 UTC (permalink / raw)
  To: openembedded-devel

On Wed, 2009-06-17 at 23:41 +0400, Roman I Khimov wrote:
> Parallel image builds race in package_update_index_ipk, so locking is
> needed and for a proper lock package_update_index_ipk needs to be a
> proper task.

Can I suggest we actually add the ability to lock from shell scripts? A
couple of python scripts (bitbake-lock/bitbake-unlock) which just call
the python locking functions should be trivial to write and then mean we
can add sane locking to specialist functions much more easily. It should
allow us to solve the opkg.conf file issue you were seeing easily too.

Adding those shell scripts is slightly trickt - I'm happy to have them
in bitbake but we'll need another release before OE.dev can depend on
use them. We're overdue a bitbake release so perhaps we can get this
done quite quickly though...

Cheers,

Richard




^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH 1/3] Make package_update_index_ipk proper task with locking
  2009-06-18 13:05 ` Richard Purdie
@ 2009-06-18 13:16   ` Phil Blundell
  2009-06-18 13:35     ` Richard Purdie
  0 siblings, 1 reply; 16+ messages in thread
From: Phil Blundell @ 2009-06-18 13:16 UTC (permalink / raw)
  To: openembedded-devel; +Cc: openembedded-devel

On Thu, 2009-06-18 at 14:05 +0100, Richard Purdie wrote:
> Can I suggest we actually add the ability to lock from shell scripts? A
> couple of python scripts (bitbake-lock/bitbake-unlock) which just call
> the python locking functions should be trivial to write and then mean we
> can add sane locking to specialist functions much more easily. It should
> allow us to solve the opkg.conf file issue you were seeing easily too.
> 
> Adding those shell scripts is slightly trickt - I'm happy to have them
> in bitbake but we'll need another release before OE.dev can depend on
> use them. We're overdue a bitbake release so perhaps we can get this
> done quite quickly though...

In this particular case there probably isn't any need for the locks to
be shared between the shell and python worlds since all the ipk handling
is done in shell.  So I guess it should be fine to just have the shell
code call something like flock(2) directly, and hence no need for a new
bitbake release.

If the issue is just that concurrent calls to ipkg-make-index will
collide, it's probably as simple as replacing the call to
ipkg-make-index with "flock some-path -c ipkg-make-index ...".

p.





^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH 1/3] Make package_update_index_ipk proper task with locking
  2009-06-18 13:16   ` Phil Blundell
@ 2009-06-18 13:35     ` Richard Purdie
  0 siblings, 0 replies; 16+ messages in thread
From: Richard Purdie @ 2009-06-18 13:35 UTC (permalink / raw)
  To: openembedded-devel

On Thu, 2009-06-18 at 14:16 +0100, Phil Blundell wrote: 
> On Thu, 2009-06-18 at 14:05 +0100, Richard Purdie wrote:
> > Can I suggest we actually add the ability to lock from shell scripts? A
> > couple of python scripts (bitbake-lock/bitbake-unlock) which just call
> > the python locking functions should be trivial to write and then mean we
> > can add sane locking to specialist functions much more easily. It should
> > allow us to solve the opkg.conf file issue you were seeing easily too.
> > 
> > Adding those shell scripts is slightly trickt - I'm happy to have them
> > in bitbake but we'll need another release before OE.dev can depend on
> > use them. We're overdue a bitbake release so perhaps we can get this
> > done quite quickly though...
> 
> In this particular case there probably isn't any need for the locks to
> be shared between the shell and python worlds since all the ipk handling
> is done in shell.  So I guess it should be fine to just have the shell
> code call something like flock(2) directly, and hence no need for a new
> bitbake release.
> 
> If the issue is just that concurrent calls to ipkg-make-index will
> collide, it's probably as simple as replacing the call to
> ipkg-make-index with "flock some-path -c ipkg-make-index ...".

Agreed, it just somehow seemed nicer to use the same locking functions
for both and its very simple to make it happen at a code level, just the
logistics are tricky.

I probably will add locking scripts to bitbake in case we do want it but
there are other ways simpler ways to solve this particular problem.

Cheers,

Richard




^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2009-06-18 13:47 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-06-17 19:41 [PATCH 1/3] Make package_update_index_ipk proper task with locking Roman I Khimov
2009-06-17 19:41 ` [PATCH 2/3] package_ipk: optimize do_package_update_index_ipk Roman I Khimov
2009-06-17 19:42   ` [PATCH 3/3] package_ipk: fix race in opkg.conf creation Roman I Khimov
2009-06-17 21:08     ` Tom Rini
2009-06-18  8:49       ` Roman I Khimov
2009-06-18 12:47     ` Richard Purdie
2009-06-17 21:06   ` [PATCH 2/3] package_ipk: optimize do_package_update_index_ipk Tom Rini
2009-06-18  9:17   ` Phil Blundell
2009-06-18 10:00     ` Roman I Khimov
2009-06-18 10:11     ` Roman I Khimov
2009-06-18 10:57       ` Phil Blundell
2009-06-18 13:00   ` Richard Purdie
2009-06-17 21:04 ` [PATCH 1/3] Make package_update_index_ipk proper task with locking Tom Rini
2009-06-18 13:05 ` Richard Purdie
2009-06-18 13:16   ` Phil Blundell
2009-06-18 13:35     ` Richard Purdie

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.