[PATCH] rootfs_ipk: allow do_populate_sdk in parallel to do_rootfs

[PATCH] rootfs_ipk: allow do_populate_sdk in parallel to do_rootfs

[Michael Ho]

From: Michael Ho <Michael.Ho@bmw.de>

Switch do_populate_sdk for the ipk package manager to use a separate target
opkg config file and separate the lockfiles restricting do_rootfs and
do_populate_sdk from running in parallel.

This way if an image recipe includes a dependency to do_populate_sdk by
default then it will run in parallel to do_rootfs saving time compared to the
sequential execution.

Signed-off-by: Michael Ho <Michael.Ho@bmw.de>
---
 meta/classes/package_ipk.bbclass       | 1 +
 meta/classes/rootfs_ipk.bbclass        | 4 ++--
 meta/lib/oe/package_manager/ipk/sdk.py | 6 ++++++
 3 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/meta/classes/package_ipk.bbclass b/meta/classes/package_ipk.bbclass
index 79cb36c..d31dba0 100644
--- a/meta/classes/package_ipk.bbclass
+++ b/meta/classes/package_ipk.bbclass
@@ -4,6 +4,7 @@ IMAGE_PKGTYPE ?= "ipk"
 
 IPKGCONF_TARGET = "${WORKDIR}/opkg.conf"
 IPKGCONF_SDK =  "${WORKDIR}/opkg-sdk.conf"
+IPKGCONF_SDK_TARGET = "${WORKDIR}/opkg-sdk-target.conf"
 
 PKGWRITEDIRIPK = "${WORKDIR}/deploy-ipks"
 
diff --git a/meta/classes/rootfs_ipk.bbclass b/meta/classes/rootfs_ipk.bbclass
index f1e0219..245c256 100644
--- a/meta/classes/rootfs_ipk.bbclass
+++ b/meta/classes/rootfs_ipk.bbclass
@@ -14,8 +14,8 @@ do_rootfs[recrdeptask] += "do_package_write_ipk do_package_qa"
 do_rootfs[vardeps] += "PACKAGE_FEED_URIS PACKAGE_FEED_BASE_PATHS PACKAGE_FEED_ARCHS"
 
 do_rootfs[lockfiles] += "${WORKDIR}/ipk.lock"
-do_populate_sdk[lockfiles] += "${WORKDIR}/ipk.lock"
-do_populate_sdk_ext[lockfiles] += "${WORKDIR}/ipk.lock"
+do_populate_sdk[lockfiles] += "${WORKDIR}/sdk-ipk.lock"
+do_populate_sdk_ext[lockfiles] += "${WORKDIR}/sdk-ipk.lock"
 
 OPKG_PREPROCESS_COMMANDS = ""
 
diff --git a/meta/lib/oe/package_manager/ipk/sdk.py b/meta/lib/oe/package_manager/ipk/sdk.py
index 37af034..e2ca415 100644
--- a/meta/lib/oe/package_manager/ipk/sdk.py
+++ b/meta/lib/oe/package_manager/ipk/sdk.py
@@ -14,6 +14,12 @@ class PkgSdk(Sdk):
     def __init__(self, d, manifest_dir=None):
         super(PkgSdk, self).__init__(d, manifest_dir)
 
+        # In sdk_list_installed_packages the call to opkg is hardcoded to
+        # always use IPKGCONF_TARGET and there's no exposed API to change this
+        # so simply override IPKGCONF_TARGET to use this separated config file.
+        ipkgconf_sdk_target = d.getVar("IPKGCONF_SDK_TARGET")
+        d.setVar("IPKGCONF_TARGET", ipkgconf_sdk_target)
+
         self.target_conf = self.d.getVar("IPKGCONF_TARGET")
         self.host_conf = self.d.getVar("IPKGCONF_SDK")
 
-- 
2.7.4

Re: [OE-core] [PATCH] rootfs_ipk: allow do_populate_sdk in parallel to do_rootfs

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 2914 bytes --]

Does deb/rpm packaging already have this feature?

Alex

On Mon, 11 Jan 2021 at 13:45, Michael Ho <Michael.Ho@bmw.de> wrote:

> From: Michael Ho <Michael.Ho@bmw.de>
>
> Switch do_populate_sdk for the ipk package manager to use a separate target
> opkg config file and separate the lockfiles restricting do_rootfs and
> do_populate_sdk from running in parallel.
>
> This way if an image recipe includes a dependency to do_populate_sdk by
> default then it will run in parallel to do_rootfs saving time compared to
> the
> sequential execution.
>
> Signed-off-by: Michael Ho <Michael.Ho@bmw.de>
> ---
>  meta/classes/package_ipk.bbclass       | 1 +
>  meta/classes/rootfs_ipk.bbclass        | 4 ++--
>  meta/lib/oe/package_manager/ipk/sdk.py | 6 ++++++
>  3 files changed, 9 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/package_ipk.bbclass
> b/meta/classes/package_ipk.bbclass
> index 79cb36c..d31dba0 100644
> --- a/meta/classes/package_ipk.bbclass
> +++ b/meta/classes/package_ipk.bbclass
> @@ -4,6 +4,7 @@ IMAGE_PKGTYPE ?= "ipk"
>
>  IPKGCONF_TARGET = "${WORKDIR}/opkg.conf"
>  IPKGCONF_SDK =  "${WORKDIR}/opkg-sdk.conf"
> +IPKGCONF_SDK_TARGET = "${WORKDIR}/opkg-sdk-target.conf"
>
>  PKGWRITEDIRIPK = "${WORKDIR}/deploy-ipks"
>
> diff --git a/meta/classes/rootfs_ipk.bbclass
> b/meta/classes/rootfs_ipk.bbclass
> index f1e0219..245c256 100644
> --- a/meta/classes/rootfs_ipk.bbclass
> +++ b/meta/classes/rootfs_ipk.bbclass
> @@ -14,8 +14,8 @@ do_rootfs[recrdeptask] += "do_package_write_ipk
> do_package_qa"
>  do_rootfs[vardeps] += "PACKAGE_FEED_URIS PACKAGE_FEED_BASE_PATHS
> PACKAGE_FEED_ARCHS"
>
>  do_rootfs[lockfiles] += "${WORKDIR}/ipk.lock"
> -do_populate_sdk[lockfiles] += "${WORKDIR}/ipk.lock"
> -do_populate_sdk_ext[lockfiles] += "${WORKDIR}/ipk.lock"
> +do_populate_sdk[lockfiles] += "${WORKDIR}/sdk-ipk.lock"
> +do_populate_sdk_ext[lockfiles] += "${WORKDIR}/sdk-ipk.lock"
>
>  OPKG_PREPROCESS_COMMANDS = ""
>
> diff --git a/meta/lib/oe/package_manager/ipk/sdk.py
> b/meta/lib/oe/package_manager/ipk/sdk.py
> index 37af034..e2ca415 100644
> --- a/meta/lib/oe/package_manager/ipk/sdk.py
> +++ b/meta/lib/oe/package_manager/ipk/sdk.py
> @@ -14,6 +14,12 @@ class PkgSdk(Sdk):
>      def __init__(self, d, manifest_dir=None):
>          super(PkgSdk, self).__init__(d, manifest_dir)
>
> +        # In sdk_list_installed_packages the call to opkg is hardcoded to
> +        # always use IPKGCONF_TARGET and there's no exposed API to change
> this
> +        # so simply override IPKGCONF_TARGET to use this separated config
> file.
> +        ipkgconf_sdk_target = d.getVar("IPKGCONF_SDK_TARGET")
> +        d.setVar("IPKGCONF_TARGET", ipkgconf_sdk_target)
> +
>          self.target_conf = self.d.getVar("IPKGCONF_TARGET")
>          self.host_conf = self.d.getVar("IPKGCONF_SDK")
>
> --
> 2.7.4
>
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 3700 bytes --]

Re: [OE-core] [PATCH] rootfs_ipk: allow do_populate_sdk in parallel to do_rootfs

[Michael Ho]

[-- Attachment #1: Type: text/plain, Size: 3995 bytes --]

Hello Alexander,

Did a quick test and it seems the rpm packaging classes don’t have this issue. The rootfs_deb bbclass however has the same lock files like with rootfs_ipk.

Should I try to patch this too to keep them aligned?

Kind regards,
Michael

--
BMW Car IT GmbH
Michael Ho
Spezialist Entwicklung – Build and Release Engineering
Lise-Meitner-Straße 14
89081 Ulm

Tel.: ­+49-731-37804-071
Mobil: +49-152-54980-471
Fax: +49-731-37804-001
Mail: michael.ho@bmw-carit.de<mailto:michael.ho@bmw-carit.de>
Web: http://www.bmw-carit.de<http://www.bmw-carit.de/>
-------------------------------------------------------------------------
BMW Car IT GmbH
Geschäftsführer: Kai-Uwe Balszuweit und Michael Böttrich
Sitz und Registergericht: München HRB 134810
-------------------------------------------------------------------------


From: Alexander Kanavin <alex.kanavin@gmail.com>
Date: Monday, 11. January 2021 at 13:46
To: "Ho Michael, JC-3UL" <Michael.Ho@bmw.de>
Cc: OE-core <openembedded-core@lists.openembedded.org>
Subject: Re: [OE-core] [PATCH] rootfs_ipk: allow do_populate_sdk in parallel to do_rootfs

Does deb/rpm packaging already have this feature?

Alex

On Mon, 11 Jan 2021 at 13:45, Michael Ho <Michael.Ho@bmw.de<mailto:Michael.Ho@bmw.de>> wrote:
From: Michael Ho <Michael.Ho@bmw.de<mailto:Michael.Ho@bmw.de>>

Switch do_populate_sdk for the ipk package manager to use a separate target
opkg config file and separate the lockfiles restricting do_rootfs and
do_populate_sdk from running in parallel.

This way if an image recipe includes a dependency to do_populate_sdk by
default then it will run in parallel to do_rootfs saving time compared to the
sequential execution.

Signed-off-by: Michael Ho <Michael.Ho@bmw.de<mailto:Michael.Ho@bmw.de>>
---
 meta/classes/package_ipk.bbclass       | 1 +
 meta/classes/rootfs_ipk.bbclass        | 4 ++--
 meta/lib/oe/package_manager/ipk/sdk.py | 6 ++++++
 3 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/meta/classes/package_ipk.bbclass b/meta/classes/package_ipk.bbclass
index 79cb36c..d31dba0 100644
--- a/meta/classes/package_ipk.bbclass
+++ b/meta/classes/package_ipk.bbclass
@@ -4,6 +4,7 @@ IMAGE_PKGTYPE ?= "ipk"

 IPKGCONF_TARGET = "${WORKDIR}/opkg.conf"
 IPKGCONF_SDK =  "${WORKDIR}/opkg-sdk.conf"
+IPKGCONF_SDK_TARGET = "${WORKDIR}/opkg-sdk-target.conf"

 PKGWRITEDIRIPK = "${WORKDIR}/deploy-ipks"

diff --git a/meta/classes/rootfs_ipk.bbclass b/meta/classes/rootfs_ipk.bbclass
index f1e0219..245c256 100644
--- a/meta/classes/rootfs_ipk.bbclass
+++ b/meta/classes/rootfs_ipk.bbclass
@@ -14,8 +14,8 @@ do_rootfs[recrdeptask] += "do_package_write_ipk do_package_qa"
 do_rootfs[vardeps] += "PACKAGE_FEED_URIS PACKAGE_FEED_BASE_PATHS PACKAGE_FEED_ARCHS"

 do_rootfs[lockfiles] += "${WORKDIR}/ipk.lock"
-do_populate_sdk[lockfiles] += "${WORKDIR}/ipk.lock"
-do_populate_sdk_ext[lockfiles] += "${WORKDIR}/ipk.lock"
+do_populate_sdk[lockfiles] += "${WORKDIR}/sdk-ipk.lock"
+do_populate_sdk_ext[lockfiles] += "${WORKDIR}/sdk-ipk.lock"

 OPKG_PREPROCESS_COMMANDS = ""

diff --git a/meta/lib/oe/package_manager/ipk/sdk.py b/meta/lib/oe/package_manager/ipk/sdk.py
index 37af034..e2ca415 100644
--- a/meta/lib/oe/package_manager/ipk/sdk.py
+++ b/meta/lib/oe/package_manager/ipk/sdk.py
@@ -14,6 +14,12 @@ class PkgSdk(Sdk):
     def __init__(self, d, manifest_dir=None):
         super(PkgSdk, self).__init__(d, manifest_dir)

+        # In sdk_list_installed_packages the call to opkg is hardcoded to
+        # always use IPKGCONF_TARGET and there's no exposed API to change this
+        # so simply override IPKGCONF_TARGET to use this separated config file.
+        ipkgconf_sdk_target = d.getVar("IPKGCONF_SDK_TARGET")
+        d.setVar("IPKGCONF_TARGET", ipkgconf_sdk_target)
+
         self.target_conf = self.d.getVar("IPKGCONF_TARGET")
         self.host_conf = self.d.getVar("IPKGCONF_SDK")

--
2.7.4




[-- Attachment #2: Type: text/html, Size: 10052 bytes --]

Re: [OE-core] [PATCH] rootfs_ipk: allow do_populate_sdk in parallel to do_rootfs

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 4375 bytes --]

If you can fix the deb packaging easily, then please do so, but it is not
mandatory, and don't spend too much time if it turns out to be a time sink.

Alex

On Mon, 11 Jan 2021 at 15:01, <Michael.Ho@bmw.de> wrote:

> Hello Alexander,
>
>
>
> Did a quick test and it seems the rpm packaging classes don’t have this
> issue. The rootfs_deb bbclass however has the same lock files like with
> rootfs_ipk.
>
> Should I try to patch this too to keep them aligned?
>
>
>
> Kind regards,
>
> Michael
>
>
>
> --
>
> *BMW Car IT GmbH*
> Michael Ho
> Spezialist Entwicklung – Build and Release Engineering
> Lise-Meitner-Straße 14
> 89081 Ulm
>
> Tel.: ­+49-731-37804-071
>
> Mobil: +49-152-54980-471
> Fax: +49-731-37804-001
> Mail: michael.ho@bmw-carit.de
> Web: http://www.bmw-carit.de
> -------------------------------------------------------------------------
> BMW Car IT GmbH
> Geschäftsführer: Kai-Uwe Balszuweit und Michael Böttrich
> Sitz und Registergericht: München HRB 134810
> -------------------------------------------------------------------------
>
>
>
>
>
> *From: *Alexander Kanavin <alex.kanavin@gmail.com>
> *Date: *Monday, 11. January 2021 at 13:46
> *To: *"Ho Michael, JC-3UL" <Michael.Ho@bmw.de>
> *Cc: *OE-core <openembedded-core@lists.openembedded.org>
> *Subject: *Re: [OE-core] [PATCH] rootfs_ipk: allow do_populate_sdk in
> parallel to do_rootfs
>
>
>
> Does deb/rpm packaging already have this feature?
>
>
>
> Alex
>
>
>
> On Mon, 11 Jan 2021 at 13:45, Michael Ho <Michael.Ho@bmw.de> wrote:
>
> From: Michael Ho <Michael.Ho@bmw.de>
>
> Switch do_populate_sdk for the ipk package manager to use a separate target
> opkg config file and separate the lockfiles restricting do_rootfs and
> do_populate_sdk from running in parallel.
>
> This way if an image recipe includes a dependency to do_populate_sdk by
> default then it will run in parallel to do_rootfs saving time compared to
> the
> sequential execution.
>
> Signed-off-by: Michael Ho <Michael.Ho@bmw.de>
> ---
>  meta/classes/package_ipk.bbclass       | 1 +
>  meta/classes/rootfs_ipk.bbclass        | 4 ++--
>  meta/lib/oe/package_manager/ipk/sdk.py | 6 ++++++
>  3 files changed, 9 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/package_ipk.bbclass
> b/meta/classes/package_ipk.bbclass
> index 79cb36c..d31dba0 100644
> --- a/meta/classes/package_ipk.bbclass
> +++ b/meta/classes/package_ipk.bbclass
> @@ -4,6 +4,7 @@ IMAGE_PKGTYPE ?= "ipk"
>
>  IPKGCONF_TARGET = "${WORKDIR}/opkg.conf"
>  IPKGCONF_SDK =  "${WORKDIR}/opkg-sdk.conf"
> +IPKGCONF_SDK_TARGET = "${WORKDIR}/opkg-sdk-target.conf"
>
>  PKGWRITEDIRIPK = "${WORKDIR}/deploy-ipks"
>
> diff --git a/meta/classes/rootfs_ipk.bbclass
> b/meta/classes/rootfs_ipk.bbclass
> index f1e0219..245c256 100644
> --- a/meta/classes/rootfs_ipk.bbclass
> +++ b/meta/classes/rootfs_ipk.bbclass
> @@ -14,8 +14,8 @@ do_rootfs[recrdeptask] += "do_package_write_ipk
> do_package_qa"
>  do_rootfs[vardeps] += "PACKAGE_FEED_URIS PACKAGE_FEED_BASE_PATHS
> PACKAGE_FEED_ARCHS"
>
>  do_rootfs[lockfiles] += "${WORKDIR}/ipk.lock"
> -do_populate_sdk[lockfiles] += "${WORKDIR}/ipk.lock"
> -do_populate_sdk_ext[lockfiles] += "${WORKDIR}/ipk.lock"
> +do_populate_sdk[lockfiles] += "${WORKDIR}/sdk-ipk.lock"
> +do_populate_sdk_ext[lockfiles] += "${WORKDIR}/sdk-ipk.lock"
>
>  OPKG_PREPROCESS_COMMANDS = ""
>
> diff --git a/meta/lib/oe/package_manager/ipk/sdk.py
> b/meta/lib/oe/package_manager/ipk/sdk.py
> index 37af034..e2ca415 100644
> --- a/meta/lib/oe/package_manager/ipk/sdk.py
> +++ b/meta/lib/oe/package_manager/ipk/sdk.py
> @@ -14,6 +14,12 @@ class PkgSdk(Sdk):
>      def __init__(self, d, manifest_dir=None):
>          super(PkgSdk, self).__init__(d, manifest_dir)
>
> +        # In sdk_list_installed_packages the call to opkg is hardcoded to
> +        # always use IPKGCONF_TARGET and there's no exposed API to change
> this
> +        # so simply override IPKGCONF_TARGET to use this separated config
> file.
> +        ipkgconf_sdk_target = d.getVar("IPKGCONF_SDK_TARGET")
> +        d.setVar("IPKGCONF_TARGET", ipkgconf_sdk_target)
> +
>          self.target_conf = self.d.getVar("IPKGCONF_TARGET")
>          self.host_conf = self.d.getVar("IPKGCONF_SDK")
>
> --
> 2.7.4
>
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 9536 bytes --]

Re: [OE-core] [PATCH] rootfs_ipk: allow do_populate_sdk in parallel to do_rootfs

[Michael Ho]

[-- Attachment #1: Type: text/plain, Size: 6284 bytes --]

Hello Alex,

Seems to not be so quick to change for the deb packages. First I had issues building the sdk on
the master branch as it runs some postinst scripts that leak out of the yocto sandbox and into
the host system [1]. Not sure if that’s a recent regression as I don’t normally use deb packages
in Yocto.

Then I found that apt will fight over a lock file written into the recipe-sysroots-native directory [2].
I am not so familiar with apt-get to resolve that trivially. I’ll try to re-visit it another day.

Kind regards,
Michael

[1]
Setting up nativesdk-shadow (4.8.1-r0) ...
mkdir: cannot create directory ‘/usr/lib/opkg’: Permission denied
dpkg: error processing package nativesdk-shadow (--configure):
 installed nativesdk-shadow package post-installation script subprocess returned error exit status 1

[2]
E: Could not get lock /nvme/poky/build-deb/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot-native/var/cache/apt/archives/lock - open (11: Resource temporarily unavailable)

--
BMW Car IT GmbH
Michael Ho
Spezialist Entwicklung – Build and Release Engineering
Lise-Meitner-Straße 14
89081 Ulm

Tel.: ­+49-731-37804-071
Mobil: +49-152-54980-471
Fax: +49-731-37804-001
Mail: michael.ho@bmw-carit.de<mailto:michael.ho@bmw-carit.de>
Web: http://www.bmw-carit.de<http://www.bmw-carit.de/>
-------------------------------------------------------------------------
BMW Car IT GmbH
Geschäftsführer: Kai-Uwe Balszuweit und Michael Böttrich
Sitz und Registergericht: München HRB 134810
-------------------------------------------------------------------------


From: Alexander Kanavin <alex.kanavin@gmail.com>
Date: Monday, 11. January 2021 at 15:45
To: "Ho Michael, JC-3UL" <Michael.Ho@bmw.de>
Cc: OE-core <openembedded-core@lists.openembedded.org>
Subject: Re: [OE-core] [PATCH] rootfs_ipk: allow do_populate_sdk in parallel to do_rootfs

If you can fix the deb packaging easily, then please do so, but it is not mandatory, and don't spend too much time if it turns out to be a time sink.

Alex

On Mon, 11 Jan 2021 at 15:01, <Michael.Ho@bmw.de<mailto:Michael.Ho@bmw.de>> wrote:
Hello Alexander,

Did a quick test and it seems the rpm packaging classes don’t have this issue. The rootfs_deb bbclass however has the same lock files like with rootfs_ipk.
Should I try to patch this too to keep them aligned?

Kind regards,
Michael

--
BMW Car IT GmbH
Michael Ho
Spezialist Entwicklung – Build and Release Engineering
Lise-Meitner-Straße 14
89081 Ulm

Tel.: ­+49-731-37804-071
Mobil: +49-152-54980-471
Fax: +49-731-37804-001
Mail: michael.ho@bmw-carit.de<mailto:michael.ho@bmw-carit.de>
Web: http://www.bmw-carit.de<http://www.bmw-carit.de/>
-------------------------------------------------------------------------
BMW Car IT GmbH
Geschäftsführer: Kai-Uwe Balszuweit und Michael Böttrich
Sitz und Registergericht: München HRB 134810
-------------------------------------------------------------------------


From: Alexander Kanavin <alex.kanavin@gmail.com<mailto:alex.kanavin@gmail.com>>
Date: Monday, 11. January 2021 at 13:46
To: "Ho Michael, JC-3UL" <Michael.Ho@bmw.de<mailto:Michael.Ho@bmw.de>>
Cc: OE-core <openembedded-core@lists.openembedded.org<mailto:openembedded-core@lists.openembedded.org>>
Subject: Re: [OE-core] [PATCH] rootfs_ipk: allow do_populate_sdk in parallel to do_rootfs

Does deb/rpm packaging already have this feature?

Alex

On Mon, 11 Jan 2021 at 13:45, Michael Ho <Michael.Ho@bmw.de<mailto:Michael.Ho@bmw.de>> wrote:
From: Michael Ho <Michael.Ho@bmw.de<mailto:Michael.Ho@bmw.de>>

Switch do_populate_sdk for the ipk package manager to use a separate target
opkg config file and separate the lockfiles restricting do_rootfs and
do_populate_sdk from running in parallel.

This way if an image recipe includes a dependency to do_populate_sdk by
default then it will run in parallel to do_rootfs saving time compared to the
sequential execution.

Signed-off-by: Michael Ho <Michael.Ho@bmw.de<mailto:Michael.Ho@bmw.de>>
---
 meta/classes/package_ipk.bbclass       | 1 +
 meta/classes/rootfs_ipk.bbclass        | 4 ++--
 meta/lib/oe/package_manager/ipk/sdk.py | 6 ++++++
 3 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/meta/classes/package_ipk.bbclass b/meta/classes/package_ipk.bbclass
index 79cb36c..d31dba0 100644
--- a/meta/classes/package_ipk.bbclass
+++ b/meta/classes/package_ipk.bbclass
@@ -4,6 +4,7 @@ IMAGE_PKGTYPE ?= "ipk"

 IPKGCONF_TARGET = "${WORKDIR}/opkg.conf"
 IPKGCONF_SDK =  "${WORKDIR}/opkg-sdk.conf"
+IPKGCONF_SDK_TARGET = "${WORKDIR}/opkg-sdk-target.conf"

 PKGWRITEDIRIPK = "${WORKDIR}/deploy-ipks"

diff --git a/meta/classes/rootfs_ipk.bbclass b/meta/classes/rootfs_ipk.bbclass
index f1e0219..245c256 100644
--- a/meta/classes/rootfs_ipk.bbclass
+++ b/meta/classes/rootfs_ipk.bbclass
@@ -14,8 +14,8 @@ do_rootfs[recrdeptask] += "do_package_write_ipk do_package_qa"
 do_rootfs[vardeps] += "PACKAGE_FEED_URIS PACKAGE_FEED_BASE_PATHS PACKAGE_FEED_ARCHS"

 do_rootfs[lockfiles] += "${WORKDIR}/ipk.lock"
-do_populate_sdk[lockfiles] += "${WORKDIR}/ipk.lock"
-do_populate_sdk_ext[lockfiles] += "${WORKDIR}/ipk.lock"
+do_populate_sdk[lockfiles] += "${WORKDIR}/sdk-ipk.lock"
+do_populate_sdk_ext[lockfiles] += "${WORKDIR}/sdk-ipk.lock"

 OPKG_PREPROCESS_COMMANDS = ""

diff --git a/meta/lib/oe/package_manager/ipk/sdk.py b/meta/lib/oe/package_manager/ipk/sdk.py
index 37af034..e2ca415 100644
--- a/meta/lib/oe/package_manager/ipk/sdk.py
+++ b/meta/lib/oe/package_manager/ipk/sdk.py
@@ -14,6 +14,12 @@ class PkgSdk(Sdk):
     def __init__(self, d, manifest_dir=None):
         super(PkgSdk, self).__init__(d, manifest_dir)

+        # In sdk_list_installed_packages the call to opkg is hardcoded to
+        # always use IPKGCONF_TARGET and there's no exposed API to change this
+        # so simply override IPKGCONF_TARGET to use this separated config file.
+        ipkgconf_sdk_target = d.getVar("IPKGCONF_SDK_TARGET")
+        d.setVar("IPKGCONF_TARGET", ipkgconf_sdk_target)
+
         self.target_conf = self.d.getVar("IPKGCONF_TARGET")
         self.host_conf = self.d.getVar("IPKGCONF_SDK")

--
2.7.4




[-- Attachment #2: Type: text/html, Size: 18258 bytes --]

Re: [OE-core] [PATCH] rootfs_ipk: allow do_populate_sdk in parallel to do_rootfs

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 6725 bytes --]

Right, then don't bother with deb. The level of support for deb in Yocto is
significantly less than for rpm/ipk, and we do not require feature parity
for all three.

Thanks,
Alex

On Mon, 11 Jan 2021 at 17:25, <Michael.Ho@bmw.de> wrote:

> Hello Alex,
>
>
>
> Seems to not be so quick to change for the deb packages. First I had
> issues building the sdk on
> the master branch as it runs some postinst scripts that leak out of the
> yocto sandbox and into
> the host system [1]. Not sure if that’s a recent regression as I don’t
> normally use deb packages
> in Yocto.
>
>
>
> Then I found that apt will fight over a lock file written into the
> recipe-sysroots-native directory [2].
> I am not so familiar with apt-get to resolve that trivially. I’ll try to
> re-visit it another day.
>
>
>
> Kind regards,
>
> Michael
>
>
>
> [1]
>
> Setting up nativesdk-shadow (4.8.1-r0) ...
>
> mkdir: cannot create directory ‘/usr/lib/opkg’: Permission denied
>
> dpkg: error processing package nativesdk-shadow (--configure):
>
>  installed nativesdk-shadow package post-installation script subprocess
> returned error exit status 1
>
>
>
> [2]
>
> E: Could not get lock
> /nvme/poky/build-deb/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot-native/var/cache/apt/archives/lock
> - open (11: Resource temporarily unavailable)
>
>
>
> --
>
> *BMW Car IT GmbH*
> Michael Ho
> Spezialist Entwicklung – Build and Release Engineering
> Lise-Meitner-Straße 14
> 89081 Ulm
>
> Tel.: ­+49-731-37804-071
>
> Mobil: +49-152-54980-471
> Fax: +49-731-37804-001
> Mail: michael.ho@bmw-carit.de
> Web: http://www.bmw-carit.de
> -------------------------------------------------------------------------
> BMW Car IT GmbH
> Geschäftsführer: Kai-Uwe Balszuweit und Michael Böttrich
> Sitz und Registergericht: München HRB 134810
> -------------------------------------------------------------------------
>
>
>
>
>
> *From: *Alexander Kanavin <alex.kanavin@gmail.com>
> *Date: *Monday, 11. January 2021 at 15:45
> *To: *"Ho Michael, JC-3UL" <Michael.Ho@bmw.de>
> *Cc: *OE-core <openembedded-core@lists.openembedded.org>
> *Subject: *Re: [OE-core] [PATCH] rootfs_ipk: allow do_populate_sdk in
> parallel to do_rootfs
>
>
>
> If you can fix the deb packaging easily, then please do so, but it is not
> mandatory, and don't spend too much time if it turns out to be a time sink.
>
>
>
> Alex
>
>
>
> On Mon, 11 Jan 2021 at 15:01, <Michael.Ho@bmw.de> wrote:
>
> Hello Alexander,
>
>
>
> Did a quick test and it seems the rpm packaging classes don’t have this
> issue. The rootfs_deb bbclass however has the same lock files like with
> rootfs_ipk.
>
> Should I try to patch this too to keep them aligned?
>
>
>
> Kind regards,
>
> Michael
>
>
>
> --
>
> *BMW Car IT GmbH*
> Michael Ho
> Spezialist Entwicklung – Build and Release Engineering
> Lise-Meitner-Straße 14
> 89081 Ulm
>
> Tel.: ­+49-731-37804-071
>
> Mobil: +49-152-54980-471
> Fax: +49-731-37804-001
> Mail: michael.ho@bmw-carit.de
> Web: http://www.bmw-carit.de
> -------------------------------------------------------------------------
> BMW Car IT GmbH
> Geschäftsführer: Kai-Uwe Balszuweit und Michael Böttrich
> Sitz und Registergericht: München HRB 134810
> -------------------------------------------------------------------------
>
>
>
>
>
> *From: *Alexander Kanavin <alex.kanavin@gmail.com>
> *Date: *Monday, 11. January 2021 at 13:46
> *To: *"Ho Michael, JC-3UL" <Michael.Ho@bmw.de>
> *Cc: *OE-core <openembedded-core@lists.openembedded.org>
> *Subject: *Re: [OE-core] [PATCH] rootfs_ipk: allow do_populate_sdk in
> parallel to do_rootfs
>
>
>
> Does deb/rpm packaging already have this feature?
>
>
>
> Alex
>
>
>
> On Mon, 11 Jan 2021 at 13:45, Michael Ho <Michael.Ho@bmw.de> wrote:
>
> From: Michael Ho <Michael.Ho@bmw.de>
>
> Switch do_populate_sdk for the ipk package manager to use a separate target
> opkg config file and separate the lockfiles restricting do_rootfs and
> do_populate_sdk from running in parallel.
>
> This way if an image recipe includes a dependency to do_populate_sdk by
> default then it will run in parallel to do_rootfs saving time compared to
> the
> sequential execution.
>
> Signed-off-by: Michael Ho <Michael.Ho@bmw.de>
> ---
>  meta/classes/package_ipk.bbclass       | 1 +
>  meta/classes/rootfs_ipk.bbclass        | 4 ++--
>  meta/lib/oe/package_manager/ipk/sdk.py | 6 ++++++
>  3 files changed, 9 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/package_ipk.bbclass
> b/meta/classes/package_ipk.bbclass
> index 79cb36c..d31dba0 100644
> --- a/meta/classes/package_ipk.bbclass
> +++ b/meta/classes/package_ipk.bbclass
> @@ -4,6 +4,7 @@ IMAGE_PKGTYPE ?= "ipk"
>
>  IPKGCONF_TARGET = "${WORKDIR}/opkg.conf"
>  IPKGCONF_SDK =  "${WORKDIR}/opkg-sdk.conf"
> +IPKGCONF_SDK_TARGET = "${WORKDIR}/opkg-sdk-target.conf"
>
>  PKGWRITEDIRIPK = "${WORKDIR}/deploy-ipks"
>
> diff --git a/meta/classes/rootfs_ipk.bbclass
> b/meta/classes/rootfs_ipk.bbclass
> index f1e0219..245c256 100644
> --- a/meta/classes/rootfs_ipk.bbclass
> +++ b/meta/classes/rootfs_ipk.bbclass
> @@ -14,8 +14,8 @@ do_rootfs[recrdeptask] += "do_package_write_ipk
> do_package_qa"
>  do_rootfs[vardeps] += "PACKAGE_FEED_URIS PACKAGE_FEED_BASE_PATHS
> PACKAGE_FEED_ARCHS"
>
>  do_rootfs[lockfiles] += "${WORKDIR}/ipk.lock"
> -do_populate_sdk[lockfiles] += "${WORKDIR}/ipk.lock"
> -do_populate_sdk_ext[lockfiles] += "${WORKDIR}/ipk.lock"
> +do_populate_sdk[lockfiles] += "${WORKDIR}/sdk-ipk.lock"
> +do_populate_sdk_ext[lockfiles] += "${WORKDIR}/sdk-ipk.lock"
>
>  OPKG_PREPROCESS_COMMANDS = ""
>
> diff --git a/meta/lib/oe/package_manager/ipk/sdk.py
> b/meta/lib/oe/package_manager/ipk/sdk.py
> index 37af034..e2ca415 100644
> --- a/meta/lib/oe/package_manager/ipk/sdk.py
> +++ b/meta/lib/oe/package_manager/ipk/sdk.py
> @@ -14,6 +14,12 @@ class PkgSdk(Sdk):
>      def __init__(self, d, manifest_dir=None):
>          super(PkgSdk, self).__init__(d, manifest_dir)
>
> +        # In sdk_list_installed_packages the call to opkg is hardcoded to
> +        # always use IPKGCONF_TARGET and there's no exposed API to change
> this
> +        # so simply override IPKGCONF_TARGET to use this separated config
> file.
> +        ipkgconf_sdk_target = d.getVar("IPKGCONF_SDK_TARGET")
> +        d.setVar("IPKGCONF_TARGET", ipkgconf_sdk_target)
> +
>          self.target_conf = self.d.getVar("IPKGCONF_TARGET")
>          self.host_conf = self.d.getVar("IPKGCONF_SDK")
>
> --
> 2.7.4
>
>
> 
>
>

[-- Attachment #2: Type: text/html, Size: 16584 bytes --]

Re: [OE-core] [PATCH] rootfs_ipk: allow do_populate_sdk in parallel to do_rootfs

[Richard Purdie]

On Mon, 2021-01-11 at 13:45 +0100, Michael Ho wrote:
> From: Michael Ho <Michael.Ho@bmw.de>
> 
> Switch do_populate_sdk for the ipk package manager to use a separate target
> opkg config file and separate the lockfiles restricting do_rootfs and
> do_populate_sdk from running in parallel.
> 
> This way if an image recipe includes a dependency to do_populate_sdk by
> default then it will run in parallel to do_rootfs saving time compared to the
> sequential execution.
> 
> Signed-off-by: Michael Ho <Michael.Ho@bmw.de>
> ---
>  meta/classes/package_ipk.bbclass       | 1 +
>  meta/classes/rootfs_ipk.bbclass        | 4 ++--
>  meta/lib/oe/package_manager/ipk/sdk.py | 6 ++++++
>  3 files changed, 9 insertions(+), 2 deletions(-)

I have to admit I'm very nervous about this change. The races we've
seen betweem rootfs and sdk can be quite unusual.

I did put this in for testing and we saw:

https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/2938

which in the pseudo.log shows:

path mismatch [2 links]: ino 372706913 db '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato/1.0-r0/tmp-wic' req '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato/1.0-r0/sdk/image/opt/poky/3.2+snapshot/sysroots/cortexa8hf-neon-poky-linux-gnueabi/usr/src/debug/puzzles'.
path mismatch [2 links]: ino 372706913 db '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato/1.0-r0/tmp-wic' req '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato/1.0-r0/sdk/image/opt/poky/3.2+snapshot/sysroots/cortexa8hf-neon-poky-linux-gnueabi/usr/src/debug/puzzles'.

its hard to know if this is due to this patch or possibly Paul's wic
changes as both were in this test series. I suspect it won't reproduce
every time since its a race.

Cheers,

Richard

Re: [OE-core] [PATCH] rootfs_ipk: allow do_populate_sdk in parallel to do_rootfs

[Paul Barker]

On Tue, 12 Jan 2021 at 09:21, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Mon, 2021-01-11 at 13:45 +0100, Michael Ho wrote:
> > From: Michael Ho <Michael.Ho@bmw.de>
> >
> > Switch do_populate_sdk for the ipk package manager to use a separate target
> > opkg config file and separate the lockfiles restricting do_rootfs and
> > do_populate_sdk from running in parallel.
> >
> > This way if an image recipe includes a dependency to do_populate_sdk by
> > default then it will run in parallel to do_rootfs saving time compared to the
> > sequential execution.
> >
> > Signed-off-by: Michael Ho <Michael.Ho@bmw.de>
> > ---
> >  meta/classes/package_ipk.bbclass       | 1 +
> >  meta/classes/rootfs_ipk.bbclass        | 4 ++--
> >  meta/lib/oe/package_manager/ipk/sdk.py | 6 ++++++
> >  3 files changed, 9 insertions(+), 2 deletions(-)
>
> I have to admit I'm very nervous about this change. The races we've
> seen betweem rootfs and sdk can be quite unusual.
>
> I did put this in for testing and we saw:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/2938
>
> which in the pseudo.log shows:
>
> path mismatch [2 links]: ino 372706913 db '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato/1.0-r0/tmp-wic' req '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato/1.0-r0/sdk/image/opt/poky/3.2+snapshot/sysroots/cortexa8hf-neon-poky-linux-gnueabi/usr/src/debug/puzzles'.
> path mismatch [2 links]: ino 372706913 db '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato/1.0-r0/tmp-wic' req '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato/1.0-r0/sdk/image/opt/poky/3.2+snapshot/sysroots/cortexa8hf-neon-poky-linux-gnueabi/usr/src/debug/puzzles'.
>
> its hard to know if this is due to this patch or possibly Paul's wic
> changes as both were in this test series. I suspect it won't reproduce
> every time since its a race.

Those path mismatches do seem to show that the `tmp-wic` directory was
removed outside of a pseudo context. That would point the finger at my
patches.

wic does delete its temporary working directory when exiting
(http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/scripts/lib/wic/plugins/imager/direct.py?h=master-next&id=b86e5b32f767b69bb770f6060cb784b15b147f22#n268)
but that should be ok as wic itself is running under pseudo when
invoked by bitbake. So when do_image_wic finishes, the path to
`tmp-wic` should no longer be present in the pseudo db. The error seen
here may be due to a race condition but the underlying question of
whether `tmp-wic` is removed outside a pseudo context (leaving the
path in the db) should hopefully be deterministic. Is there a way to
query the pseudo db to see if a path is present or not? I'd like to
try that after running do_image_wic.

Thanks,

-- 
Paul Barker
Konsulko Group

Re: [OE-core] [PATCH] rootfs_ipk: allow do_populate_sdk in parallel to do_rootfs

[Richard Purdie]

On Tue, 2021-01-12 at 09:21 +0000, Richard Purdie via
lists.openembedded.org wrote:
> On Mon, 2021-01-11 at 13:45 +0100, Michael Ho wrote:
> > From: Michael Ho <Michael.Ho@bmw.de>
> > 
> > Switch do_populate_sdk for the ipk package manager to use a separate target
> > opkg config file and separate the lockfiles restricting do_rootfs and
> > do_populate_sdk from running in parallel.
> > 
> > This way if an image recipe includes a dependency to do_populate_sdk by
> > default then it will run in parallel to do_rootfs saving time compared to the
> > sequential execution.
> > 
> > Signed-off-by: Michael Ho <Michael.Ho@bmw.de>
> > ---
> >  meta/classes/package_ipk.bbclass       | 1 +
> >  meta/classes/rootfs_ipk.bbclass        | 4 ++--
> >  meta/lib/oe/package_manager/ipk/sdk.py | 6 ++++++
> >  3 files changed, 9 insertions(+), 2 deletions(-)
> 
> I have to admit I'm very nervous about this change. The races we've
> seen betweem rootfs and sdk can be quite unusual.
> 
> I did put this in for testing and we saw:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/2938
> 
> which in the pseudo.log shows:
> 
> path mismatch [2 links]: ino 372706913 db '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato/1.0-r0/tmp-wic' req '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato/1.0-r0/sdk/image/opt/poky/3.2+snapshot/sysroots/cortexa8hf-neon-poky-linux-gnueabi/usr/src/debug/puzzles'.
> path mismatch [2 links]: ino 372706913 db '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato/1.0-r0/tmp-wic' req '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato/1.0-r0/sdk/image/opt/poky/3.2+snapshot/sysroots/cortexa8hf-neon-poky-linux-gnueabi/usr/src/debug/puzzles'.
> 
> its hard to know if this is due to this patch or possibly Paul's wic
> changes as both were in this test series. I suspect it won't reproduce
> every time since its a race.

Just to confirm, this is Paul's patch as it was seen with rpm packaging
and therefore couldn't be the ipk change.

I am worried we don't test ipk extensively enough to spot the races in
the ipk code though, I know I've had to fix quite a few over the years
:(. 

Cheers,

Richard

Re: [OE-core] [PATCH] rootfs_ipk: allow do_populate_sdk in parallel to do_rootfs

[Michael Ho]

Hi RP,

I would trust your judgement call here, I understand not taking this type of patch in
to avoid complicating the ipk package management.

We will soon start running in our CI essentially the dunfell patch soon so we can
test for race conditions quite heavily there. But given the refactoring of the package
managers in Yocto, it won't be the same as the master branch patch.

I'm not so deep into the oe.selftest stuff. Would it help if I try to add a test that calls
core-image-minimal:do_rootfs and core-image-minimal:do_populate_sdk at the
same time and check the build doesn't crash and try running that a few hundred times?

Kind regards,
Michael

-- 
BMW Car IT GmbH
Michael Ho
Spezialist Entwicklung – Build and Release Engineering
Lise-Meitner-Straße 14
89081 Ulm

Tel.: ­+49-731-37804-071
Mobil: +49-152-54980-471
Fax: +49-731-37804-001
Mail: michael.ho@bmw-carit.de 
Web: http://www.bmw-carit.de <http://www.bmw-carit.de/>
-------------------------------------------------------------------------
BMW Car IT GmbH
Geschäftsführer: Kai-Uwe Balszuweit und Michael Böttrich
Sitz und Registergericht: München HRB 134810
-------------------------------------------------------------------------
 

On 12.01.21, 12:58, "Richard Purdie" <richard.purdie@linuxfoundation.org> wrote:

    On Tue, 2021-01-12 at 09:21 +0000, Richard Purdie via
    lists.openembedded.org wrote:
    > On Mon, 2021-01-11 at 13:45 +0100, Michael Ho wrote:
    > > From: Michael Ho <Michael.Ho@bmw.de>
    > > 
    > > Switch do_populate_sdk for the ipk package manager to use a separate target
    > > opkg config file and separate the lockfiles restricting do_rootfs and
    > > do_populate_sdk from running in parallel.
    > > 
    > > This way if an image recipe includes a dependency to do_populate_sdk by
    > > default then it will run in parallel to do_rootfs saving time compared to the
    > > sequential execution.
    > > 
    > > Signed-off-by: Michael Ho <Michael.Ho@bmw.de>
    > > ---
    > >  meta/classes/package_ipk.bbclass       | 1 +
    > >  meta/classes/rootfs_ipk.bbclass        | 4 ++--
    > >  meta/lib/oe/package_manager/ipk/sdk.py | 6 ++++++
    > >  3 files changed, 9 insertions(+), 2 deletions(-)
    > 
    > I have to admit I'm very nervous about this change. The races we've
    > seen betweem rootfs and sdk can be quite unusual.
    > 
    > I did put this in for testing and we saw:
    > 
    > https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/2938
    > 
    > which in the pseudo.log shows:
    > 
    > path mismatch [2 links]: ino 372706913 db '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato/1.0-r0/tmp-wic' req '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato/1.0-r0/sdk/image/opt/poky/3.2+snapshot/sysroots/cortexa8hf-neon-poky-linux-gnueabi/usr/src/debug/puzzles'.
    > path mismatch [2 links]: ino 372706913 db '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato/1.0-r0/tmp-wic' req '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato/1.0-r0/sdk/image/opt/poky/3.2+snapshot/sysroots/cortexa8hf-neon-poky-linux-gnueabi/usr/src/debug/puzzles'.
    > 
    > its hard to know if this is due to this patch or possibly Paul's wic
    > changes as both were in this test series. I suspect it won't reproduce
    > every time since its a race.

    Just to confirm, this is Paul's patch as it was seen with rpm packaging
    and therefore couldn't be the ipk change.

    I am worried we don't test ipk extensively enough to spot the races in
    the ipk code though, I know I've had to fix quite a few over the years
    :(. 

    Cheers,

    Richard