[PATCH 2/2] multiconfig: Enable multiconfig dependencies on oe-core

[PATCH 2/2] multiconfig: Enable multiconfig dependencies on oe-core

[Alejandro Enedino Hernandez Samaniego]

This patch enables multiconfig dependencies (mcdepends) to be used on
recipes using the following format:

task[mcdepends] = "multiconfig:FROM-MC:TO-MC:PN:task-to-depend-on"

For the sake of simplicity consider the following example:

Assuming we have set up multiconfig builds, one for qemux86 and one for
qemuarm, named x86 and arm respectively.

Adding the following line to an image recipe (core-image-sato):
do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs"

Would state that core-image-sato:do_image from x86 will depend on
core-image-minimal:do_rootfs from arm so it can be executed.

This patch makes modifications to bitbake.conf to enable mcdepends, and
to sstatesig and staging.bbclass to avoid conflicts between packages from
different multiconfigs.

[YOCTO #10681]

Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
---
 meta/classes/staging.bbclass | 8 +++++++-
 meta/conf/bitbake.conf       | 2 +-
 meta/lib/oe/sstatesig.py     | 9 ++++++++-
 3 files changed, 16 insertions(+), 3 deletions(-)

diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
index 41df883..27b012e 100644
--- a/meta/classes/staging.bbclass
+++ b/meta/classes/staging.bbclass
@@ -256,7 +256,7 @@ python extend_recipe_sysroot() {
     workdir = d.getVar("WORKDIR")
     #bb.warn(str(taskdepdata))
     pn = d.getVar("PN")
-
+    mc = d.getVar("BB_CURRENT_MC")
     stagingdir = d.getVar("STAGING_DIR")
     sharedmanifests = d.getVar("COMPONENTS_DIR") + "/manifests"
     recipesysroot = d.getVar("RECIPE_SYSROOT")
@@ -443,7 +443,13 @@ python extend_recipe_sysroot() {
 
     msg_exists = []
     msg_adding = []
+
     for dep in configuredeps:
+        if mc != 'default':
+            # We should not care about other multiconfigs
+            depmc = dep.split(':')[1]
+            if depmc != mc:
+                continue
         c = setscenedeps[dep][0]
         if c not in installed:
             continue
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index e28f3c7..f68954c 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -877,7 +877,7 @@ BB_SIGNATURE_EXCLUDE_FLAGS ?= "doc deps depends \
     file-checksums python func task export unexport noexec nostamp dirs cleandirs \
     sstate-lockfile-shared prefuncs postfuncs export_func deptask rdeptask \
     recrdeptask nodeprrecs stamp-extra-info sstate-outputdirs filename lineno \
-    progress"
+    progress mcdepends"
 
 MLPREFIX ??= ""
 MULTILIB_VARIANTS ??= ""
diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index 5dcc2f5..18c5a35 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -150,16 +150,23 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
         if recipename in self.unlockedrecipes:
             unlocked = True
         else:
+            def get_mc(tid):
+                tid = tid.rsplit('.', 1)[0]
+                if tid.startswith('multiconfig:'):
+                    elems = tid.split(':')
+                    return elems[1]
             def recipename_from_dep(dep):
                 # The dep entry will look something like
                 # /path/path/recipename.bb.task, virtual:native:/p/foo.bb.task,
                 # ...
+
                 fn = dep.rsplit('.', 1)[0]
                 return dataCache.pkg_fn[fn]
 
+            mc = get_mc(fn)
             # If any unlocked recipe is in the direct dependencies then the
             # current recipe should be unlocked as well.
-            depnames = [ recipename_from_dep(x) for x in deps ]
+            depnames = [ recipename_from_dep(x) for x in deps if mc == get_mc(x)]
             if any(x in y for y in depnames for x in self.unlockedrecipes):
                 self.unlockedrecipes[recipename] = ''
                 unlocked = True
-- 
2.7.4

Re: [PATCH 2/2] multiconfig: Enable multiconfig dependencies on oe-core

[Khem Raj]

[-- Attachment #1.1: Type: text/plain, Size: 4394 bytes --]



On 7/25/18 9:05 AM, Alejandro Enedino Hernandez Samaniego wrote:
> This patch enables multiconfig dependencies (mcdepends) to be used on
> recipes using the following format:
> 
> task[mcdepends] = "multiconfig:FROM-MC:TO-MC:PN:task-to-depend-on"
> 
> For the sake of simplicity consider the following example:
> 
> Assuming we have set up multiconfig builds, one for qemux86 and one for
> qemuarm, named x86 and arm respectively.
> 
> Adding the following line to an image recipe (core-image-sato):
> do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs"
> 

Do we really need to add multiconfig to keyword namespace ? can we
deduce this from "arch1:arch2:..."

secondy, this need to be well documented, with examples for users

> Would state that core-image-sato:do_image from x86 will depend on
> core-image-minimal:do_rootfs from arm so it can be executed.
> 
> This patch makes modifications to bitbake.conf to enable mcdepends, and
> to sstatesig and staging.bbclass to avoid conflicts between packages from
> different multiconfigs.
> 
> [YOCTO #10681]
> 
> Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
> ---
>  meta/classes/staging.bbclass | 8 +++++++-
>  meta/conf/bitbake.conf       | 2 +-
>  meta/lib/oe/sstatesig.py     | 9 ++++++++-
>  3 files changed, 16 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
> index 41df883..27b012e 100644
> --- a/meta/classes/staging.bbclass
> +++ b/meta/classes/staging.bbclass
> @@ -256,7 +256,7 @@ python extend_recipe_sysroot() {
>      workdir = d.getVar("WORKDIR")
>      #bb.warn(str(taskdepdata))
>      pn = d.getVar("PN")
> -
> +    mc = d.getVar("BB_CURRENT_MC")
>      stagingdir = d.getVar("STAGING_DIR")
>      sharedmanifests = d.getVar("COMPONENTS_DIR") + "/manifests"
>      recipesysroot = d.getVar("RECIPE_SYSROOT")
> @@ -443,7 +443,13 @@ python extend_recipe_sysroot() {
>  
>      msg_exists = []
>      msg_adding = []
> +
>      for dep in configuredeps:
> +        if mc != 'default':
> +            # We should not care about other multiconfigs
> +            depmc = dep.split(':')[1]
> +            if depmc != mc:
> +                continue
>          c = setscenedeps[dep][0]
>          if c not in installed:
>              continue
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index e28f3c7..f68954c 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -877,7 +877,7 @@ BB_SIGNATURE_EXCLUDE_FLAGS ?= "doc deps depends \
>      file-checksums python func task export unexport noexec nostamp dirs cleandirs \
>      sstate-lockfile-shared prefuncs postfuncs export_func deptask rdeptask \
>      recrdeptask nodeprrecs stamp-extra-info sstate-outputdirs filename lineno \
> -    progress"
> +    progress mcdepends"
>  
>  MLPREFIX ??= ""
>  MULTILIB_VARIANTS ??= ""
> diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
> index 5dcc2f5..18c5a35 100644
> --- a/meta/lib/oe/sstatesig.py
> +++ b/meta/lib/oe/sstatesig.py
> @@ -150,16 +150,23 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
>          if recipename in self.unlockedrecipes:
>              unlocked = True
>          else:
> +            def get_mc(tid):
> +                tid = tid.rsplit('.', 1)[0]
> +                if tid.startswith('multiconfig:'):
> +                    elems = tid.split(':')
> +                    return elems[1]
>              def recipename_from_dep(dep):
>                  # The dep entry will look something like
>                  # /path/path/recipename.bb.task, virtual:native:/p/foo.bb.task,
>                  # ...
> +
>                  fn = dep.rsplit('.', 1)[0]
>                  return dataCache.pkg_fn[fn]
>  
> +            mc = get_mc(fn)
>              # If any unlocked recipe is in the direct dependencies then the
>              # current recipe should be unlocked as well.
> -            depnames = [ recipename_from_dep(x) for x in deps ]
> +            depnames = [ recipename_from_dep(x) for x in deps if mc == get_mc(x)]
>              if any(x in y for y in depnames for x in self.unlockedrecipes):
>                  self.unlockedrecipes[recipename] = ''
>                  unlocked = True
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: [PATCH 2/2] multiconfig: Enable multiconfig dependencies on oe-core

[Alejandro Enedino Hernandez Samaniego]

Hey Khem,


On 07/26/2018 08:36 PM, Khem Raj wrote:
>
> On 7/25/18 9:05 AM, Alejandro Enedino Hernandez Samaniego wrote:
>> This patch enables multiconfig dependencies (mcdepends) to be used on
>> recipes using the following format:
>>
>> task[mcdepends] = "multiconfig:FROM-MC:TO-MC:PN:task-to-depend-on"
>>
>> For the sake of simplicity consider the following example:
>>
>> Assuming we have set up multiconfig builds, one for qemux86 and one for
>> qemuarm, named x86 and arm respectively.
>>
>> Adding the following line to an image recipe (core-image-sato):
>> do_image[mcdepends] = "multiconfig:x86:arm:core-image-minimal:do_rootfs"
>>
> Do we really need to add multiconfig to keyword namespace ? can we
> deduce this from "arch1:arch2:..."
Bitbake already uses this and I believed adopting it would make it 
easier for the user to understand its usage.

For example, since the multiconfig names are also provided during setup, 
if you assume that someone set up
the build and a different user is going through the recipe, arch1:arch2 
wouldn't mean a lot for that user
because they may not map 1:1 to MACHINE names.

>
> secondy, this need to be well documented, with examples for users
I agree we could add more documentation to it, where do you think it 
should go?

>> Would state that core-image-sato:do_image from x86 will depend on
>> core-image-minimal:do_rootfs from arm so it can be executed.
>>
>> This patch makes modifications to bitbake.conf to enable mcdepends, and
>> to sstatesig and staging.bbclass to avoid conflicts between packages from
>> different multiconfigs.
>>
>> [YOCTO #10681]
>>
>> Signed-off-by: Alejandro Enedino Hernandez Samaniego <alejandr@xilinx.com>
>> ---
>>   meta/classes/staging.bbclass | 8 +++++++-
>>   meta/conf/bitbake.conf       | 2 +-
>>   meta/lib/oe/sstatesig.py     | 9 ++++++++-
>>   3 files changed, 16 insertions(+), 3 deletions(-)
>>
>> diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
>> index 41df883..27b012e 100644
>> --- a/meta/classes/staging.bbclass
>> +++ b/meta/classes/staging.bbclass
>> @@ -256,7 +256,7 @@ python extend_recipe_sysroot() {
>>       workdir = d.getVar("WORKDIR")
>>       #bb.warn(str(taskdepdata))
>>       pn = d.getVar("PN")
>> -
>> +    mc = d.getVar("BB_CURRENT_MC")
>>       stagingdir = d.getVar("STAGING_DIR")
>>       sharedmanifests = d.getVar("COMPONENTS_DIR") + "/manifests"
>>       recipesysroot = d.getVar("RECIPE_SYSROOT")
>> @@ -443,7 +443,13 @@ python extend_recipe_sysroot() {
>>   
>>       msg_exists = []
>>       msg_adding = []
>> +
>>       for dep in configuredeps:
>> +        if mc != 'default':
>> +            # We should not care about other multiconfigs
>> +            depmc = dep.split(':')[1]
>> +            if depmc != mc:
>> +                continue
>>           c = setscenedeps[dep][0]
>>           if c not in installed:
>>               continue
>> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
>> index e28f3c7..f68954c 100644
>> --- a/meta/conf/bitbake.conf
>> +++ b/meta/conf/bitbake.conf
>> @@ -877,7 +877,7 @@ BB_SIGNATURE_EXCLUDE_FLAGS ?= "doc deps depends \
>>       file-checksums python func task export unexport noexec nostamp dirs cleandirs \
>>       sstate-lockfile-shared prefuncs postfuncs export_func deptask rdeptask \
>>       recrdeptask nodeprrecs stamp-extra-info sstate-outputdirs filename lineno \
>> -    progress"
>> +    progress mcdepends"
>>   
>>   MLPREFIX ??= ""
>>   MULTILIB_VARIANTS ??= ""
>> diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
>> index 5dcc2f5..18c5a35 100644
>> --- a/meta/lib/oe/sstatesig.py
>> +++ b/meta/lib/oe/sstatesig.py
>> @@ -150,16 +150,23 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
>>           if recipename in self.unlockedrecipes:
>>               unlocked = True
>>           else:
>> +            def get_mc(tid):
>> +                tid = tid.rsplit('.', 1)[0]
>> +                if tid.startswith('multiconfig:'):
>> +                    elems = tid.split(':')
>> +                    return elems[1]
>>               def recipename_from_dep(dep):
>>                   # The dep entry will look something like
>>                   # /path/path/recipename.bb.task, virtual:native:/p/foo.bb.task,
>>                   # ...
>> +
>>                   fn = dep.rsplit('.', 1)[0]
>>                   return dataCache.pkg_fn[fn]
>>   
>> +            mc = get_mc(fn)
>>               # If any unlocked recipe is in the direct dependencies then the
>>               # current recipe should be unlocked as well.
>> -            depnames = [ recipename_from_dep(x) for x in deps ]
>> +            depnames = [ recipename_from_dep(x) for x in deps if mc == get_mc(x)]
>>               if any(x in y for y in depnames for x in self.unlockedrecipes):
>>                   self.unlockedrecipes[recipename] = ''
>>                   unlocked = True
>>
Cheers,

Alejandro

Re: [PATCH 2/2] multiconfig: Enable multiconfig dependencies on oe-core

[Richard Purdie]

On Mon, 2018-07-30 at 15:05 -0700, Alejandro Enedino Hernandez
Samaniego wrote:
> On 07/26/2018 08:36 PM, Khem Raj wrote:
> > 
> > On 7/25/18 9:05 AM, Alejandro Enedino Hernandez Samaniego wrote:
> > > This patch enables multiconfig dependencies (mcdepends) to be
> > > used on
> > > recipes using the following format:
> > > 
> > > task[mcdepends] = "multiconfig:FROM-MC:TO-MC:PN:task-to-depend-
> > > on"
> > > 
> > > For the sake of simplicity consider the following example:
> > > 
> > > Assuming we have set up multiconfig builds, one for qemux86 and
> > > one for
> > > qemuarm, named x86 and arm respectively.
> > > 
> > > Adding the following line to an image recipe (core-image-sato):
> > > do_image[mcdepends] = "multiconfig:x86:arm:core-image-
> > > minimal:do_rootfs"
> > > 
> > 
> > Do we really need to add multiconfig to keyword namespace ? can we
> > deduce this from "arch1:arch2:..."
> 
> Bitbake already uses this and I believed adopting it would make it 
> easier for the user to understand its usage.
>
> For example, since the multiconfig names are also provided during
> setup, if you assume that someone set up the build and a different
> user is going through the recipe, arch1:arch2 wouldn't mean a lot for
> that user because they may not map 1:1 to MACHINE names.

It is consistent with the rest of the system. I'm still torn on whether
we should shorten "multiconfig:" to "mc:" but we need to be consistent.

> > secondy, this need to be well documented, with examples for users
> 
> I agree we could add more documentation to it, where do you think it 
> should go?

I'm going to queue the patches for testing in -next, they look good to
me just reading the patches.

Could you start to work with Scott (cc'd) to get this new dependency
type listed in the bitbake manual and anywhere else it needs to be
documented?

Its great to see this finally working!

Cheers,

Richard

Re: [PATCH 2/2] multiconfig: Enable multiconfig dependencies on oe-core

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 2243 bytes --]

On Tue, Jul 31, 2018 at 4:59 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Mon, 2018-07-30 at 15:05 -0700, Alejandro Enedino Hernandez
> Samaniego wrote:
> > On 07/26/2018 08:36 PM, Khem Raj wrote:
> > >
> > > On 7/25/18 9:05 AM, Alejandro Enedino Hernandez Samaniego wrote:
> > > > This patch enables multiconfig dependencies (mcdepends) to be
> > > > used on
> > > > recipes using the following format:
> > > >
> > > > task[mcdepends] = "multiconfig:FROM-MC:TO-MC:PN:task-to-depend-
> > > > on"
> > > >
> > > > For the sake of simplicity consider the following example:
> > > >
> > > > Assuming we have set up multiconfig builds, one for qemux86 and
> > > > one for
> > > > qemuarm, named x86 and arm respectively.
> > > >
> > > > Adding the following line to an image recipe (core-image-sato):
> > > > do_image[mcdepends] = "multiconfig:x86:arm:core-image-
> > > > minimal:do_rootfs"
> > > >
> > >
> > > Do we really need to add multiconfig to keyword namespace ? can we
> > > deduce this from "arch1:arch2:..."
> >
> > Bitbake already uses this and I believed adopting it would make it
> > easier for the user to understand its usage.
> >
> > For example, since the multiconfig names are also provided during
> > setup, if you assume that someone set up the build and a different
> > user is going through the recipe, arch1:arch2 wouldn't mean a lot for
> > that user because they may not map 1:1 to MACHINE names.
>
> It is consistent with the rest of the system. I'm still torn on whether
> we should shorten "multiconfig:" to "mc:" but we need to be consistent.
>

Given a choice I would suggest for using multiconfig for lesser chance of
conflicts

>
> > > secondy, this need to be well documented, with examples for users
> >
> > I agree we could add more documentation to it, where do you think it
> > should go?
>
> I'm going to queue the patches for testing in -next, they look good to
> me just reading the patches.
>
> Could you start to work with Scott (cc'd) to get this new dependency
> type listed in the bitbake manual and anywhere else it needs to be
> documented?
>
> Its great to see this finally working!
>
> Cheers,
>
> Richard
>
>

[-- Attachment #2: Type: text/html, Size: 3074 bytes --]

Re: [PATCH 2/2] multiconfig: Enable multiconfig dependencies on oe-core

[Alejandro Enedino Hernandez Samaniego]

[-- Attachment #1: Type: text/plain, Size: 2703 bytes --]

Hey Khem, Richard,


On 08/01/2018 10:44 AM, Khem Raj wrote:
>
>
> On Tue, Jul 31, 2018 at 4:59 AM Richard Purdie 
> <richard.purdie@linuxfoundation.org 
> <mailto:richard.purdie@linuxfoundation.org>> wrote:
>
>     On Mon, 2018-07-30 at 15:05 -0700, Alejandro Enedino Hernandez
>     Samaniego wrote:
>     > On 07/26/2018 08:36 PM, Khem Raj wrote:
>     > >
>     > > On 7/25/18 9:05 AM, Alejandro Enedino Hernandez Samaniego wrote:
>     > > > This patch enables multiconfig dependencies (mcdepends) to be
>     > > > used on
>     > > > recipes using the following format:
>     > > >
>     > > > task[mcdepends] = "multiconfig:FROM-MC:TO-MC:PN:task-to-depend-
>     > > > on"
>     > > >
>     > > > For the sake of simplicity consider the following example:
>     > > >
>     > > > Assuming we have set up multiconfig builds, one for qemux86 and
>     > > > one for
>     > > > qemuarm, named x86 and arm respectively.
>     > > >
>     > > > Adding the following line to an image recipe (core-image-sato):
>     > > > do_image[mcdepends] = "multiconfig:x86:arm:core-image-
>     > > > minimal:do_rootfs"
>     > > >
>     > >
>     > > Do we really need to add multiconfig to keyword namespace ? can we
>     > > deduce this from "arch1:arch2:..."
>     >
>     > Bitbake already uses this and I believed adopting it would make it
>     > easier for the user to understand its usage.
>     >
>     > For example, since the multiconfig names are also provided during
>     > setup, if you assume that someone set up the build and a different
>     > user is going through the recipe, arch1:arch2 wouldn't mean a
>     lot for
>     > that user because they may not map 1:1 to MACHINE names.
>
>     It is consistent with the rest of the system. I'm still torn on
>     whether
>     we should shorten "multiconfig:" to "mc:" but we need to be
>     consistent.
>
>
> Given a choice I would suggest for using multiconfig for lesser chance 
> of conflicts
I also agree here that its better to use multiconfig.
>
>
>     > > secondy, this need to be well documented, with examples for users
>     >
>     > I agree we could add more documentation to it, where do you
>     think it
>     > should go?
>
>     I'm going to queue the patches for testing in -next, they look good to
>     me just reading the patches.
>
>     Could you start to work with Scott (cc'd) to get this new dependency
>     type listed in the bitbake manual and anywhere else it needs to be
>     documented?
>
Sure, I'll sync with Scott

Thanks!,

Alejandro
>
>
>     Its great to see this finally working!
>
>     Cheers,
>
>     Richard
>


[-- Attachment #2: Type: text/html, Size: 5428 bytes --]