All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 0/1] stress-ng: Add bash to DEPENDS
@ 2019-08-06  7:05 Robert Yang
  2019-08-06  7:05 ` [PATCH 1/1] " Robert Yang
  0 siblings, 1 reply; 17+ messages in thread
From: Robert Yang @ 2019-08-06  7:05 UTC (permalink / raw)
  To: openembedded-core

The following changes since commit 6c7c0cefd34067311144a1d4c01986fe0a4aef26:

  sstate: Reduce race windows (2019-08-03 23:56:01 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib rbt/stress-ng
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/stress-ng

Robert Yang (1):
  stress-ng: Add bash to DEPENDS

 meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
2.7.4



^ permalink raw reply	[flat|nested] 17+ messages in thread

* [PATCH 1/1] stress-ng: Add bash to DEPENDS
  2019-08-06  7:05 [PATCH 0/1] stress-ng: Add bash to DEPENDS Robert Yang
@ 2019-08-06  7:05 ` Robert Yang
  2019-08-06 10:43   ` Richard Purdie
  0 siblings, 1 reply; 17+ messages in thread
From: Robert Yang @ 2019-08-06  7:05 UTC (permalink / raw)
  To: openembedded-core

Fixed:
$ bitbake stress-ng bash -ccleansstate && bitbake stress-ng
ERROR: stress-ng-0.10.00-r0 do_package_qa: QA Issue: /usr/share/bash-completion/completions/stress-ng contained in package stress-ng-bash-completion requires /bin/bash, but no providers found in RDEPENDS_stress-ng-bash-completion? [file-rdeps]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb b/meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb
index f7faf97..904eb0c 100644
--- a/meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb
+++ b/meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb
@@ -11,7 +11,7 @@ SRC_URI = "https://kernel.ubuntu.com/~cking/tarballs/${BPN}/${BP}.tar.xz \
 SRC_URI[md5sum] = "46aa41d37690324ceab4febfcc549018"
 SRC_URI[sha256sum] = "d09dd2a1aea549e478995bf9be90b38906a4cdf33ea7b245ef9d46aa5213c074"
 
-DEPENDS = "coreutils-native"
+DEPENDS = "coreutils-native bash"
 
 inherit bash-completion
 
-- 
2.7.4



^ permalink raw reply related	[flat|nested] 17+ messages in thread

* Re: [PATCH 1/1] stress-ng: Add bash to DEPENDS
  2019-08-06  7:05 ` [PATCH 1/1] " Robert Yang
@ 2019-08-06 10:43   ` Richard Purdie
  2019-08-06 11:01     ` Robert Yang
  0 siblings, 1 reply; 17+ messages in thread
From: Richard Purdie @ 2019-08-06 10:43 UTC (permalink / raw)
  To: Robert Yang, openembedded-core

On Tue, 2019-08-06 at 15:05 +0800, Robert Yang wrote:
> Fixed:
> $ bitbake stress-ng bash -ccleansstate && bitbake stress-ng
> ERROR: stress-ng-0.10.00-r0 do_package_qa: QA Issue: /usr/share/bash-completion/completions/stress-ng contained in package stress-ng-bash-completion requires /bin/bash, but no providers found in RDEPENDS_stress-ng-bash-completion? [file-rdeps]
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb b/meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb
> index f7faf97..904eb0c 100644
> --- a/meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb
> +++ b/meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb
> @@ -11,7 +11,7 @@ SRC_URI = "https://kernel.ubuntu.com/~cking/tarballs/${BPN}/${BP}.tar.xz \
>  SRC_URI[md5sum] = "46aa41d37690324ceab4febfcc549018"
>  SRC_URI[sha256sum] = "d09dd2a1aea549e478995bf9be90b38906a4cdf33ea7b245ef9d46aa5213c074"
>  
> -DEPENDS = "coreutils-native"
> +DEPENDS = "coreutils-native bash"

Technically you just want to make this an RDEPENDS of the appropriate
package, which would give some small dependency improvement too.

Cheers,

Richard



^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 1/1] stress-ng: Add bash to DEPENDS
  2019-08-06 10:43   ` Richard Purdie
@ 2019-08-06 11:01     ` Robert Yang
  2019-08-06 11:14       ` Alexander Kanavin
  0 siblings, 1 reply; 17+ messages in thread
From: Robert Yang @ 2019-08-06 11:01 UTC (permalink / raw)
  To: Richard Purdie, openembedded-core



On 8/6/19 6:43 PM, Richard Purdie wrote:
> On Tue, 2019-08-06 at 15:05 +0800, Robert Yang wrote:
>> Fixed:
>> $ bitbake stress-ng bash -ccleansstate && bitbake stress-ng
>> ERROR: stress-ng-0.10.00-r0 do_package_qa: QA Issue: /usr/share/bash-completion/completions/stress-ng contained in package stress-ng-bash-completion requires /bin/bash, but no providers found in RDEPENDS_stress-ng-bash-completion? [file-rdeps]
>>
>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>> ---
>>   meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb b/meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb
>> index f7faf97..904eb0c 100644
>> --- a/meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb
>> +++ b/meta/recipes-extended/stress-ng/stress-ng_0.10.00.bb
>> @@ -11,7 +11,7 @@ SRC_URI = "https://kernel.ubuntu.com/~cking/tarballs/${BPN}/${BP}.tar.xz \
>>   SRC_URI[md5sum] = "46aa41d37690324ceab4febfcc549018"
>>   SRC_URI[sha256sum] = "d09dd2a1aea549e478995bf9be90b38906a4cdf33ea7b245ef9d46aa5213c074"
>>   
>> -DEPENDS = "coreutils-native"
>> +DEPENDS = "coreutils-native bash"
> 
> Technically you just want to make this an RDEPENDS of the appropriate
> package, which would give some small dependency improvement too.

Yes, exactly, I will send a V2 for it. What mis-leaded me was that
stress-ng-bash-completion RDEPENDS on bash-completion, and bash-completion
RDEPENDS on bash, so bash is on the dependencies chain, but stress-ng can't
find it, and there is no qa issues if we build bash firstly:

$ bitbake bash
$ bitbake stress-ng
No qa issues.

So the reproducer is:
$ bitbake stress-ng bash -ccleansstate && bitbake stress-ng

This might be a bug?

// Robert

> 
> Cheers,
> 
> Richard
> 
> 


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 1/1] stress-ng: Add bash to DEPENDS
  2019-08-06 11:01     ` Robert Yang
@ 2019-08-06 11:14       ` Alexander Kanavin
  2019-08-06 11:18         ` Ross Burton
  2019-08-06 11:24         ` Yang, Liezhi
  0 siblings, 2 replies; 17+ messages in thread
From: Alexander Kanavin @ 2019-08-06 11:14 UTC (permalink / raw)
  To: Robert Yang; +Cc: OE-core

[-- Attachment #1: Type: text/plain, Size: 807 bytes --]

On Tue, 6 Aug 2019 at 13:01, Robert Yang <liezhi.yang@windriver.com> wrote:

>
> Yes, exactly, I will send a V2 for it. What mis-leaded me was that
> stress-ng-bash-completion RDEPENDS on bash-completion, and bash-completion
> RDEPENDS on bash, so bash is on the dependencies chain, but stress-ng can't
> find it, and there is no qa issues if we build bash firstly:
>
> $ bitbake bash
> $ bitbake stress-ng
> No qa issues.
>
> So the reproducer is:
> $ bitbake stress-ng bash -ccleansstate && bitbake stress-ng
>
> This might be a bug?
>

Note that bash-completion files are not executable, and we do not observe
this with other recipes. We might have a false positive here - can you look
at the actual contents of the package please and see why _qa thinks it
needs /bin/bash?

Alex

[-- Attachment #2: Type: text/html, Size: 1153 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 1/1] stress-ng: Add bash to DEPENDS
  2019-08-06 11:14       ` Alexander Kanavin
@ 2019-08-06 11:18         ` Ross Burton
  2019-08-06 11:56           ` Alexander Kanavin
  2019-08-06 11:24         ` Yang, Liezhi
  1 sibling, 1 reply; 17+ messages in thread
From: Ross Burton @ 2019-08-06 11:18 UTC (permalink / raw)
  To: openembedded-core

On 06/08/2019 12:14, Alexander Kanavin wrote:
> On Tue, 6 Aug 2019 at 13:01, Robert Yang <liezhi.yang@windriver.com 
> <mailto:liezhi.yang@windriver.com>> wrote:
> 
> 
>     Yes, exactly, I will send a V2 for it. What mis-leaded me was that
>     stress-ng-bash-completion RDEPENDS on bash-completion, and
>     bash-completion
>     RDEPENDS on bash, so bash is on the dependencies chain, but
>     stress-ng can't
>     find it, and there is no qa issues if we build bash firstly:
> 
>     $ bitbake bash
>     $ bitbake stress-ng
>     No qa issues.
> 
>     So the reproducer is:
>     $ bitbake stress-ng bash -ccleansstate && bitbake stress-ng
> 
>     This might be a bug?
> 
> 
> Note that bash-completion files are not executable, and we do not 
> observe this with other recipes. We might have a false positive here - 
> can you look at the actual contents of the package please and see why 
> _qa thinks it needs /bin/bash?

Also I don't see this with master.  Do you have some local patches or QA 
changes?

Ross


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 1/1] stress-ng: Add bash to DEPENDS
  2019-08-06 11:14       ` Alexander Kanavin
  2019-08-06 11:18         ` Ross Burton
@ 2019-08-06 11:24         ` Yang, Liezhi
  1 sibling, 0 replies; 17+ messages in thread
From: Yang, Liezhi @ 2019-08-06 11:24 UTC (permalink / raw)
  To: Alexander Kanavin; +Cc: OE-core

[-- Attachment #1: Type: text/plain, Size: 1101 bytes --]



Sent from mobile phone

在 2019年8月6日,19:14,Alexander Kanavin <alex.kanavin@gmail.com<mailto:alex.kanavin@gmail.com>> 写道:

On Tue, 6 Aug 2019 at 13:01, Robert Yang <liezhi.yang@windriver.com<mailto:liezhi.yang@windriver.com>> wrote:

Yes, exactly, I will send a V2 for it. What mis-leaded me was that
stress-ng-bash-completion RDEPENDS on bash-completion, and bash-completion
RDEPENDS on bash, so bash is on the dependencies chain, but stress-ng can't
find it, and there is no qa issues if we build bash firstly:

$ bitbake bash
$ bitbake stress-ng
No qa issues.

So the reproducer is:
$ bitbake stress-ng bash -ccleansstate && bitbake stress-ng

This might be a bug?

Note that bash-completion files are not executable, and we do not observe this with other recipes. We might have a false positive here - can you look at the actual contents of the package please and see why _qa thinks it needs /bin/bash?

I had checked it, the completion file has a #/bin/bash header. I will check its details tomorrow.

// Robert



Alex

[-- Attachment #2: Type: text/html, Size: 1984 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 1/1] stress-ng: Add bash to DEPENDS
  2019-08-06 11:18         ` Ross Burton
@ 2019-08-06 11:56           ` Alexander Kanavin
  2019-08-06 12:16             ` Martin Jansa
  0 siblings, 1 reply; 17+ messages in thread
From: Alexander Kanavin @ 2019-08-06 11:56 UTC (permalink / raw)
  To: Ross Burton; +Cc: OE-core

[-- Attachment #1: Type: text/plain, Size: 715 bytes --]

On Tue, 6 Aug 2019 at 13:18, Ross Burton <ross.burton@intel.com> wrote:

> > Note that bash-completion files are not executable, and we do not
> > observe this with other recipes. We might have a false positive here -
> > can you look at the actual contents of the package please and see why
> > _qa thinks it needs /bin/bash?
>
> Also I don't see this with master.  Do you have some local patches or QA
> changes?
>

Actually there is a determinism problem here. I have reproduced it with
master, and the error happens if and only if bash's do_package task has not
yet run. If it has already run, then the package_qa for stress-ng passes
fine. I'll poke at it to see if it's a recent issue.

Alex

[-- Attachment #2: Type: text/html, Size: 1065 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 1/1] stress-ng: Add bash to DEPENDS
  2019-08-06 11:56           ` Alexander Kanavin
@ 2019-08-06 12:16             ` Martin Jansa
  2019-08-06 12:27               ` Alexander Kanavin
  0 siblings, 1 reply; 17+ messages in thread
From: Martin Jansa @ 2019-08-06 12:16 UTC (permalink / raw)
  To: Alexander Kanavin; +Cc: OE-core

[-- Attachment #1: Type: text/plain, Size: 1115 bytes --]

Maybe another case of
https://bugzilla.yoctoproject.org/show_bug.cgi?id=9217#c6 ?

On Tue, Aug 6, 2019 at 1:57 PM Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> On Tue, 6 Aug 2019 at 13:18, Ross Burton <ross.burton@intel.com> wrote:
>
>> > Note that bash-completion files are not executable, and we do not
>> > observe this with other recipes. We might have a false positive here -
>> > can you look at the actual contents of the package please and see why
>> > _qa thinks it needs /bin/bash?
>>
>> Also I don't see this with master.  Do you have some local patches or QA
>> changes?
>>
>
> Actually there is a determinism problem here. I have reproduced it with
> master, and the error happens if and only if bash's do_package task has not
> yet run. If it has already run, then the package_qa for stress-ng passes
> fine. I'll poke at it to see if it's a recent issue.
>
> Alex
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 2005 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 1/1] stress-ng: Add bash to DEPENDS
  2019-08-06 12:16             ` Martin Jansa
@ 2019-08-06 12:27               ` Alexander Kanavin
  2019-08-06 12:51                 ` Richard Purdie
  0 siblings, 1 reply; 17+ messages in thread
From: Alexander Kanavin @ 2019-08-06 12:27 UTC (permalink / raw)
  To: Martin Jansa; +Cc: OE-core

[-- Attachment #1: Type: text/plain, Size: 1437 bytes --]

Could be, yes. I will next try to untangle the file-rdeps and see why it
fails if bash hasn't been packaged yet, and passes if it has. I have
reproduced this on thud, so this is an old problem.

Alex


On Tue, 6 Aug 2019 at 14:16, Martin Jansa <martin.jansa@gmail.com> wrote:

> Maybe another case of
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=9217#c6 ?
>
> On Tue, Aug 6, 2019 at 1:57 PM Alexander Kanavin <alex.kanavin@gmail.com>
> wrote:
>
>> On Tue, 6 Aug 2019 at 13:18, Ross Burton <ross.burton@intel.com> wrote:
>>
>>> > Note that bash-completion files are not executable, and we do not
>>> > observe this with other recipes. We might have a false positive here -
>>> > can you look at the actual contents of the package please and see why
>>> > _qa thinks it needs /bin/bash?
>>>
>>> Also I don't see this with master.  Do you have some local patches or QA
>>> changes?
>>>
>>
>> Actually there is a determinism problem here. I have reproduced it with
>> master, and the error happens if and only if bash's do_package task has not
>> yet run. If it has already run, then the package_qa for stress-ng passes
>> fine. I'll poke at it to see if it's a recent issue.
>>
>> Alex
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>
>

[-- Attachment #2: Type: text/html, Size: 2646 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 1/1] stress-ng: Add bash to DEPENDS
  2019-08-06 12:27               ` Alexander Kanavin
@ 2019-08-06 12:51                 ` Richard Purdie
  2019-08-06 13:22                   ` Alexander Kanavin
  0 siblings, 1 reply; 17+ messages in thread
From: Richard Purdie @ 2019-08-06 12:51 UTC (permalink / raw)
  To: Alexander Kanavin, Martin Jansa; +Cc: OE-core

On Tue, 2019-08-06 at 14:27 +0200, Alexander Kanavin wrote:
> Could be, yes. I will next try to untangle the file-rdeps and see why
> it fails if bash hasn't been packaged yet, and passes if it has. I
> have reproduced this on thud, so this is an old problem.
> 
> Alex
> 
> 
> On Tue, 6 Aug 2019 at 14:16, Martin Jansa <martin.jansa@gmail.com>
> wrote:
> > Maybe another case of 
> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=9217#c6 ?
> > 
> > On Tue, Aug 6, 2019 at 1:57 PM Alexander Kanavin <
> > alex.kanavin@gmail.com> wrote:
> > > On Tue, 6 Aug 2019 at 13:18, Ross Burton <ross.burton@intel.com>
> > > wrote:
> > > > > Note that bash-completion files are not executable, and we do
> > > > not 
> > > > > observe this with other recipes. We might have a false
> > > > positive here - 
> > > > > can you look at the actual contents of the package please and
> > > > see why 
> > > > > _qa thinks it needs /bin/bash?
> > > > 
> > > > Also I don't see this with master.  Do you have some local
> > > > patches or QA 
> > > > changes?
> > > 
> > > Actually there is a determinism problem here. I have reproduced
> > > it with master, and the error happens if and only if bash's
> > > do_package task has not yet run. If it has already run, then the
> > > package_qa for stress-ng passes fine. I'll poke at it to see if
> > > it's a recent issue.

We now filter the pkgdata to the things a recipe can see in its
dependencies after:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=99206cb7b41be7a124ea8339bc891077f02128a2

Does that help? If not, is it because some code isn't using the
filtered copy? Or could we use this technique somewhere else?

I've not looked at the underlying issues enough to tell which code is
actually triggering it but that kind of approach should help with this
kind of problem and at least make it deterministic...

Cheers,

Richard



^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 1/1] stress-ng: Add bash to DEPENDS
  2019-08-06 12:51                 ` Richard Purdie
@ 2019-08-06 13:22                   ` Alexander Kanavin
  2019-08-06 14:05                     ` Alexander Kanavin
  2019-08-06 14:10                     ` Richard Purdie
  0 siblings, 2 replies; 17+ messages in thread
From: Alexander Kanavin @ 2019-08-06 13:22 UTC (permalink / raw)
  To: Richard Purdie; +Cc: OE-core

[-- Attachment #1: Type: text/plain, Size: 973 bytes --]

On Tue, 6 Aug 2019 at 14:51, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

>
> We now filter the pkgdata to the things a recipe can see in its
> dependencies after:
>
>
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=99206cb7b41be7a124ea8339bc891077f02128a2
>
> Does that help? If not, is it because some code isn't using the
> filtered copy? Or could we use this technique somewhere else?
>
> I've not looked at the underlying issues enough to tell which code is
> actually triggering it but that kind of approach should help with this
> kind of problem and at least make it deterministic...
>

I think the issue might be actually in the QA check where it looks beyond
the recipe's immediate RDEPENDS (and then fails or not depending on whether
those indirect dependencies have been packaged). I need to better
understand what happens there so will add some printouts first. The problem
does happen with latest master.

Alex

[-- Attachment #2: Type: text/html, Size: 1489 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 1/1] stress-ng: Add bash to DEPENDS
  2019-08-06 13:22                   ` Alexander Kanavin
@ 2019-08-06 14:05                     ` Alexander Kanavin
  2019-08-06 14:16                       ` Richard Purdie
  2019-08-06 14:10                     ` Richard Purdie
  1 sibling, 1 reply; 17+ messages in thread
From: Alexander Kanavin @ 2019-08-06 14:05 UTC (permalink / raw)
  To: Richard Purdie; +Cc: OE-core

[-- Attachment #1: Type: text/plain, Size: 1846 bytes --]

On Tue, 6 Aug 2019 at 15:22, Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> I think the issue might be actually in the QA check where it looks beyond
> the recipe's immediate RDEPENDS (and then fails or not depending on whether
> those indirect dependencies have been packaged). I need to better
> understand what happens there so will add some printouts first. The problem
> does happen with latest master.
>

I can confirm that this bit in insane.bbclass is non-deterministic:

                # Find all the rdepends on the dependency chain
                while next:
                    new = []
                    for rdep in next:
                        rdep_data = oe.packagedata.read_subpkgdata(rdep, d)
                        sub_rdeps = rdep_data.get("RDEPENDS_" + rdep)
                        if not sub_rdeps:
                            continue
                        for sub_rdep in bb.utils.explode_deps(sub_rdeps):
                            if sub_rdep in done:
                                continue
                            if oe.packagedata.has_subpkgdata(sub_rdep, d):
                                # It's a new rdep
                                done.append(sub_rdep)
                                new.append(sub_rdep)
                    next = new

It only fully expands the dependencies if all of them have been already
packaged. As there is no mechanism to ensure that package_qa runs after
packaging all of the recursive RDEPENDS, this is causing a race between
packaging them, and running the package qa check.

I think the right way to address this is to actually drop the recursive
RDEPENDS resolution, and only consider the immediate runtime dependencies.
Alternatively, we may introduce additional task dependencies to fix the
race. Thoughts?

Alex

[-- Attachment #2: Type: text/html, Size: 2470 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 1/1] stress-ng: Add bash to DEPENDS
  2019-08-06 13:22                   ` Alexander Kanavin
  2019-08-06 14:05                     ` Alexander Kanavin
@ 2019-08-06 14:10                     ` Richard Purdie
  1 sibling, 0 replies; 17+ messages in thread
From: Richard Purdie @ 2019-08-06 14:10 UTC (permalink / raw)
  To: Alexander Kanavin; +Cc: OE-core

On Tue, 2019-08-06 at 15:22 +0200, Alexander Kanavin wrote:
> On Tue, 6 Aug 2019 at 14:51, Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
> > We now filter the pkgdata to the things a recipe can see in its
> > dependencies after:
> > 
> > http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=99206cb7b41be7a124ea8339bc891077f02128a2
> > 
> > Does that help? If not, is it because some code isn't using the
> > filtered copy? Or could we use this technique somewhere else?
> > 
> > I've not looked at the underlying issues enough to tell which code
> > is
> > actually triggering it but that kind of approach should help with
> > this
> > kind of problem and at least make it deterministic...
> 
> I think the issue might be actually in the QA check where it looks
> beyond the recipe's immediate RDEPENDS (and then fails or not
> depending on whether those indirect dependencies have been packaged).
> I need to better understand what happens there so will add some
> printouts first. The problem does happen with latest master.

I had a quick look and package_qa_check_rdepends() has a call to:

oe.packagedata.read_subpkgdata(pkg, d)

which effectively calls:

def get_subpkgedata_fn(pkg, d):
    return d.expand('${PKGDATA_DIR}/runtime/%s' % pkg)

(from packagedata.py)

I think that this should be changed to WORKDIR_PKGDATA, at least for
the purposes of insane.bbclass. The copy in WORKDIR_PKGDATA is limited
to the recipe's dependencies.

Its not quite that simple as we need to ensure that WORKDIR_PKGDATA
when the insane code runs but you hopefully get the idea...

Cheers,

Richard






^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 1/1] stress-ng: Add bash to DEPENDS
  2019-08-06 14:05                     ` Alexander Kanavin
@ 2019-08-06 14:16                       ` Richard Purdie
  2019-08-06 14:30                         ` Alexander Kanavin
  0 siblings, 1 reply; 17+ messages in thread
From: Richard Purdie @ 2019-08-06 14:16 UTC (permalink / raw)
  To: Alexander Kanavin; +Cc: OE-core

On Tue, 2019-08-06 at 16:05 +0200, Alexander Kanavin wrote:
> On Tue, 6 Aug 2019 at 15:22, Alexander Kanavin <
> alex.kanavin@gmail.com> wrote:
> > I think the issue might be actually in the QA check where it looks
> > beyond the recipe's immediate RDEPENDS (and then fails or not
> > depending on whether those indirect dependencies have been
> > packaged). I need to better understand what happens there so will
> > add some printouts first. The problem does happen with latest
> > master.
> > 
> 
> I can confirm that this bit in insane.bbclass is non-deterministic:
> 
>                 # Find all the rdepends on the dependency chain
>                 while next:
>                     new = []
>                     for rdep in next:
>                         rdep_data =
> oe.packagedata.read_subpkgdata(rdep, d)
>                         sub_rdeps = rdep_data.get("RDEPENDS_" + rdep)
>                         if not sub_rdeps:
>                             continue
>                         for sub_rdep in
> bb.utils.explode_deps(sub_rdeps):
>                             if sub_rdep in done:
>                                 continue
>                             if
> oe.packagedata.has_subpkgdata(sub_rdep, d):
>                                 # It's a new rdep
>                                 done.append(sub_rdep)
>                                 new.append(sub_rdep)
>                     next = new
> 
> It only fully expands the dependencies if all of them have been
> already packaged. As there is no mechanism to ensure that package_qa
> runs after packaging all of the recursive RDEPENDS, this is causing a
> race between packaging them, and running the package qa check.
>
> I think the right way to address this is to actually drop the
> recursive RDEPENDS resolution, and only consider the immediate
> runtime dependencies. Alternatively, we may introduce additional task
> dependencies to fix the race. Thoughts?

I partly agree with this. There is:

insane.bbclass:do_package_qa[rdeptask] = "do_packagedata"

which does help the situation by forcing package_qa after all the
runtime dependencies.

I suspect the real problem is that it can also see RDEPENDS for things
it doesn't rdepend upon...

You might want package_prepare_pkgdata() from package_pkgdata called
for package_qa's context on a separate directory, then to use that as
the pkgdata to resolve against?

Note that package_prepare_pkgdata() called for do_package is different
than package_prepare_pkgdata() called for do_package_qa as the tasks
have different dependencies...

Cheers,

Richard









^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 1/1] stress-ng: Add bash to DEPENDS
  2019-08-06 14:16                       ` Richard Purdie
@ 2019-08-06 14:30                         ` Alexander Kanavin
  2019-08-06 15:56                           ` Alexander Kanavin
  0 siblings, 1 reply; 17+ messages in thread
From: Alexander Kanavin @ 2019-08-06 14:30 UTC (permalink / raw)
  To: Richard Purdie; +Cc: OE-core

[-- Attachment #1: Type: text/plain, Size: 1494 bytes --]

On Tue, 6 Aug 2019 at 16:16, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> I partly agree with this. There is:
>
> insane.bbclass:do_package_qa[rdeptask] = "do_packagedata"
>
> which does help the situation by forcing package_qa after all the
> runtime dependencies.
>
> I suspect the real problem is that it can also see RDEPENDS for things
> it doesn't rdepend upon...
>
> You might want package_prepare_pkgdata() from package_pkgdata called
> for package_qa's context on a separate directory, then to use that as
> the pkgdata to resolve against?
>
> Note that package_prepare_pkgdata() called for do_package is different
> than package_prepare_pkgdata() called for do_package_qa as the tasks
> have different dependencies...
>

I am actually inclined to remove the recursive resolver in insane.bbclass.
It gives a QA pass to a situation when there is a bash script in A, and the
bash runtime dependency is satisfied indirectly through a chain of
arbitrary length: A -> bash-completion -> bash. It becomes hard to reason
where the dependencies come from; I think any runtime dependencies should
be explicitly listed. We may need to fix up a few recipes, but the outcome
is a better definition of runtime dependencies.

I have confirmed that for stress-ng removing the recursive resolver and
adding an explicit RDEPENDS on bash solves the issue (stress-ng package_qa
does not start until bash packaging completes, unlike previously).

Alex

[-- Attachment #2: Type: text/html, Size: 1903 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: [PATCH 1/1] stress-ng: Add bash to DEPENDS
  2019-08-06 14:30                         ` Alexander Kanavin
@ 2019-08-06 15:56                           ` Alexander Kanavin
  0 siblings, 0 replies; 17+ messages in thread
From: Alexander Kanavin @ 2019-08-06 15:56 UTC (permalink / raw)
  To: Richard Purdie; +Cc: OE-core

[-- Attachment #1: Type: text/plain, Size: 1635 bytes --]

On Tue, 6 Aug 2019 at 16:30, Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> On Tue, 6 Aug 2019 at 16:16, Richard Purdie <
> richard.purdie@linuxfoundation.org> wrote:
>
>> I partly agree with this. There is:
>>
>> insane.bbclass:do_package_qa[rdeptask] = "do_packagedata"
>>
>> which does help the situation by forcing package_qa after all the
>> runtime dependencies.
>>
>> I suspect the real problem is that it can also see RDEPENDS for things
>> it doesn't rdepend upon...
>>
>> You might want package_prepare_pkgdata() from package_pkgdata called
>> for package_qa's context on a separate directory, then to use that as
>> the pkgdata to resolve against?
>>
>> Note that package_prepare_pkgdata() called for do_package is different
>> than package_prepare_pkgdata() called for do_package_qa as the tasks
>> have different dependencies...
>>
>
> I am actually inclined to remove the recursive resolver in insane.bbclass.
> It gives a QA pass to a situation when there is a bash script in A, and the
> bash runtime dependency is satisfied indirectly through a chain of
> arbitrary length: A -> bash-completion -> bash. It becomes hard to reason
> where the dependencies come from; I think any runtime dependencies should
> be explicitly listed. We may need to fix up a few recipes, but the outcome
> is a better definition of runtime dependencies.
>

I made a patch for this:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=akanavin/package-version-updates&id=77bb080eb15de21901c15072898303f2b5275093

Now building oe-core world, to see what kind of fallout it causes.

Alex

[-- Attachment #2: Type: text/html, Size: 2449 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

end of thread, other threads:[~2019-08-06 15:57 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-08-06  7:05 [PATCH 0/1] stress-ng: Add bash to DEPENDS Robert Yang
2019-08-06  7:05 ` [PATCH 1/1] " Robert Yang
2019-08-06 10:43   ` Richard Purdie
2019-08-06 11:01     ` Robert Yang
2019-08-06 11:14       ` Alexander Kanavin
2019-08-06 11:18         ` Ross Burton
2019-08-06 11:56           ` Alexander Kanavin
2019-08-06 12:16             ` Martin Jansa
2019-08-06 12:27               ` Alexander Kanavin
2019-08-06 12:51                 ` Richard Purdie
2019-08-06 13:22                   ` Alexander Kanavin
2019-08-06 14:05                     ` Alexander Kanavin
2019-08-06 14:16                       ` Richard Purdie
2019-08-06 14:30                         ` Alexander Kanavin
2019-08-06 15:56                           ` Alexander Kanavin
2019-08-06 14:10                     ` Richard Purdie
2019-08-06 11:24         ` Yang, Liezhi

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.