All of lore.kernel.org
 help / color / mirror / Atom feed
* Does recipe specific sysrooot (or whatelse in current oe) break native dependencies?
@ 2017-02-13  0:26 Andreas Müller
  2017-02-13 13:47 ` Andreas Müller
  0 siblings, 1 reply; 17+ messages in thread
From: Andreas Müller @ 2017-02-13  0:26 UTC (permalink / raw)
  To: Patches and discussions about the oe-core layer

Hi

I try to get meta-qt5-extra fit for latest oe-core (RSS). One thing
that causes my attention is that many recipes's configure can be fixed
by adding gettext-native. This seems odd to me because all of these
recipes depend upon kdoctools which depends on gettext-native.
For instance for kconfigwidgets depending on kdoctools:

* in kdoctool'sworkdir/recipe-sysroot-native/usr/bin there is a gettext binary
* but in kconfigwidgets'sworkdir/recipe-sysroot-native/usr/bin there
isn't a gettext binary

Is it possible that recipe specific sysroot broke native dependency chain?

One note which might be important for this issue: In meta-qt5-extra I
chose a the following design decision: If there is a pair of
native/cross recipes
* each cross recipe depends on native recipe
* all other recipes depend on cross recipes

This reduced maintenance efforts (I don't have to care if a recipe
depends on cross libs or native executables) and avoids race trouble
with cmake's toolchain path sequence 1. cross 2. native

So what goes wrong here - or where am I mistaken?

Andreas


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Does recipe specific sysrooot (or whatelse in current oe) break native dependencies?
  2017-02-13  0:26 Does recipe specific sysrooot (or whatelse in current oe) break native dependencies? Andreas Müller
@ 2017-02-13 13:47 ` Andreas Müller
  2017-02-13 14:36   ` Martin Jansa
  0 siblings, 1 reply; 17+ messages in thread
From: Andreas Müller @ 2017-02-13 13:47 UTC (permalink / raw)
  To: Patches and discussions about the oe-core layer

On Mon, Feb 13, 2017 at 1:26 AM, Andreas Müller
<schnitzeltony@googlemail.com> wrote:
> Hi
>
> I try to get meta-qt5-extra fit for latest oe-core (RSS). One thing
> that causes my attention is that many recipes's configure can be fixed
> by adding gettext-native. This seems odd to me because all of these
> recipes depend upon kdoctools which depends on gettext-native.
> For instance for kconfigwidgets depending on kdoctools:
>
> * in kdoctool'sworkdir/recipe-sysroot-native/usr/bin there is a gettext binary
> * but in kconfigwidgets'sworkdir/recipe-sysroot-native/usr/bin there
> isn't a gettext binary
>
> Is it possible that recipe specific sysroot broke native dependency chain?
>
> One note which might be important for this issue: In meta-qt5-extra I
> chose a the following design decision: If there is a pair of
> native/cross recipes
> * each cross recipe depends on native recipe
> * all other recipes depend on cross recipes
>
> This reduced maintenance efforts (I don't have to care if a recipe
> depends on cross libs or native executables) and avoids race trouble
> with cmake's toolchain path sequence 1. cross 2. native
>
> So what goes wrong here - or where am I mistaken?
>
Another more common examples:

* in meta-qt5-extra I could fix many recipes by depending on
qtbase-native although they already depended on qtbase (qtbase depends
on qtbase-native)
* right after recipe specific sysroot a dependency on qtbase-native
was added in cmake_qt5.bbclass although it depended on qtbase already
and qtbase depends on qtbase-native

Before we carry on adding dozens of dependencies and blacklisting
world: Are missing native dependencies a bug or a feature - or where
am I going wrong? What scares me most is that my way of depending
always on cross recipes in meta-qt5-extra will not work anymore
(mentioned in my previous mail) and turn into a unmaintainable burden.

Andreas


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Does recipe specific sysrooot (or whatelse in current oe) break native dependencies?
  2017-02-13 13:47 ` Andreas Müller
@ 2017-02-13 14:36   ` Martin Jansa
  2017-02-13 15:15     ` Andreas Müller
  2017-02-13 15:24     ` Patrick Ohly
  0 siblings, 2 replies; 17+ messages in thread
From: Martin Jansa @ 2017-02-13 14:36 UTC (permalink / raw)
  To: Andreas Müller; +Cc: Patches and discussions about the oe-core layer

[-- Attachment #1: Type: text/plain, Size: 2582 bytes --]

Hi Andreas,

I think it's feature which was already there, but almost never triggered
(even in test-dependencies.sh tests), but with RSS it fails reliably.

See:
http://lists.openembedded.org/pipermail/openembedded-core/2016-July/124435.html

On Mon, Feb 13, 2017 at 2:47 PM, Andreas Müller <
schnitzeltony@googlemail.com> wrote:

> On Mon, Feb 13, 2017 at 1:26 AM, Andreas Müller
> <schnitzeltony@googlemail.com> wrote:
> > Hi
> >
> > I try to get meta-qt5-extra fit for latest oe-core (RSS). One thing
> > that causes my attention is that many recipes's configure can be fixed
> > by adding gettext-native. This seems odd to me because all of these
> > recipes depend upon kdoctools which depends on gettext-native.
> > For instance for kconfigwidgets depending on kdoctools:
> >
> > * in kdoctool'sworkdir/recipe-sysroot-native/usr/bin there is a gettext
> binary
> > * but in kconfigwidgets'sworkdir/recipe-sysroot-native/usr/bin there
> > isn't a gettext binary
> >
> > Is it possible that recipe specific sysroot broke native dependency
> chain?
> >
> > One note which might be important for this issue: In meta-qt5-extra I
> > chose a the following design decision: If there is a pair of
> > native/cross recipes
> > * each cross recipe depends on native recipe
> > * all other recipes depend on cross recipes
> >
> > This reduced maintenance efforts (I don't have to care if a recipe
> > depends on cross libs or native executables) and avoids race trouble
> > with cmake's toolchain path sequence 1. cross 2. native
> >
> > So what goes wrong here - or where am I mistaken?
> >
> Another more common examples:
>
> * in meta-qt5-extra I could fix many recipes by depending on
> qtbase-native although they already depended on qtbase (qtbase depends
> on qtbase-native)
> * right after recipe specific sysroot a dependency on qtbase-native
> was added in cmake_qt5.bbclass although it depended on qtbase already
> and qtbase depends on qtbase-native
>
> Before we carry on adding dozens of dependencies and blacklisting
> world: Are missing native dependencies a bug or a feature - or where
> am I going wrong? What scares me most is that my way of depending
> always on cross recipes in meta-qt5-extra will not work anymore
> (mentioned in my previous mail) and turn into a unmaintainable burden.
>
> Andreas
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 3557 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Does recipe specific sysrooot (or whatelse in current oe) break native dependencies?
  2017-02-13 14:36   ` Martin Jansa
@ 2017-02-13 15:15     ` Andreas Müller
  2017-02-13 15:24     ` Patrick Ohly
  1 sibling, 0 replies; 17+ messages in thread
From: Andreas Müller @ 2017-02-13 15:15 UTC (permalink / raw)
  To: Martin Jansa; +Cc: Patches and discussions about the oe-core layer

On Mon, Feb 13, 2017 at 3:36 PM, Martin Jansa <martin.jansa@gmail.com> wrote:
> Hi Andreas,
>
> I think it's feature which was already there, but almost never triggered
> (even in test-dependencies.sh tests), but with RSS it fails reliably.
>
> See:
> http://lists.openembedded.org/pipermail/openembedded-core/2016-July/124435.html
>
Yeah now I remember failing on this too - thanks for the pointer.

Andreas


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Does recipe specific sysrooot (or whatelse in current oe) break native dependencies?
  2017-02-13 14:36   ` Martin Jansa
  2017-02-13 15:15     ` Andreas Müller
@ 2017-02-13 15:24     ` Patrick Ohly
  2017-02-13 15:32       ` Martin Jansa
                         ` (2 more replies)
  1 sibling, 3 replies; 17+ messages in thread
From: Patrick Ohly @ 2017-02-13 15:24 UTC (permalink / raw)
  To: Martin Jansa; +Cc: Patches and discussions about the oe-core layer

On Mon, 2017-02-13 at 15:36 +0100, Martin Jansa wrote:
> Hi Andreas,
> 
> 
> I think it's feature which was already there, but almost never
> triggered (even in test-dependencies.sh tests), but with RSS it fails
> reliably.
> 
> 
> See:
> http://lists.openembedded.org/pipermail/openembedded-core/2016-July/124435.html

That's not quite the same, if I understand it correctly. In that email,
Richard was talking about "dependencies of that target are not needed
and not installed" and used "quilt-native" and "compiler/toolchain" as
example. In other words, if recipe foo DEPENDS on bar for getting foo
compiled, that dependency on bar gets ignored when installing "foo" into
the recipe specific sysroot because it shouldn't be needed anymore.

But the example here is a recipe foo which has a runtime dependency on
bar, so bar must be installed in addition to foo, otherwise foo will not
work.

This is where it gets tricky: do native recipes have RDEPENDS? They are
not getting packaged, so I suppose not. One could collect all RDEPENDS_*
values (regardless of the actual package), but that might be too broad
(for example, when "packaging" the native recipe wouldn't even produce
that package).

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.





^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Does recipe specific sysrooot (or whatelse in current oe) break native dependencies?
  2017-02-13 15:24     ` Patrick Ohly
@ 2017-02-13 15:32       ` Martin Jansa
  2017-02-13 15:37         ` Patrick Ohly
  2017-02-13 15:45       ` Andreas Müller
  2017-02-13 17:03       ` Patrick Ohly
  2 siblings, 1 reply; 17+ messages in thread
From: Martin Jansa @ 2017-02-13 15:32 UTC (permalink / raw)
  To: Patrick Ohly; +Cc: Patches and discussions about the oe-core layer

[-- Attachment #1: Type: text/plain, Size: 1721 bytes --]

On Mon, Feb 13, 2017 at 04:24:15PM +0100, Patrick Ohly wrote:
> On Mon, 2017-02-13 at 15:36 +0100, Martin Jansa wrote:
> > Hi Andreas,
> > 
> > 
> > I think it's feature which was already there, but almost never
> > triggered (even in test-dependencies.sh tests), but with RSS it fails
> > reliably.
> > 
> > 
> > See:
> > http://lists.openembedded.org/pipermail/openembedded-core/2016-July/124435.html
> 
> That's not quite the same, if I understand it correctly. In that email,
> Richard was talking about "dependencies of that target are not needed
> and not installed" and used "quilt-native" and "compiler/toolchain" as
> example. In other words, if recipe foo DEPENDS on bar for getting foo
> compiled, that dependency on bar gets ignored when installing "foo" into
> the recipe specific sysroot because it shouldn't be needed anymore.

I'm not sure if it's the same or not, but back then I got very rarely
missing dependency on wayland-native or qtwayland-native, now with RSS I
got 20-30 recipes failing because of missing dependency on
wayland-native, qtwayland-native or qtbase-native to stage the scanner,
moc and other native tools in recipe-sysroot-native.

And Andreas issues look similar to that.

> But the example here is a recipe foo which has a runtime dependency on
> bar, so bar must be installed in addition to foo, otherwise foo will not
> work.
> 
> This is where it gets tricky: do native recipes have RDEPENDS? They are
> not getting packaged, so I suppose not. One could collect all RDEPENDS_*
> values (regardless of the actual package), but that might be too broad
> (for example, when "packaging" the native recipe wouldn't even produce
> that package).

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Does recipe specific sysrooot (or whatelse in current oe) break native dependencies?
  2017-02-13 15:32       ` Martin Jansa
@ 2017-02-13 15:37         ` Patrick Ohly
  2017-02-13 15:52           ` Max Krummenacher
  0 siblings, 1 reply; 17+ messages in thread
From: Patrick Ohly @ 2017-02-13 15:37 UTC (permalink / raw)
  To: Martin Jansa; +Cc: Patches and discussions about the oe-core layer

On Mon, 2017-02-13 at 16:32 +0100, Martin Jansa wrote:
> On Mon, Feb 13, 2017 at 04:24:15PM +0100, Patrick Ohly wrote:
> > On Mon, 2017-02-13 at 15:36 +0100, Martin Jansa wrote:
> > > Hi Andreas,
> > > 
> > > 
> > > I think it's feature which was already there, but almost never
> > > triggered (even in test-dependencies.sh tests), but with RSS it fails
> > > reliably.
> > > 
> > > 
> > > See:
> > > http://lists.openembedded.org/pipermail/openembedded-core/2016-July/124435.html
> > 
> > That's not quite the same, if I understand it correctly. In that email,
> > Richard was talking about "dependencies of that target are not needed
> > and not installed" and used "quilt-native" and "compiler/toolchain" as
> > example. In other words, if recipe foo DEPENDS on bar for getting foo
> > compiled, that dependency on bar gets ignored when installing "foo" into
> > the recipe specific sysroot because it shouldn't be needed anymore.
> 
> I'm not sure if it's the same or not, but back then I got very rarely
> missing dependency on wayland-native or qtwayland-native, now with RSS I
> got 20-30 recipes failing because of missing dependency on
> wayland-native, qtwayland-native or qtbase-native to stage the scanner,
> moc and other native tools in recipe-sysroot-native.
> 
> And Andreas issues look similar to that.

I'm sure Andreas can clarify, but "kdoctools which depends on
gettext-native" sounds like a runtime-dependency to me (but I'm just
guessing that kdoctools wraps gettext, I don't really know).

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.





^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Does recipe specific sysrooot (or whatelse in current oe) break native dependencies?
  2017-02-13 15:24     ` Patrick Ohly
  2017-02-13 15:32       ` Martin Jansa
@ 2017-02-13 15:45       ` Andreas Müller
  2017-02-13 18:05         ` Richard Purdie
  2017-02-13 17:03       ` Patrick Ohly
  2 siblings, 1 reply; 17+ messages in thread
From: Andreas Müller @ 2017-02-13 15:45 UTC (permalink / raw)
  To: Patrick Ohly; +Cc: Patches and discussions about the oe-core layer

On Mon, Feb 13, 2017 at 4:24 PM, Patrick Ohly <patrick.ohly@intel.com> wrote:
> On Mon, 2017-02-13 at 15:36 +0100, Martin Jansa wrote:
>> Hi Andreas,
>>
>>
>> I think it's feature which was already there, but almost never
>> triggered (even in test-dependencies.sh tests), but with RSS it fails
>> reliably.
>>
>>
>> See:
>> http://lists.openembedded.org/pipermail/openembedded-core/2016-July/124435.html

Richard wrote
| What it does mean is that any recipe needing a -native recipe to build
| should list it in DEPENDS directly, not rely on other dependencies to
| pull it in for them. This applies to pkgconfig-native, intltool-native
| and also to wayland-native.

This answers my question and leaves me unhappy - I am out for a while..

Andreas


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Does recipe specific sysrooot (or whatelse in current oe) break native dependencies?
  2017-02-13 15:37         ` Patrick Ohly
@ 2017-02-13 15:52           ` Max Krummenacher
  0 siblings, 0 replies; 17+ messages in thread
From: Max Krummenacher @ 2017-02-13 15:52 UTC (permalink / raw)
  To: Patrick Ohly; +Cc: Patches and discussions about the oe-core layer

Hi

I got the same issue, formulated in the following question:
http://lists.openembedded.org/pipermail/openembedded-core/2017-February/132604.html

Here a recipe failed because a native package which were in DEPENDS
did not get the RDEPENDS of those native package installed.
Something I would have expected to happen.

Assuming such native package A gets updated and newly RDEPENDS on
something additional we will have to hunt all packages which DEPEND on
A and a the additional RDEPENDS...

Max

2017-02-13 16:37 GMT+01:00 Patrick Ohly <patrick.ohly@intel.com>:
> On Mon, 2017-02-13 at 16:32 +0100, Martin Jansa wrote:
>> On Mon, Feb 13, 2017 at 04:24:15PM +0100, Patrick Ohly wrote:
>> > On Mon, 2017-02-13 at 15:36 +0100, Martin Jansa wrote:
>> > > Hi Andreas,
>> > >
>> > >
>> > > I think it's feature which was already there, but almost never
>> > > triggered (even in test-dependencies.sh tests), but with RSS it fails
>> > > reliably.
>> > >
>> > >
>> > > See:
>> > > http://lists.openembedded.org/pipermail/openembedded-core/2016-July/124435.html
>> >
>> > That's not quite the same, if I understand it correctly. In that email,
>> > Richard was talking about "dependencies of that target are not needed
>> > and not installed" and used "quilt-native" and "compiler/toolchain" as
>> > example. In other words, if recipe foo DEPENDS on bar for getting foo
>> > compiled, that dependency on bar gets ignored when installing "foo" into
>> > the recipe specific sysroot because it shouldn't be needed anymore.
>>
>> I'm not sure if it's the same or not, but back then I got very rarely
>> missing dependency on wayland-native or qtwayland-native, now with RSS I
>> got 20-30 recipes failing because of missing dependency on
>> wayland-native, qtwayland-native or qtbase-native to stage the scanner,
>> moc and other native tools in recipe-sysroot-native.
>>
>> And Andreas issues look similar to that.
>
> I'm sure Andreas can clarify, but "kdoctools which depends on
> gettext-native" sounds like a runtime-dependency to me (but I'm just
> guessing that kdoctools wraps gettext, I don't really know).
>
> --
> Best Regards, Patrick Ohly
>
> The content of this message is my personal opinion only and although
> I am an employee of Intel, the statements I make here in no way
> represent Intel's position on the issue, nor am I authorized to speak
> on behalf of Intel on this matter.
>
>
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Does recipe specific sysrooot (or whatelse in current oe) break native dependencies?
  2017-02-13 15:24     ` Patrick Ohly
  2017-02-13 15:32       ` Martin Jansa
  2017-02-13 15:45       ` Andreas Müller
@ 2017-02-13 17:03       ` Patrick Ohly
  2017-02-13 17:06         ` Patrick Ohly
  2017-02-14 10:26         ` Patrick Ohly
  2 siblings, 2 replies; 17+ messages in thread
From: Patrick Ohly @ 2017-02-13 17:03 UTC (permalink / raw)
  To: Martin Jansa; +Cc: Patches and discussions about the oe-core layer

On Mon, 2017-02-13 at 16:24 +0100, Patrick Ohly wrote:
> On Mon, 2017-02-13 at 15:36 +0100, Martin Jansa wrote:
> > Hi Andreas,
> > 
> > 
> > I think it's feature which was already there, but almost never
> > triggered (even in test-dependencies.sh tests), but with RSS it fails
> > reliably.
> > 
> > 
> > See:
> > http://lists.openembedded.org/pipermail/openembedded-core/2016-July/124435.html
> 
> That's not quite the same, if I understand it correctly. In that email,
> Richard was talking about "dependencies of that target are not needed
> and not installed" and used "quilt-native" and "compiler/toolchain" as
> example. In other words, if recipe foo DEPENDS on bar for getting foo
> compiled, that dependency on bar gets ignored when installing "foo" into
> the recipe specific sysroot because it shouldn't be needed anymore.
> 
> But the example here is a recipe foo which has a runtime dependency on
> bar, so bar must be installed in addition to foo, otherwise foo will not
> work.
> 
> This is where it gets tricky: do native recipes have RDEPENDS? They are
> not getting packaged, so I suppose not. One could collect all RDEPENDS_*
> values (regardless of the actual package), but that might be too broad
> (for example, when "packaging" the native recipe wouldn't even produce
> that package).

Apparently RDEPENDS do work, also for native recipes. Based on some
testing, it seems that all RDEPENDS are considered, even those that
refer to packages that would normally be empty, i.e. the sysroot
potentially contains more than strictly needed, but that shouldn't be a
problem.

Andreas, does adding RDEPENDS instead of (or, where needed, in addition
to) DEPENDS fix you problem?

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.





^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Does recipe specific sysrooot (or whatelse in current oe) break native dependencies?
  2017-02-13 17:03       ` Patrick Ohly
@ 2017-02-13 17:06         ` Patrick Ohly
  2017-02-14 10:26         ` Patrick Ohly
  1 sibling, 0 replies; 17+ messages in thread
From: Patrick Ohly @ 2017-02-13 17:06 UTC (permalink / raw)
  To: Martin Jansa; +Cc: Patches and discussions about the oe-core layer

On Mon, 2017-02-13 at 18:03 +0100, Patrick Ohly wrote:
> On Mon, 2017-02-13 at 16:24 +0100, Patrick Ohly wrote:
> > On Mon, 2017-02-13 at 15:36 +0100, Martin Jansa wrote:
> > > Hi Andreas,
> > > 
> > > 
> > > I think it's feature which was already there, but almost never
> > > triggered (even in test-dependencies.sh tests), but with RSS it fails
> > > reliably.
> > > 
> > > 
> > > See:
> > > http://lists.openembedded.org/pipermail/openembedded-core/2016-July/124435.html
> > 
> > That's not quite the same, if I understand it correctly. In that email,
> > Richard was talking about "dependencies of that target are not needed
> > and not installed" and used "quilt-native" and "compiler/toolchain" as
> > example. In other words, if recipe foo DEPENDS on bar for getting foo
> > compiled, that dependency on bar gets ignored when installing "foo" into
> > the recipe specific sysroot because it shouldn't be needed anymore.
> > 
> > But the example here is a recipe foo which has a runtime dependency on
> > bar, so bar must be installed in addition to foo, otherwise foo will not
> > work.
> > 
> > This is where it gets tricky: do native recipes have RDEPENDS? They are
> > not getting packaged, so I suppose not. One could collect all RDEPENDS_*
> > values (regardless of the actual package), but that might be too broad
> > (for example, when "packaging" the native recipe wouldn't even produce
> > that package).
> 
> Apparently RDEPENDS do work, also for native recipes. Based on some
> testing, it seems that all RDEPENDS are considered, even those that
> refer to packages that would normally be empty, i.e. the sysroot
> potentially contains more than strictly needed, but that shouldn't be a
> problem.
> 
> Andreas, does adding RDEPENDS instead of (or, where needed, in addition
> to) DEPENDS fix you problem?

... and to avoid confusion: I meant adding gettext to RDEPEND_${PN} in
kdoctools, not adding it to every recipe which uses kdoctools.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.





^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Does recipe specific sysrooot (or whatelse in current oe) break native dependencies?
  2017-02-13 15:45       ` Andreas Müller
@ 2017-02-13 18:05         ` Richard Purdie
  2017-02-13 18:17           ` Andreas Müller
  2017-03-05  0:55           ` Andreas Müller
  0 siblings, 2 replies; 17+ messages in thread
From: Richard Purdie @ 2017-02-13 18:05 UTC (permalink / raw)
  To: Andreas Müller, Patrick Ohly
  Cc: Patches and discussions about the oe-core layer

On Mon, 2017-02-13 at 16:45 +0100, Andreas Müller wrote:
> On Mon, Feb 13, 2017 at 4:24 PM, Patrick Ohly <patrick.ohly@intel.com
> > wrote:
> > On Mon, 2017-02-13 at 15:36 +0100, Martin Jansa wrote:
> > > I think it's feature which was already there, but almost never
> > > triggered (even in test-dependencies.sh tests), but with RSS it
> > > fails
> > > reliably.
> > > 
> > > See:
> > > http://lists.openembedded.org/pipermail/openembedded-core/2016-Ju
> > > ly/124435.html
> Richard wrote
> > 
> > What it does mean is that any recipe needing a -native recipe to
> > build
> > should list it in DEPENDS directly, not rely on other dependencies
> > to
> > pull it in for them. This applies to pkgconfig-native, intltool-
> > native
> > and also to wayland-native.
> This answers my question and leaves me unhappy - I am out for a
> while..

To be clear, you can't depend on the build dependencies of some other
recipe to be available to your recipe just because you DEPEND on it.

The RDEPENDS issue that Patrick mentions is a different one, its a
valid recipe markup problem we have processing RDEPENDS information in
the native case.

The kind of indirection that Andreas sounds like he's been relying on
may have happened to work most of the time but I'd bet there were ways
that a semi populated sstate cache could break it.

With RSS, we encoded the sstate behaviour in a deterministic way so the
races that might have happened now do happen all the time.

The simple solution for repetitive dependencies and shortcuts is just
to create a class, list all the things you need in there and then
inherit the class in the recipes as appropriate. I'd hope that isn't
too hard to sort out...

Cheers,

Richard


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Does recipe specific sysrooot (or whatelse in current oe) break native dependencies?
  2017-02-13 18:05         ` Richard Purdie
@ 2017-02-13 18:17           ` Andreas Müller
  2017-03-05  0:55           ` Andreas Müller
  1 sibling, 0 replies; 17+ messages in thread
From: Andreas Müller @ 2017-02-13 18:17 UTC (permalink / raw)
  To: Richard Purdie; +Cc: Patches and discussions about the oe-core layer

On Mon, Feb 13, 2017 at 7:05 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> To be clear, you can't depend on the build dependencies of some other
> recipe to be available to your recipe just because you DEPEND on it.
>
Yes I expected that my fault. Together with my hack on sysroot
alignments I think this is the end of kf5-kde in meta-qt5-extra.

Andreas


^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Does recipe specific sysrooot (or whatelse in current oe) break native dependencies?
  2017-02-13 17:03       ` Patrick Ohly
  2017-02-13 17:06         ` Patrick Ohly
@ 2017-02-14 10:26         ` Patrick Ohly
  2017-02-19 22:26           ` Richard Purdie
  1 sibling, 1 reply; 17+ messages in thread
From: Patrick Ohly @ 2017-02-14 10:26 UTC (permalink / raw)
  To: Martin Jansa, Richard Purdie
  Cc: Patches and discussions about the oe-core layer

On Mon, 2017-02-13 at 18:03 +0100, Patrick Ohly wrote:
> On Mon, 2017-02-13 at 16:24 +0100, Patrick Ohly wrote:
> > On Mon, 2017-02-13 at 15:36 +0100, Martin Jansa wrote:
> > > Hi Andreas,
> > > 
> > > 
> > > I think it's feature which was already there, but almost never
> > > triggered (even in test-dependencies.sh tests), but with RSS it fails
> > > reliably.
> > > 
> > > 
> > > See:
> > > http://lists.openembedded.org/pipermail/openembedded-core/2016-July/124435.html
> > 
> > That's not quite the same, if I understand it correctly. In that email,
> > Richard was talking about "dependencies of that target are not needed
> > and not installed" and used "quilt-native" and "compiler/toolchain" as
> > example. In other words, if recipe foo DEPENDS on bar for getting foo
> > compiled, that dependency on bar gets ignored when installing "foo" into
> > the recipe specific sysroot because it shouldn't be needed anymore.
> > 
> > But the example here is a recipe foo which has a runtime dependency on
> > bar, so bar must be installed in addition to foo, otherwise foo will not
> > work.
> > 
> > This is where it gets tricky: do native recipes have RDEPENDS? They are
> > not getting packaged, so I suppose not. One could collect all RDEPENDS_*
> > values (regardless of the actual package), but that might be too broad
> > (for example, when "packaging" the native recipe wouldn't even produce
> > that package).
> 
> Apparently RDEPENDS do work, also for native recipes. Based on some
> testing, it seems that all RDEPENDS are considered, even those that
> refer to packages that would normally be empty, i.e. the sysroot
> potentially contains more than strictly needed, but that shouldn't be a
> problem.

My testing was flawed: in addition to the RDEPENDS there also was a
DEPENDS with the same entry, and despite what was said earlier about
build dependencies, that entry did have an effect. So it is a bit more
complicated.

The way I'd expect this to work for native tools is this:
     1. DEPENDS should be ignored (build dependencies like compiler
        which are not needed when using the resulting tool)
     2. RDEPENDS_${PN} needs to be used (essential runtime dependencies,
        set the same way as for the target recipes, because then
        BBCLASSEXTEND native mostly works automatically)
     3. RDEPENDS_foo for foo != ${PN} is *not* used - that's basically a
        convention, and addresses the issue I mentioned where it is
        unclear for a native recipe whether it really has such a package

But the actual implementation isn't quite doing this. Instead,
extend_recipe_sysroot() in staging.bbclass and setscene_depvalid() in
sstate.bbclass look at the task dependencies.

Here's an example, using Poky e758547db = current master:
$ bitbake wic-tools
...
$ grep gettext tmp/work/i586-poky-linux/wic-tools/1.0-r0/temp/log.do_prepare_recipe_sysroot
Considering setscene task: ['gettext-native', 'do_populate_sysroot']
  considering dependency: ['gettext-native', 'do_populate_sysroot']
Skipping setscene dependency virtual:native:/work/poky/meta/recipes-core/gettext/gettext_0.19.8.1.bb:do_populate_sysroot for installation into the sysroot
...
$ find tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native/
-name gettext
tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native/usr/share/gettext
tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native/installeddeps/gettext

"gettext" is not installed.

Let's add it to RDEPENDS_${PN} of dosfstools-native, one of the
dependencies of wic-tools:
$ echo 'RDEPENDS_dosfstools-native_append_pn-dosfstools-native = " gettext-native"' >conf/auto.conf
$ bitbake -e dosfstools-native | grep ^RDEPENDS
RDEPENDS_dosfstools-native="gettext-native"
RDEPENDS=""
RDEPENDS_kernel-base=""
RDEPENDS_dosfstools-native-staticdev="dosfstools-native-dev (= 4.1-r0)"
RDEPENDS_dosfstools-native-dev="dosfstools-native (= 4.1-r0)"

First observation: "bitbake wic-tools:do_prepare_recipe_sysroot" does
not run again. But even forcing it to run again doesn't change anything,
i.e. either "gettext-native" in RDEPENDS_dosfstools-native or
RDEPENDS_dosfstools-native are ignored.

Let's compare adding something new to RDEPENDS vs. DEPENDS:
$ echo 'DEPENDS_append_pn-dosfstools-native = " socat-native"' >conf/auto.conf
$ bitbake -g wic-tools
$ grep -e '->.*socat-native.do_populate_sysroot' task-depends.dot 
"dosfstools-native.do_prepare_recipe_sysroot" -> "socat-native.do_populate_sysroot"

$ echo 'RDEPENDS_dosfstools-native_append_pn-dosfstools-native = " gettext-native"' >conf/auto.conf
$ bitbake -g wic-tools
$ grep -e '->.*socat-native.do_populate_sysroot' task-depends.dot 

Nothing. RDEPENDS_dosfstools-native was completely ignored. To me that
looks like a missing feature or even a bug, if it was meant to work
already.

I'm leaning towards the "bug" theory. There are examples where RDEPENDS
is set explicitly for native recipes, albeit inconsistently:
meta/recipes-devtools/xmlto/xmlto_0.0.28.bb:RDEPENDS_class-native = "libxslt-native"
meta/recipes-graphics/xorg-app/mkfontdir_1.0.7.bb:RDEPENDS_${PN}_class-native += "mkfontscale-native"

Of these two, only setting RDEPENDS_class-native actually has an effect
(tested by adding socat-native to the assignment). mkfontscale-native
gets installed through some other, indirect dependency.

Digging deeper, it seems that it also depends on whether a recipe sets
PACKAGES. The insane.bbclass pkgvarcheck allows RDEPENDS (without
suffix) if PACKAGES is empty, which is the case for native recipes -
unless a recipe sets PACKAGES explicitly, as libnewt does with
PACKAGES_append.

It gets more confusing. Based on the observation about xmlto, I'd expect
that this change to libnewt should have an effect:

$ git diff meta/recipes-extended/newt
diff --git a/meta/recipes-extended/newt/libnewt_0.52.19.bb b/meta/recipes-extended/newt/libnewt_0.52.19.bb
index a26ce1fbe75..0a1d693e110 100644
--- a/meta/recipes-extended/newt/libnewt_0.52.19.bb
+++ b/meta/recipes-extended/newt/libnewt_0.52.19.bb
@@ -38,7 +38,8 @@ CLEANBROKEN = "1"
 
 export CPPFLAGS
 
-PACKAGES_prepend = "whiptail "
+PACKAGES_prepend_class-target = "whiptail "
+RDEPENDS_class-native = "socat-native"
 
 do_configure_prepend() {
     sh autogen.sh

It changes PACKAGES and RDEPENDS as expected, but tasks dependencies do
not:

$ bitbake -e libnewt-native | grep -e PACKAGES= -e ^RDEPENDS=
RDEPENDS="socat-native"
PACKAGES=""
$ bitbake -g libnewt-native
$ grep -e '->.*socat-native' task-depends.dot 

Nothing.

Here's the same test for xmlto:

$ git diff meta/recipes-devtools/xmlto
diff --git a/meta/recipes-devtools/xmlto/xmlto_0.0.28.bb
b/meta/recipes-devtools/xmlto/xmlto_0.0.28.bb
index ce5d1e0c502..9e995fe5e9d 100644
--- a/meta/recipes-devtools/xmlto/xmlto_0.0.28.bb
+++ b/meta/recipes-devtools/xmlto/xmlto_0.0.28.bb
@@ -13,7 +13,7 @@ SRC_URI[md5sum] = "a1fefad9d83499a15576768f60f847c6"
 SRC_URI[sha256sum] =
"2f986b7c9a0e9ac6728147668e776d405465284e13c74d4146c9cbc51fd8aad3"
 
 inherit autotools
-RDEPENDS_class-native = "libxslt-native"
+RDEPENDS_class-native = "libxslt-native socat-native"
 
 # xmlto needs getopt/xmllint/xsltproc/bash/tail at runtime
 RDEPENDS_${PN} = "docbook-xml-dtd4 \

$ bitbake -e xmlto-native | grep -e PACKAGES= -e ^RDEPENDS=
RDEPENDS="libxslt-native socat-native"
PACKAGES=""
$ bitbake -g xmlto-native 
$ grep -e '->.*socat-native' task-depends.dot 
"socat-native.do_populate_sysroot" -> "socat-native.do_install"
"socat-native.do_patch" -> "socat-native.do_unpack"
"socat-native.do_unpack" -> "socat-native.do_fetch"
"socat-native.do_compile" -> "socat-native.do_configure"
"socat-native.do_prepare_recipe_sysroot" -> "socat-native.do_fetch"
"socat-native.do_configure" -> "socat-native.do_prepare_recipe_sysroot"
"socat-native.do_configure" -> "socat-native.do_patch"
"socat-native.do_install" -> "socat-native.do_compile"
"xmlto-native.do_populate_sysroot" -> "socat-native.do_populate_sysroot"

If this email sounds confused, then that's because I am - sorry for that ;-}

Can someone clarify how RDEPENDS really works at the moment for native
recipes?

Now regarding DEPENDS being ignored: that's not quite the case either.

$ echo 'DEPENDS_append_pn-dosfstools-native = " socat-native"' >conf/auto.conf
$ bitbake wic-tools
...
$ find tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native/ -name socat
tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native/usr/bin/socat

In other words, a build dependency got installed although it shouldn't
really be needed.

Finally, re-running do_populate_sysroot_setscene does not remove entries
which no longer should be there. Test case:

$ echo 'DEPENDS_append_pn-dosfstools-native = " socat-native"' >conf/auto.conf
$ bitbake wic-tools:do_prepare_recipe_sysroot | cat
...
NOTE: recipe wic-tools-1.0-r0: task do_prepare_recipe_sysroot: Started
NOTE: recipe wic-tools-1.0-r0: task do_prepare_recipe_sysroot: Succeeded
NOTE: Tasks Summary: Attempted 682 tasks of which 681 didn't need to be rerun and all succeeded.
$ find tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native/ -name socat
tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native/usr/bin/socat
$ echo  >conf/auto.conf
$ bitbake wic-tools:do_prepare_recipe_sysroot | cat
...
NOTE: recipe wic-tools-1.0-r0: task do_prepare_recipe_sysroot: Started
NOTE: recipe wic-tools-1.0-r0: task do_prepare_recipe_sysroot: Succeeded
NOTE: Tasks Summary: Attempted 674 tasks of which 673 didn't need to be rerun and all succeeded.
$ find tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native/ -name socat
tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native/usr/bin/socat
$ bitbake -c clean wic-tools
...
$ bitbake wic-tools:do_prepare_recipe_sysroot | cat
...
NOTE: recipe wic-tools-1.0-r0: task do_prepare_recipe_sysroot: Started
NOTE: recipe wic-tools-1.0-r0: task do_prepare_recipe_sysroot: Succeeded
NOTE: Tasks Summary: Attempted 674 tasks of which 672 didn't need to be rerun and all succeeded.
$ find tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-native/ -name socat

Is there a missing [cleandirs] for the recipe-sysroot-native or is this
reuse of the existing content intentional?

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.





^ permalink raw reply related	[flat|nested] 17+ messages in thread

* Re: Does recipe specific sysrooot (or whatelse in current oe) break native dependencies?
  2017-02-14 10:26         ` Patrick Ohly
@ 2017-02-19 22:26           ` Richard Purdie
  2017-02-20 14:14             ` Patrick Ohly
  0 siblings, 1 reply; 17+ messages in thread
From: Richard Purdie @ 2017-02-19 22:26 UTC (permalink / raw)
  To: Patrick Ohly, Martin Jansa
  Cc: Patches and discussions about the oe-core layer

On Tue, 2017-02-14 at 11:26 +0100, Patrick Ohly wrote:
> My testing was flawed: in addition to the RDEPENDS there also was a
> DEPENDS with the same entry, and despite what was said earlier about
> build dependencies, that entry did have an effect. So it is a bit
> more
> complicated.
> 
> The way I'd expect this to work for native tools is this:
>      1. DEPENDS should be ignored (build dependencies like compiler
>         which are not needed when using the resulting tool)

Firstly. this is not true at all. Native recipes have build
dependencies just the same as target recipes. They depend on the native
compiler but you could say that is in ASSUME_PROVIDED. A recipe which
uses gettext macros would DEPENDS on gettext-native, it gets
autoreconf'd just like a target recipe and hence has the same kind of
dependency requirements. Obviously native recipe only depend on other
native recipes though. qemu-native DEPENDing on things like libsdl-
native is another example.

By comparison RDEPENDS of a native recipe are only needed after its
been compiled and we're about to run it (at least by definition).

>      2. RDEPENDS_${PN} needs to be used (essential runtime
> dependencies,
>         set the same way as for the target recipes, because then
>         BBCLASSEXTEND native mostly works automatically)
>      3. RDEPENDS_foo for foo != ${PN} is *not* used - that's
> basically a
>         convention, and addresses the issue I mentioned where it is
>         unclear for a native recipe whether it really has such a
> package
> 
> But the actual implementation isn't quite doing this. Instead,
> extend_recipe_sysroot() in staging.bbclass and setscene_depvalid() in
> sstate.bbclass look at the task dependencies.

Effectively staging.bbclass calls into sstate.bbclass and they both run
off exactly the same base dependency data which is represented by task-
depends.dot.

> Here's an example, using Poky e758547db = current master:
> $ bitbake wic-tools
> ...
> $ grep gettext tmp/work/i586-poky-linux/wic-tools/1.0-
> r0/temp/log.do_prepare_recipe_sysroot
> Considering setscene task: ['gettext-native', 'do_populate_sysroot']
>   considering dependency: ['gettext-native', 'do_populate_sysroot']
> Skipping setscene dependency virtual:native:/work/poky/meta/recipes-
> core/gettext/gettext_0.19.8.1.bb:do_populate_sysroot for installation
> into the sysroot
> ...
> $ find tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-
> native/
> -name gettext
> tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-
> native/usr/share/gettext
> tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-
> native/installeddeps/gettext
> 
> "gettext" is not installed.

wic-tools is a target recipe (whether it should be or not is a
different question). From an sstate perspective, execution of a target
recipe will not need any indirect native build tools so it is correctly
ignoring gettext-native.

> Let's add it to RDEPENDS_${PN} of dosfstools-native, one of the
> dependencies of wic-tools:
> $ echo 'RDEPENDS_dosfstools-native_append_pn-dosfstools-native = "
> gettext-native"' >conf/auto.conf
> $ bitbake -e dosfstools-native | grep ^RDEPENDS
> RDEPENDS_dosfstools-native="gettext-native"
> RDEPENDS=""
> RDEPENDS_kernel-base=""
> RDEPENDS_dosfstools-native-staticdev="dosfstools-native-dev (= 4.1-
> r0)"
> RDEPENDS_dosfstools-native-dev="dosfstools-native (= 4.1-r0)"

I checked the bitbake code and it will only see RDEPENDS, not RDEPENDS-_${PN} since PACKAGES is empty in the -native case. That explains what
you see above however I tried dropping the _${PN} piece and it still doesn't find the gettext-native dependency.

With a little more digging, I realised that base.bbclass does:

do_prepare_recipe_sysroot[deptask] = "do_populate_sysroot"

Note that this is "deptask", not "rdeptask".

The lack of the "r" means it will work off DEPENDS, not RDEPENDS.

[...]

> Nothing. RDEPENDS_dosfstools-native was completely ignored. To me 
> that looks like a missing feature or even a bug, if it was meant to 
> work already.
>
> I'm leaning towards the "bug" theory. There are examples where
> RDEPENDS
> is set explicitly for native recipes, albeit inconsistently:
> meta/recipes-devtools/xmlto/xmlto_0.0.28.bb:RDEPENDS_class-native =
> "libxslt-native"
> meta/recipes-graphics/xorg-
> app/mkfontdir_1.0.7.bb:RDEPENDS_${PN}_class-native += "mkfontscale-
> native"

I think the system is working as designed however the design is
suboptimal as I've hinted at before and the metadata is confused and
mainly appears to work by luck.

> Of these two, only setting RDEPENDS_class-native actually has an
> effect
> (tested by adding socat-native to the assignment). mkfontscale-native
> gets installed through some other, indirect dependency.
> 
> Digging deeper, it seems that it also depends on whether a recipe
> sets
> PACKAGES. The insane.bbclass pkgvarcheck allows RDEPENDS (without
> suffix) if PACKAGES is empty, which is the case for native recipes -
> unless a recipe sets PACKAGES explicitly, as libnewt does with
> PACKAGES_append.

I think my explanation above does clarify that?

> It gets more confusing. Based on the observation about xmlto, I'd
> expect
> that this change to libnewt should have an effect:
> 
> $ git diff meta/recipes-extended/newt
> diff --git a/meta/recipes-extended/newt/libnewt_0.52.19.bb
> b/meta/recipes-extended/newt/libnewt_0.52.19.bb
> index a26ce1fbe75..0a1d693e110 100644
> --- a/meta/recipes-extended/newt/libnewt_0.52.19.bb
> +++ b/meta/recipes-extended/newt/libnewt_0.52.19.bb
> @@ -38,7 +38,8 @@ CLEANBROKEN = "1"
>  
>  export CPPFLAGS
>  
> -PACKAGES_prepend = "whiptail "
> +PACKAGES_prepend_class-target = "whiptail "
> +RDEPENDS_class-native = "socat-native"
>  
>  do_configure_prepend() {
>      sh autogen.sh
> 
> It changes PACKAGES and RDEPENDS as expected, but tasks dependencies
> do
> not:
> 
> $ bitbake -e libnewt-native | grep -e PACKAGES= -e ^RDEPENDS=
> RDEPENDS="socat-native"
> PACKAGES=""
> $ bitbake -g libnewt-native
> $ grep -e '->.*socat-native' task-depends.dot 
> 
> Nothing.

This is because of the deptask verses rdeptask.

> Here's the same test for xmlto:
> 
> $ git diff meta/recipes-devtools/xmlto
> diff --git a/meta/recipes-devtools/xmlto/xmlto_0.0.28.bb
> b/meta/recipes-devtools/xmlto/xmlto_0.0.28.bb
> index ce5d1e0c502..9e995fe5e9d 100644
> --- a/meta/recipes-devtools/xmlto/xmlto_0.0.28.bb
> +++ b/meta/recipes-devtools/xmlto/xmlto_0.0.28.bb
> @@ -13,7 +13,7 @@ SRC_URI[md5sum] =
> "a1fefad9d83499a15576768f60f847c6"
>  SRC_URI[sha256sum] =
> "2f986b7c9a0e9ac6728147668e776d405465284e13c74d4146c9cbc51fd8aad3"
>  
>  inherit autotools
> -RDEPENDS_class-native = "libxslt-native"
> +RDEPENDS_class-native = "libxslt-native socat-native"
>  
>  # xmlto needs getopt/xmllint/xsltproc/bash/tail at runtime
>  RDEPENDS_${PN} = "docbook-xml-dtd4 \
> 
> $ bitbake -e xmlto-native | grep -e PACKAGES= -e ^RDEPENDS=
> RDEPENDS="libxslt-native socat-native"
> PACKAGES=""
> $ bitbake -g xmlto-native 
> $ grep -e '->.*socat-native' task-depends.dot 
> "socat-native.do_populate_sysroot" -> "socat-native.do_install"
> "socat-native.do_patch" -> "socat-native.do_unpack"
> "socat-native.do_unpack" -> "socat-native.do_fetch"
> "socat-native.do_compile" -> "socat-native.do_configure"
> "socat-native.do_prepare_recipe_sysroot" -> "socat-native.do_fetch"
> "socat-native.do_configure" -> "socat-
> native.do_prepare_recipe_sysroot"
> "socat-native.do_configure" -> "socat-native.do_patch"
> "socat-native.do_install" -> "socat-native.do_compile"
> "xmlto-native.do_populate_sysroot" -> "socat-
> native.do_populate_sysroot"
> 
> If this email sounds confused, then that's because I am - sorry for
> that ;-}
> 
> Can someone clarify how RDEPENDS really works at the moment for
> native
> recipes?
> 
> Now regarding DEPENDS being ignored: that's not quite the case
> either.
> 
> $ echo 'DEPENDS_append_pn-dosfstools-native = " socat-native"'
> >conf/auto.conf
> $ bitbake wic-tools
> ...
> $ find tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-
> native/ -name socat
> tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-
> native/usr/bin/socat
> 
> In other words, a build dependency got installed although it
> shouldn't really be needed.

We can't tell the difference between a build dependency which may
contain a library we link to and need at runtime and one that does not.

> Finally, re-running do_populate_sysroot_setscene does not remove
> entries
> which no longer should be there. Test case:
> 
> $ echo 'DEPENDS_append_pn-dosfstools-native = " socat-native"'
> >conf/auto.conf
> $ bitbake wic-tools:do_prepare_recipe_sysroot | cat
> ...
> NOTE: recipe wic-tools-1.0-r0: task do_prepare_recipe_sysroot:
> Started
> NOTE: recipe wic-tools-1.0-r0: task do_prepare_recipe_sysroot:
> Succeeded
> NOTE: Tasks Summary: Attempted 682 tasks of which 681 didn't need to
> be rerun and all succeeded.
> $ find tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-
> native/ -name socat
> tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-
> native/usr/bin/socat
> $ echo  >conf/auto.conf
> $ bitbake wic-tools:do_prepare_recipe_sysroot | cat
> ...
> NOTE: recipe wic-tools-1.0-r0: task do_prepare_recipe_sysroot:
> Started
> NOTE: recipe wic-tools-1.0-r0: task do_prepare_recipe_sysroot:
> Succeeded
> NOTE: Tasks Summary: Attempted 674 tasks of which 673 didn't need to
> be rerun and all succeeded.
> $ find tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-
> native/ -name socat
> tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-
> native/usr/bin/socat
> $ bitbake -c clean wic-tools
> ...
> $ bitbake wic-tools:do_prepare_recipe_sysroot | cat
> ...
> NOTE: recipe wic-tools-1.0-r0: task do_prepare_recipe_sysroot:
> Started
> NOTE: recipe wic-tools-1.0-r0: task do_prepare_recipe_sysroot:
> Succeeded
> NOTE: Tasks Summary: Attempted 674 tasks of which 672 didn't need to
> be rerun and all succeeded.
> $ find tmp/work/i586-poky-linux/wic-tools/1.0-r0/recipe-sysroot-
> native/ -name socat
> 
> Is there a missing [cleandirs] for the recipe-sysroot-native or is
> this
> reuse of the existing content intentional?

It is "intentional" since you can have two tasks running in parallel,
each with different dependencies which both have to be installed into
the sysroot. Neither will know about the dependencies the other has.

What does happen is that any changed/unreachable dependency will be
removed from the sysroot though since we can detect those. I wish we
could do better but since its recipe specific, not task specific, we're
stuck with this problem as far as I can tell.

Hopefully that at least explains some of what is happening. As I've
said, I do think we need work/cleanup in this area.

Cheers,

Richard




^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Does recipe specific sysrooot (or whatelse in current oe) break native dependencies?
  2017-02-19 22:26           ` Richard Purdie
@ 2017-02-20 14:14             ` Patrick Ohly
  0 siblings, 0 replies; 17+ messages in thread
From: Patrick Ohly @ 2017-02-20 14:14 UTC (permalink / raw)
  To: Richard Purdie; +Cc: Patches and discussions about the oe-core layer

On Sun, 2017-02-19 at 14:26 -0800, Richard Purdie wrote:
> On Tue, 2017-02-14 at 11:26 +0100, Patrick Ohly wrote:
> > My testing was flawed: in addition to the RDEPENDS there also was a
> > DEPENDS with the same entry, and despite what was said earlier about
> > build dependencies, that entry did have an effect. So it is a bit
> > more
> > complicated.
> > 
> > The way I'd expect this to work for native tools is this:
> >      1. DEPENDS should be ignored (build dependencies like compiler
> >         which are not needed when using the resulting tool)
> 
> Firstly. this is not true at all. Native recipes have build
> dependencies just the same as target recipes.

Yes, of course. But here I was talking about installing an already built
tool in the RSS, and in that case the build dependencies are not always
necessary anymore.

What I hadn't considered is that there's no packaging and thus also no
analysis of ELF files to detect library dependencies. Without that, one
has no choice and must install all DEPENDS, even those that only provide
build tools that aren't needed anymore.

I'm not suggesting to add such packaging. It's another big change and it
is not clear whether the reduced size of the resulting sysroots really
would outweigh the extra costs and complexity (for example,
boot-strapping the ELF analysis).

> By comparison RDEPENDS of a native recipe are only needed after its
> been compiled and we're about to run it (at least by definition).

Let's focus on RDEPENDS.

> With a little more digging, I realised that base.bbclass does:
> 
> do_prepare_recipe_sysroot[deptask] = "do_populate_sysroot"
> 
> Note that this is "deptask", not "rdeptask".
> 
> The lack of the "r" means it will work off DEPENDS, not RDEPENDS.

So just to be clear, all RDEPENDS* variables are currently ignored when
populating the RSS. I think that's what surprised me. Some sort of
support for it also in the native case would make sense, because it is
cleaner and avoids rebuilding tools when only they runtime dependencies
change.

We just need to agree on something and then make it work. I propose we
use RDEPENDS_$i for i in PACKAGES, because unsetting PACKAGES in
native.bbclass when used via BBCLASSEXTEND is a hack that already fails
in some cases (like the PACKAGES_prepend that I mentioned earlier) and
because it is more consistent with target recipes.

To handle that case that a RDEPENDS_foobar pulls in a dependency that is
irrelevant because the foobar package would have been empty and skipped
if packaging was done, PACKAGES_class-native can be used to set a
smaller list. What about RDEPENDS for native-sdk? Does it matter there?

Can do_prepare_recipe_sysroot be extended to consider both DEPENDS and
RDEPENDS_$i?

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.





^ permalink raw reply	[flat|nested] 17+ messages in thread

* Re: Does recipe specific sysrooot (or whatelse in current oe) break native dependencies?
  2017-02-13 18:05         ` Richard Purdie
  2017-02-13 18:17           ` Andreas Müller
@ 2017-03-05  0:55           ` Andreas Müller
  1 sibling, 0 replies; 17+ messages in thread
From: Andreas Müller @ 2017-03-05  0:55 UTC (permalink / raw)
  To: Richard Purdie; +Cc: Patches and discussions about the oe-core layer

[-- Attachment #1: Type: text/plain, Size: 857 bytes --]

On Mon, Feb 13, 2017 at 7:05 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> To be clear, you can't depend on the build dependencies of some other
> recipe to be available to your recipe just because you DEPEND on it.
Sorry me again - but is it possible that this behavior has changed?
With recent oe-core I see

* many configure logs finding pkg-config from recipe-sysroot-native
although the recipes do not inherit pkgconfig and do not have
pkgconfig-native in depends
* adding cmake to inherit adds "bzip2-replacement-native expat-native
xz-native zlib-native curl-native" to
recipe-sysroot-recipe-sysroot-native/sysroot-providers.

I reduced the recipe I was currently working on to no dependencies and
inherit cmake only (for test attached).

Can somebody tell me what was changed or what I am missing?

Andreas

[-- Attachment #2: supertuxkart_0.9.2.bb --]
[-- Type: application/octet-stream, Size: 423 bytes --]

LICENSE = "GPLv2 & GPLv3+ "
LIC_FILES_CHKSUM = "file://COPYING;md5=a71cb78659d60f2ced58a594cb65bfba"

inherit cmake

S = "${WORKDIR}/supertuxkart-${PV}"

SRC_URI = "http://sourceforge.net/projects/supertuxkart/files/SuperTuxKart/${PV}/supertuxkart-${PV}-src.tar.xz;protocol=https"

SRC_URI[md5sum] = "f1f5081fd41b8eeb310b4edc07b9ee12"
SRC_URI[sha256sum] = "0b080bb098a26adb552d6fd48905bcb6b1e873ef1567457d7268d7d3aaa48282"

^ permalink raw reply	[flat|nested] 17+ messages in thread

end of thread, other threads:[~2017-03-05  0:55 UTC | newest]

Thread overview: 17+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-02-13  0:26 Does recipe specific sysrooot (or whatelse in current oe) break native dependencies? Andreas Müller
2017-02-13 13:47 ` Andreas Müller
2017-02-13 14:36   ` Martin Jansa
2017-02-13 15:15     ` Andreas Müller
2017-02-13 15:24     ` Patrick Ohly
2017-02-13 15:32       ` Martin Jansa
2017-02-13 15:37         ` Patrick Ohly
2017-02-13 15:52           ` Max Krummenacher
2017-02-13 15:45       ` Andreas Müller
2017-02-13 18:05         ` Richard Purdie
2017-02-13 18:17           ` Andreas Müller
2017-03-05  0:55           ` Andreas Müller
2017-02-13 17:03       ` Patrick Ohly
2017-02-13 17:06         ` Patrick Ohly
2017-02-14 10:26         ` Patrick Ohly
2017-02-19 22:26           ` Richard Purdie
2017-02-20 14:14             ` Patrick Ohly

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.