RSS difficulties

RSS difficulties

[Phil Blundell]

I'm having a few problems adapting our build setup to work with recipe
specific sysroots.  More specifically, I am finding it difficult to get
the correct set of components installed into the recipe-sysroot in all
cases.  In some cases I am getting too much (which causes build
failures if oe attempts to stage two things that install the same file)
and in other cases I am not getting enough.

Could somebody explain, ideally using only small words, how the set of
things to populate in the recipe-sysroot is determined?  I assume it is
based on the task dependency graph somehow but I'm not quite sure which
relationships I ought to be looking at.

thanks

p.

Re: RSS difficulties

[Richard Purdie]

On Fri, 2017-02-03 at 11:10 +0000, Phil Blundell wrote:
> I'm having a few problems adapting our build setup to work with
> recipe specific sysroots.  More specifically, I am finding it
> difficult to get the correct set of components installed into the
> recipe-sysroot in all cases.  In some cases I am getting too much
> (which causes build failures if oe attempts to stage two things that
> install the same file) and in other cases I am not getting enough.
> 
> Could somebody explain, ideally using only small words, how the set
> of things to populate in the recipe-sysroot is determined?  I assume
> it is based on the task dependency graph somehow but I'm not quite
> sure which relationships I ought to be looking at.

It comes down to what setscene_depvalid() in sstate.bbclass returns. If
that returns False a given dependency is installed, it if returns True,
it is skipped and not installed.

Reading that function should give some clues as to the logic it
applies. It operates not on a full task graph but on version collapsed
only to sstate tasks.

The logic should mirror what would get installed from sstate if that
sstate were available, hence the logic is somewhat tested and not just
new for RSS. That doesn't mean its right in all cases though!

If that doesn't help, perhaps some specific examples of what you think
should be being installed and isn't or things which are being installed
and shouldn't might help.

Cheers,

Richard

Re: RSS difficulties

[Phil Blundell]

On Fri, 2017-02-03 at 11:20 +0000, Richard Purdie wrote:
> 
> It comes down to what setscene_depvalid() in sstate.bbclass returns.
> If
> that returns False a given dependency is installed, it if returns
> True,
> it is skipped and not installed.
> 
> Reading that function should give some clues as to the logic it
> applies. It operates not on a full task graph but on version
> collapsed
> only to sstate tasks.

Ah, right, thanks.  The particular piece of magic I was missing was:

            # Nothing need depend on libc-initial/gcc-cross-initial
            if "-initial" in taskdependees[task][0]:
                continue

which I think explains all the difficulties I was having.

p.

Re: RSS difficulties

[Phil Blundell]

[-- Attachment #1: Type: text/plain, Size: 1540 bytes --]

On Fri, 2017-02-03 at 12:29 +0000, Phil Blundell wrote:
> Ah, right, thanks.  The particular piece of magic I was missing was:
> 
>             # Nothing need depend on libc-initial/gcc-cross-initial
>             if "-initial" in taskdependees[task][0]:
>                 continue
> 
> which I think explains all the difficulties I was having.

Actually, that explains most of my difficulties but not all of them. 
The remaining ones seem to come down to missing dependencies but I am
not quite clear on why the dependencies are missing.

The problematic recipes are Java ones.  The Java toolchain, as you
might expect, has a fairly complicated dependency tree but essentially:

- any Java package has DEPENDS = "virtual/javac-native", which is
PROVIDEd by ecj-bootstrap-native.bb

- ecj-bootstrap-native.bb is basically just a wrapper script which runs
the Java compiler.  The actual compiler implementation is in libecj-
bootstrap-native.bb and ecj-bootstrap-native.bb RDEPENDS on that.

If I run "bitbake -D -D ..." then I see:

DEBUG: Added runtime dependencies ['libecj-bootstrap-native'] for
.../meta-java/recipes-core/ecj/ecj-bootstrap-native.bb

but this dependency edge doesn't show up in task-depends.dot and, sure
enough, libecj-bootstrap-native isn't getting installed into the
recipe-sysroot. 

Is there some other piece of logic that is squashing these
dependencies?  Or is there somewhere else that I ought to be looking
for them?

thanks

p.

[-- Attachment #2: Type: text/html, Size: 2086 bytes --]

Re: RSS difficulties

[Richard Purdie]

On Mon, 2017-02-06 at 13:42 +0000, Phil Blundell wrote:
> On Fri, 2017-02-03 at 12:29 +0000, Phil Blundell wrote:
> > Ah, right, thanks.  The particular piece of magic I was missing
> > was:
> > 
> >             # Nothing need depend on libc-initial/gcc-cross-initial
> >             if "-initial" in taskdependees[task][0]:
> >                 continue
> > 
> > which I think explains all the difficulties I was having.
> Actually, that explains most of my difficulties but not all of them.
> The remaining ones seem to come down to missing dependencies but I am
> not quite clear on why the dependencies are missing.
> 
> The problematic recipes are Java ones. The Java toolchain, as you
> might expect, has a fairly complicated dependency tree but
> essentially:
> 
> - any Java package has DEPENDS = "virtual/javac-native", which is
> PROVIDEd by ecj-bootstrap-native.bb
> 
> - ecj-bootstrap-native.bb is basically just a wrapper script which
> runs the Java compiler. The actual compiler implementation is in
> libecj-bootstrap-native.bb and ecj-bootstrap-native.bb RDEPENDS on
> that.
> 
> If I run "bitbake -D -D ..." then I see:
> 
> DEBUG: Added runtime dependencies ['libecj-bootstrap-native'] for
> .../meta-java/recipes-core/ecj/ecj-bootstrap-native.bb
> 
> but this dependency edge doesn't show up in task-depends.dot and,
> sure enough, libecj-bootstrap-native isn't getting installed into the
> recipe-sysroot.
> 
> Is there some other piece of logic that is squashing these
> dependencies? Or is there somewhere else that I ought to be looking
> for them?

If its not in task-depends.dot, there is a much higher level problem
with bitbake itself not seeing the dependencies before sstate/RSS gets
anywhere near them.

I'm going to guess its some combination of -native and RDEPENDS
together which are known to not work entirely "correctly".

Some example open bugs:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=10113
https://bugzilla.yoctoproject.org/show_bug.cgi?id=6317

The trouble is that for native, PACKAGES = "" so how would bitbake
known which RDEPENDS_<pkg> to look at? What exists today is all a bit
of a hack which works for our common cases but has problems. I've been
meaning to find time to take a good look at it and come up with a
proper plan but so far it hasn't made the top of the agenda.

What happens if you set DEPENDS?

Cheers,

Richard

Re: RSS difficulties

[Phil Blundell]

[-- Attachment #1: Type: text/plain, Size: 826 bytes --]

On Mon, 2017-02-06 at 14:27 +0000, Richard Purdie wrote:
> What happens if you set DEPENDS?

With DEPENDS I do at least get some entries for ecj-bootstrap-native in
task-depends.dot.  The relevant ones seem to be:

"gnujaf-native.do_prepare_recipe_sysroot" -> "ecj-bootstrap-
native.do_populate_sysroot"
"ecj-bootstrap-native.do_prepare_recipe_sysroot" -> "libecj-bootstrap-
native.do_populate_sysroot"

which looks fairly encouraging.  And in log.do_prepare_recipe_sysroot I
see:

Considering setscene task: ['libecj-bootstrap-native',
'do_populate_sysroot']
  considering dependency: ['ecj-bootstrap-native',
'do_populate_sysroot']
Adding dependency on libecj-bootstrap-native

But sadly the contents of libecj-bootstrap-native doesn't actually show
up in gnujaf-native's recipe-sysroot-native.

p.

[-- Attachment #2: Type: text/html, Size: 1106 bytes --]

Re: RSS difficulties

[Phil Blundell]

[-- Attachment #1: Type: text/plain, Size: 269 bytes --]

By the way, this message:

                   msgbuf.append("Skipping setscene dependency %s for
installation into the sysroot")
 
is not as illuminating as it could be.  I suppose it wants "%
setscenedeps[datadep][0])" on the end. :-}

p.

[-- Attachment #2: Type: text/html, Size: 474 bytes --]

Re: RSS difficulties

[Phil Blundell]

[-- Attachment #1: Type: text/plain, Size: 832 bytes --]

On Mon, 2017-02-06 at 14:44 +0000, Phil Blundell wrote:
> But sadly the contents of libecj-bootstrap-native doesn't actually
> show up in gnujaf-native's recipe-sysroot-native.

A bit of further sleuthing reveals that although this problem only
started with rss the blame lies somewhere else.  It transpires that
meta-java's java-library.bbclass contains the somewhat epically
unhelpful (in this context) fragment:

# Java "source" distributions often contain precompiled things
# we want to delete first.
do_removebinaries() {
  find ${WORKDIR} -name "*.jar" -exec rm {} \;
  find ${WORKDIR} -name "*.class" -exec rm {} \;
}

addtask removebinaries after do_unpack before do_patch

And, since recipe-sysroot-native is inside ${WORKDIR}, this blows away
everything that rss carefully placed there.  Doh.

p.

[-- Attachment #2: Type: text/html, Size: 1108 bytes --]

Re: RSS difficulties

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 328 bytes --]

On 6 February 2017 at 16:55, Phil Blundell <pb@pbcl.net> wrote:

> And, since recipe-sysroot-native is inside ${WORKDIR}, this blows away
> everything that rss carefully placed there. Doh.
>

Ha, nice.  I'm sure that was well intended but blasting all of ${WORKDIR}
seems overkeen.  I bet that was fun to chase!

Ross

[-- Attachment #2: Type: text/html, Size: 747 bytes --]

Re: RSS difficulties

[Khem Raj]

On Mon, Feb 6, 2017 at 8:55 AM, Phil Blundell <pb@pbcl.net> wrote:
> On Mon, 2017-02-06 at 14:44 +0000, Phil Blundell wrote:
>
> But sadly the contents of libecj-bootstrap-native doesn't actually show up
> in gnujaf-native's recipe-sysroot-native.
>
>
> A bit of further sleuthing reveals that although this problem only started
> with rss the blame lies somewhere else. It transpires that meta-java's
> java-library.bbclass contains the somewhat epically unhelpful (in this
> context) fragment:
>
> # Java "source" distributions often contain precompiled things
> # we want to delete first.
> do_removebinaries() {
>   find ${WORKDIR} -name "*.jar" -exec rm {} \;
>   find ${WORKDIR} -name "*.class" -exec rm {} \;
> }
>
> addtask removebinaries after do_unpack before do_patch
>
> And, since recipe-sysroot-native is inside ${WORKDIR}, this blows away
> everything that rss carefully placed there. Doh.

good job. yeah this pre-rss assumptions, I also had another issue
where clang cross compiler was happily searching native sysroot for
headers. Not rss fault but exposed a latent problem.