Re: HOSTTOOLS and HOSTTOOLS_NONFATAL and ASSUME_PROVIDED

Re: HOSTTOOLS and HOSTTOOLS_NONFATAL and ASSUME_PROVIDED

[Robert P. J. Day]

  following up (briefly?) on richard's last post, a question or two
about HOSTTOOLS, whose ref manual entry reads:

"A space-separated list (filter) of tools on the build host that
should be allowed to be called from within build tasks. Using this
filter helps reduce the possibility of host contamination. If a tool
specified in the value of HOSTTOOLS is not found on the build host,
the OpenEmbedded build system produces an error and the build is not
started."

  i would first suggest that that opening sentence is a bit wussy ...
it's not so much that the listed tools "should be allowed to be
called", it's more, "here are the individual tools that absolutely
must exist on the host or you're not going anywhere." yes, i realize
the end of that entry gets around to that, but that should be
emphasized from the beginning. i might even throw in a link for the
reader to here:

https://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#required-packages-for-the-build-host

  also, the required presence of those tools on the host is totally
independent of whatever ASSUME_PROVIDED is set to; the
setup_hosttools_dir() routine in base.bbclass has no interest in the
value of ASSUME_PROVIDED. so even if some of those tools had been
built natively by the OE build, they would be ignored entirely. and
these required host tools are supported by symlinking from
${HOSTTOOLS_DIR}. so far, so good.

  as i read it, the difference with HOSTTOOLS_NONFATAL is that, if the
tool *does* exist on the host, that's the one that will be called
(again, via linking from ${HOSTTOOLS_DIR}). if it doesn't, then that
tool had better have been built natively as part of the build,
correct? seems obvious, but just checking.

  last part -- it only makes sense that if ASSUME_PROVIDED identifies
a recipe that should not be built, then HOSTTOOLS had better identify
whatever executables from that recipe that should be available on the
host. in short, ASSUME_PROVIDED and HOSTTOOLS do need to make some
effort to stay at least moderately in sync (as is being done now).

  i'm only yammering on about this, as the layer i'm perusing right
now has some *very* odd settings for these variables, suggesting the
people who set it up didn't quite understand how those variables
worked. at least not fully.

rday

Re: [docs] HOSTTOOLS and HOSTTOOLS_NONFATAL and ASSUME_PROVIDED

[Richard Purdie]

On Wed, 2021-03-24 at 12:06 -0400, Robert P. J. Day wrote:
>   following up (briefly?) on richard's last post, a question or two
> about HOSTTOOLS, whose ref manual entry reads:
> 
> "A space-separated list (filter) of tools on the build host that
> should be allowed to be called from within build tasks. Using this
> filter helps reduce the possibility of host contamination. If a tool
> specified in the value of HOSTTOOLS is not found on the build host,
> the OpenEmbedded build system produces an error and the build is not
> started."
> 
>   i would first suggest that that opening sentence is a bit wussy ...
> it's not so much that the listed tools "should be allowed to be
> called", it's more, "here are the individual tools that absolutely
> must exist on the host or you're not going anywhere."

It is both. It is actually really important to describe what these are
being used for as well as what it implies.

The important thing here is we explicitly list which tools from the build
host you can call from tasks. A side effect of that is you get an error
if they don't exist. So no, I disagree the opening sentence is incorrect,
you're just viewing this from a different perspective.


>  yes, i realize
> the end of that entry gets around to that, but that should be
> emphasized from the beginning. i might even throw in a link for the
> reader to here:
> 
> https://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#required-packages-for-the-build-host
> 
>   also, the required presence of those tools on the host is totally
> independent of whatever ASSUME_PROVIDED is set to; the
> setup_hosttools_dir() routine in base.bbclass has no interest in the
> value of ASSUME_PROVIDED. so even if some of those tools had been
> built natively by the OE build, they would be ignored entirely.

No, they would not. If they are built natively, they would be injected
into PATH ahead of HOSTTOOLS and used.

>  and these required host tools are supported by symlinking from
> ${HOSTTOOLS_DIR}. so far, so good.
> 
>   as i read it, the difference with HOSTTOOLS_NONFATAL is that, if the
> tool *does* exist on the host, that's the one that will be called
> (again, via linking from ${HOSTTOOLS_DIR}). if it doesn't, then that
> tool had better have been built natively as part of the build,
> correct? seems obvious, but just checking.
> 
>   last part -- it only makes sense that if ASSUME_PROVIDED identifies
> a recipe that should not be built, then HOSTTOOLS had better identify
> whatever executables from that recipe that should be available on the
> host. in short, ASSUME_PROVIDED and HOSTTOOLS do need to make some
> effort to stay at least moderately in sync (as is being done now).

They're related and need to be in sync, yes.

Cheers,

Richard

Re: [docs] HOSTTOOLS and HOSTTOOLS_NONFATAL and ASSUME_PROVIDED

[Robert P. J. Day]

[-- Attachment #1: Type: text/plain, Size: 2426 bytes --]

On Wed, 24 Mar 2021, Richard Purdie wrote:

> On Wed, 2021-03-24 at 12:06 -0400, Robert P. J. Day wrote:
> >   following up (briefly?) on richard's last post, a question or
> > two about HOSTTOOLS, whose ref manual entry reads:
> >
> > "A space-separated list (filter) of tools on the build host that
> > should be allowed to be called from within build tasks. Using this
> > filter helps reduce the possibility of host contamination. If a
> > tool specified in the value of HOSTTOOLS is not found on the build
> > host, the OpenEmbedded build system produces an error and the
> > build is not started."
> >
> >   i would first suggest that that opening sentence is a bit wussy
> > ... it's not so much that the listed tools "should be allowed to
> > be called", it's more, "here are the individual tools that
> > absolutely must exist on the host or you're not going anywhere."
>
> It is both. It is actually really important to describe what these
> are being used for as well as what it implies.
>
> The important thing here is we explicitly list which tools from the
> build host you can call from tasks. A side effect of that is you get
> an error if they don't exist. So no, I disagree the opening sentence
> is incorrect, you're just viewing this from a different perspective.

  ah, then you might want to ignore that last docs patch i sent.

> >  yes, i realize the end of that entry gets around to that, but
> > that should be emphasized from the beginning. i might even throw
> > in a link for the reader to here:
> >
> > https://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#required-packages-for-the-build-host
> >
> >   also, the required presence of those tools on the host is
> > totally independent of whatever ASSUME_PROVIDED is set to; the
> > setup_hosttools_dir() routine in base.bbclass has no interest in
> > the value of ASSUME_PROVIDED. so even if some of those tools had
> > been built natively by the OE build, they would be ignored
> > entirely.
>
> No, they would not. If they are built natively, they would be injected
> into PATH ahead of HOSTTOOLS and used.

  really? i was unaware of that, i should have checked more carefully.
i find that confusing -- effectively, this is saying that certain
tools *must* exist on the build host, but even if they do, they might
not be the ones used? that seems counter-intuitive.

rday

Re: [docs] HOSTTOOLS and HOSTTOOLS_NONFATAL and ASSUME_PROVIDED

[Richard Purdie]

On Wed, 2021-03-24 at 12:49 -0400, Robert P. J. Day wrote:
> On Wed, 24 Mar 2021, Richard Purdie wrote:
> > On Wed, 2021-03-24 at 12:06 -0400, Robert P. J. Day wrote:
> > >  yes, i realize the end of that entry gets around to that, but
> > > that should be emphasized from the beginning. i might even throw
> > > in a link for the reader to here:
> > > 
> > > https://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#required-packages-for-the-build-host
> > > 
> > >   also, the required presence of those tools on the host is
> > > totally independent of whatever ASSUME_PROVIDED is set to; the
> > > setup_hosttools_dir() routine in base.bbclass has no interest in
> > > the value of ASSUME_PROVIDED. so even if some of those tools had
> > > been built natively by the OE build, they would be ignored
> > > entirely.
> > 
> > No, they would not. If they are built natively, they would be injected
> > into PATH ahead of HOSTTOOLS and used.
> 
>   really? i was unaware of that, i should have checked more carefully.
> i find that confusing -- effectively, this is saying that certain
> tools *must* exist on the build host, but even if they do, they might
> not be the ones used? that seems counter-intuitive.

If you add something into DEPENDS, it is used instead of what is on the
host as long as it is not in ASSUME_PROVIDED.

Adding something to DEPENDS and it not being used instead replaced by
the host's tools would seem worse...

Cheers,

Richard

Re: [docs] HOSTTOOLS and HOSTTOOLS_NONFATAL and ASSUME_PROVIDED

[Robert P. J. Day]

[-- Attachment #1: Type: text/plain, Size: 2830 bytes --]

On Wed, 24 Mar 2021, Richard Purdie wrote:

> On Wed, 2021-03-24 at 12:49 -0400, Robert P. J. Day wrote:
> > On Wed, 24 Mar 2021, Richard Purdie wrote:
> > > On Wed, 2021-03-24 at 12:06 -0400, Robert P. J. Day wrote:
> > > >  yes, i realize the end of that entry gets around to that, but
> > > > that should be emphasized from the beginning. i might even throw
> > > > in a link for the reader to here:
> > > >
> > > > https://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#required-packages-for-the-build-host
> > > >
> > > >   also, the required presence of those tools on the host is
> > > > totally independent of whatever ASSUME_PROVIDED is set to; the
> > > > setup_hosttools_dir() routine in base.bbclass has no interest in
> > > > the value of ASSUME_PROVIDED. so even if some of those tools had
> > > > been built natively by the OE build, they would be ignored
> > > > entirely.
> > >
> > > No, they would not. If they are built natively, they would be injected
> > > into PATH ahead of HOSTTOOLS and used.
> >
> >   really? i was unaware of that, i should have checked more carefully.
> > i find that confusing -- effectively, this is saying that certain
> > tools *must* exist on the build host, but even if they do, they might
> > not be the ones used? that seems counter-intuitive.
>
> If you add something into DEPENDS, it is used instead of what is on
> the host as long as it is not in ASSUME_PROVIDED.
>
> Adding something to DEPENDS and it not being used instead replaced
> by the host's tools would seem worse...

  i was about to say that makes no sense until i think i finally clued
in, so let me try to summarize what seems logical (i'll eventually
verify this against the actual code ... someday).

  obviously, HOSTTOOLS represents what absolutely has to be on the
build system -- effectively, minimal host requirements to bootstrap
the build. so the very early part of the build verifies the existence
of these tools on the host, then downloads and configures enough to
start the real part of the build.

  at this point, the search path is extended (prepended) to now start
using the tools built natively. (if i was explaining this to someone
else, i'd say something like, "you need the host git executable to
grab source code, including git itself, but once you build those
tools natively, you'll switch to the natively-built git.")

  *now* that makes sense since, while the host tools might vary in
terms of version, tools compiled by the build will be absolutely
predictable, thereby guaranteeing a stable build environment. in
short, the stuff specified in HOSTTOOLS is needed only early on, then
it's superseded by the built native tools.

  except where ASSUME_PROVIDED comes in, but i think that's clear as
well.

rday

Re: [docs] HOSTTOOLS and HOSTTOOLS_NONFATAL and ASSUME_PROVIDED

[Richard Purdie]

On Wed, 2021-03-24 at 16:06 -0400, Robert P. J. Day wrote:
> On Wed, 24 Mar 2021, Richard Purdie wrote:
> 
> > On Wed, 2021-03-24 at 12:49 -0400, Robert P. J. Day wrote:
> > > On Wed, 24 Mar 2021, Richard Purdie wrote:
> > > > On Wed, 2021-03-24 at 12:06 -0400, Robert P. J. Day wrote:
> > > > >  yes, i realize the end of that entry gets around to that, but
> > > > > that should be emphasized from the beginning. i might even throw
> > > > > in a link for the reader to here:
> > > > > 
> > > > > https://www.yoctoproject.org/docs/latest/ref-manual/ref-manual.html#required-packages-for-the-build-host
> > > > > 
> > > > >   also, the required presence of those tools on the host is
> > > > > totally independent of whatever ASSUME_PROVIDED is set to; the
> > > > > setup_hosttools_dir() routine in base.bbclass has no interest in
> > > > > the value of ASSUME_PROVIDED. so even if some of those tools had
> > > > > been built natively by the OE build, they would be ignored
> > > > > entirely.
> > > > 
> > > > No, they would not. If they are built natively, they would be injected
> > > > into PATH ahead of HOSTTOOLS and used.
> > > 
> > >   really? i was unaware of that, i should have checked more carefully.
> > > i find that confusing -- effectively, this is saying that certain
> > > tools *must* exist on the build host, but even if they do, they might
> > > not be the ones used? that seems counter-intuitive.
> > 
> > If you add something into DEPENDS, it is used instead of what is on
> > the host as long as it is not in ASSUME_PROVIDED.
> > 
> > Adding something to DEPENDS and it not being used instead replaced
> > by the host's tools would seem worse...
> 
>   i was about to say that makes no sense until i think i finally clued
> in, so let me try to summarize what seems logical (i'll eventually
> verify this against the actual code ... someday).
> 
>   obviously, HOSTTOOLS represents what absolutely has to be on the
> build system -- effectively, minimal host requirements to bootstrap
> the build. so the very early part of the build verifies the existence
> of these tools on the host, then downloads and configures enough to
> start the real part of the build.
> 
>   at this point, the search path is extended (prepended) to now start
> using the tools built natively. (if i was explaining this to someone
> else, i'd say something like, "you need the host git executable to
> grab source code, including git itself, but once you build those
> tools natively, you'll switch to the natively-built git.")
> 
>   *now* that makes sense since, while the host tools might vary in
> terms of version, tools compiled by the build will be absolutely
> predictable, thereby guaranteeing a stable build environment. in
> short, the stuff specified in HOSTTOOLS is needed only early on, then
> it's superseded by the built native tools.
> 
>   except where ASSUME_PROVIDED comes in, but i think that's clear as
> well.

Right. We switch to our native tools in some cases and not others, it 
depends how stable/predictable those tools are. In general we don't use 
git-native but some recipes have a specific make-native DEPENDS for example.

You'd think make was stable, but no...

Cheers,

Richard