Questions on HOSTTOOLS and the documented host packages in the manual

Questions on HOSTTOOLS and the documented host packages in the manual

[Gunnar Andersson]

Hi everyone

To get repeatable quality builds we ensure the build agents in CI are
identical and that they install a _minimal_ list of packages taken from the
instructions in the Yocto manual [1].  But from time to time some dependency
is missing, especially since things change in Yocto over time (e.g.
gzipnative class, and such).   So we were even missing gzip in one build
failure.  Now I couldn't confirm that the gzip binary is indeed missing - it
seems to be on our standard image, but possibly the configure step was
looking for some -devel package, headers or whatnot.

Still we have had our share of "works on my machine" answers from users that
don't use a controlled, docker-container like that of the CI build agents.

Sidestepping, since I anticipate this in your response:  I know there's
continuous work on repeatability, and tried to read what I can in the manual
on the new HOSTTOOL variable.  The mechanics of $PATH and softlinking the
binaries are clear, but I think I missed the intended usage and process
around it.   First, is it expected/recommended that each layer modifies this
variable?  If so, would layers add new host dependencies, or is the
intention to have layers further limit this with its own whitelist but still
based on a predefined set coming from the Poky layer?

I get that technically speaking you build in a lot of freedom, but from a
process perspective I think if the Yocto project defines a minimal set of
required host tools, then a Yocto-based project (like GDP) could take the
stance that all other dependencies must come from DEPENDS declaration on a
-native package.   Unless there is a clear agreed baseline, it's difficult
to file a bug on a recipe with respect to their -native dependencies.

Please confirm, is the manual (I refer to the Mega Manual [1]) intended to
be such a canonical list of host dependencies?

But if so:  Is it intentional or a bug that gzip (and bzip2, perhaps others)
is listed to be installed for Fedora, but not listed to be installed for
Ubuntu/Debian, see [1]?   Maybe the idea was that some things are always
there by default on those distros, but that seems brittle.  Shouldn't every
package be listed very explicitly?

Appreciate your comments...

- Gunnar

[1] http://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#packages

Re: Questions on HOSTTOOLS and the documented host packages in the manual

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 2102 bytes --]

On 10 July 2017 at 12:40, Gunnar Andersson <gandersson@genivi.org> wrote:

> Sidestepping, since I anticipate this in your response:  I know there's
> continuous work on repeatability, and tried to read what I can in the
> manual
> on the new HOSTTOOL variable.  The mechanics of $PATH and softlinking the
> binaries are clear, but I think I missed the intended usage and process
> around it.   First, is it expected/recommended that each layer modifies
> this
> variable?  If so, would layers add new host dependencies, or is the
> intention to have layers further limit this with its own whitelist but
> still
> based on a predefined set coming from the Poky layer?
>

Layers can extend the variables if they have explicit host dependencies,
but I'd be surprised and concerned if this was a common occurrence.

For example I see that meta-oe adds "id" to HOSTTOOLS (for the crash
recipe), but if that was submitted to oe-core it would be rejected.  The
only reason as far as I can tell to call id is to get the current uid/gid,
and that then introduces another vector of non-determinism in builds.  We
recommend that be patched out and replaced with constants: see also the
kernel build which by default uses whoami to embed the username of the
kernel builder, but we export KBUILD_BUILD_USER=oe-user to remove that
source of non-determinism.

Please confirm, is the manual (I refer to the Mega Manual [1]) intended to
> be such a canonical list of host dependencies?
>
> But if so:  Is it intentional or a bug that gzip (and bzip2, perhaps
> others)
> is listed to be installed for Fedora, but not listed to be installed for
> Ubuntu/Debian, see [1]?   Maybe the idea was that some things are always
> there by default on those distros, but that seems brittle.  Shouldn't every
> package be listed very explicitly?
>

Yes, please file a bug for missing ones.  The lists are intended to be
packages installed over the base install, but container images being *very*
minimal are often missing dependencies so we need to be sure everything is
covered.

Ross

[-- Attachment #2: Type: text/html, Size: 2765 bytes --]