OpenEmbedded Layer Index autobuild errors

OpenEmbedded Layer Index autobuild errors

[Daniel Dickinson]

Hi,

Since it wasn't clear from my last posting about this issue, that the
issued described was that the recipe parsing for the three pre-alpha
layers I submitted recently triggered an error condition (missing
HOSTTOOLS) which isn't something that's an issue with the layer or
recipes (or so I'm given to understand), I'm reposing the error message
and this request for the hosts to get the necessary packages so the
layer index can be updated.

Layers are:

meta-cshored - small distro along the lines of poky-tiny but for
read-only-rootfs and taking advantage of the following layers:

meta-earlyinit - instead of initramfs run 'earlyinit' to prepare things
for the main init and which runs from the kerne-mounted rootfs.

meta-runit-init - use busybox 'runit' commands as the main init.


The error message is:


Traceback (most recent call last):
  File "update_layer.py", line 716, in <module>
    main()
  File "update_layer.py", line 216, in main
    (tinfoil, tempdir) = recipeparse.init_parser(settings, branch, bitbakepath, nocheckout=options.nocheckout, logger=logger)
  File "/opt/layerindex/layerindex/recipeparse.py", line 75, in init_parser
    tinfoil = utils.setup_tinfoil(bitbakepath, enable_tracking)
  File "/opt/layerindex/layerindex/utils.py", line 144, in setup_tinfoil
    tinfoil.prepare(config_only = True)
  File "/opt/workdir/bitbake/lib/bb/tinfoil.py", line 210, in prepare
    extrafeatures)
  File "/opt/workdir/bitbake/lib/bb/main.py", line 517, in setup_bitbake
    server = start_server(servermodule, configParams, configuration, featureset)
  File "/opt/workdir/bitbake/lib/bb/main.py", line 357, in start_server
    cooker = bb.cooker.BBCooker(configuration, features)
  File "/opt/workdir/bitbake/lib/bb/cooker.py", line 195, in __init__
    self.initConfigurationData()
  File "/opt/workdir/bitbake/lib/bb/cooker.py", line 358, in initConfigurationData
    self.databuilder.parseBaseConfiguration()
  File "/opt/workdir/bitbake/lib/bb/cookerdata.py", line 299, in parseBaseConfiguration
    raise bb.BBHandledException
bb.BBHandledException
ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH, please install them in order to proceed:
  cpio chrpath gawk diffstat makeinfo

Regards,

Daniel

Re: OpenEmbedded Layer Index autobuild errors

[Max Krummenacher]

Hi

Am Sonntag, den 19.03.2017, 00:28 -0400 schrieb Daniel Dickinson:
> Hi,
> 
> Since it wasn't clear from my last posting about this issue, that the
> issued described was that the recipe parsing for the three pre-alpha
> layers I submitted recently triggered an error condition (missing
> HOSTTOOLS) which isn't something that's an issue with the layer or
> recipes (or so I'm given to understand), I'm reposing the error message
> and this request for the hosts to get the necessary packages so the
> layer index can be updated.
> 
> Layers are:
> 
> meta-cshored - small distro along the lines of poky-tiny but for
> read-only-rootfs and taking advantage of the following layers:
> 
> meta-earlyinit - instead of initramfs run 'earlyinit' to prepare things
> for the main init and which runs from the kerne-mounted rootfs.
> 
> meta-runit-init - use busybox 'runit' commands as the main init.
> 
> 
> The error message is:
> 
> 
> Traceback (most recent call last):
>   File "update_layer.py", line 716, in <module>
>     main()
>   File "update_layer.py", line 216, in main
>     (tinfoil, tempdir) = recipeparse.init_parser(settings, branch, bitbakepath,
> nocheckout=options.nocheckout, logger=logger)
>   File "/opt/layerindex/layerindex/recipeparse.py", line 75, in init_parser
>     tinfoil = utils.setup_tinfoil(bitbakepath, enable_tracking)
>   File "/opt/layerindex/layerindex/utils.py", line 144, in setup_tinfoil
>     tinfoil.prepare(config_only = True)
>   File "/opt/workdir/bitbake/lib/bb/tinfoil.py", line 210, in prepare
>     extrafeatures)
>   File "/opt/workdir/bitbake/lib/bb/main.py", line 517, in setup_bitbake
>     server = start_server(servermodule, configParams, configuration, featureset)
>   File "/opt/workdir/bitbake/lib/bb/main.py", line 357, in start_server
>     cooker = bb.cooker.BBCooker(configuration, features)
>   File "/opt/workdir/bitbake/lib/bb/cooker.py", line 195, in __init__
>     self.initConfigurationData()
>   File "/opt/workdir/bitbake/lib/bb/cooker.py", line 358, in initConfigurationData
>     self.databuilder.parseBaseConfiguration()
>   File "/opt/workdir/bitbake/lib/bb/cookerdata.py", line 299, in parseBaseConfiguration
>     raise bb.BBHandledException
> bb.BBHandledException
> ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH,
> please install them in order to proceed:
>   cpio chrpath gawk diffstat makeinfo

This recent commit tries to hide host applications which should not be used by recipes as they
should be provided from their -native packages (from DEPENDS) during the build.
http://cgit.openembedded.org/openembedded-core/commit/?id=fa764a403da34bb0ca9fa3767a9e9dba8d685965

With that change all tools which must be installed on the host need to be present, even if in your
use case some of them might not be used.
Did you install the prerequisites?
http://www.yoctoproject.org/docs/2.2.1/ref-manual/ref-manual.html#required-packages-for-the-host-dev
elopment-system

Max

> 
> Regards,
> 
> Daniel

Re: OpenEmbedded Layer Index autobuild errors

[Gary Thomas]

On 2017-03-19 10:30, Max Krummenacher wrote:
> Hi
>
> Am Sonntag, den 19.03.2017, 00:28 -0400 schrieb Daniel Dickinson:
>> Hi,
>>
>> Since it wasn't clear from my last posting about this issue, that the
>> issued described was that the recipe parsing for the three pre-alpha
>> layers I submitted recently triggered an error condition (missing
>> HOSTTOOLS) which isn't something that's an issue with the layer or
>> recipes (or so I'm given to understand), I'm reposing the error message
>> and this request for the hosts to get the necessary packages so the
>> layer index can be updated.
>>
>> Layers are:
>>
>> meta-cshored - small distro along the lines of poky-tiny but for
>> read-only-rootfs and taking advantage of the following layers:
>>
>> meta-earlyinit - instead of initramfs run 'earlyinit' to prepare things
>> for the main init and which runs from the kerne-mounted rootfs.
>>
>> meta-runit-init - use busybox 'runit' commands as the main init.
>>
>>
>> The error message is:
>>
>>
>> Traceback (most recent call last):
>>   File "update_layer.py", line 716, in <module>
>>     main()
>>   File "update_layer.py", line 216, in main
>>     (tinfoil, tempdir) = recipeparse.init_parser(settings, branch, bitbakepath,
>> nocheckout=options.nocheckout, logger=logger)
>>   File "/opt/layerindex/layerindex/recipeparse.py", line 75, in init_parser
>>     tinfoil = utils.setup_tinfoil(bitbakepath, enable_tracking)
>>   File "/opt/layerindex/layerindex/utils.py", line 144, in setup_tinfoil
>>     tinfoil.prepare(config_only = True)
>>   File "/opt/workdir/bitbake/lib/bb/tinfoil.py", line 210, in prepare
>>     extrafeatures)
>>   File "/opt/workdir/bitbake/lib/bb/main.py", line 517, in setup_bitbake
>>     server = start_server(servermodule, configParams, configuration, featureset)
>>   File "/opt/workdir/bitbake/lib/bb/main.py", line 357, in start_server
>>     cooker = bb.cooker.BBCooker(configuration, features)
>>   File "/opt/workdir/bitbake/lib/bb/cooker.py", line 195, in __init__
>>     self.initConfigurationData()
>>   File "/opt/workdir/bitbake/lib/bb/cooker.py", line 358, in initConfigurationData
>>     self.databuilder.parseBaseConfiguration()
>>   File "/opt/workdir/bitbake/lib/bb/cookerdata.py", line 299, in parseBaseConfiguration
>>     raise bb.BBHandledException
>> bb.BBHandledException
>> ERROR: The following required tools (as specified by HOSTTOOLS) appear to be unavailable in PATH,
>> please install them in order to proceed:
>>   cpio chrpath gawk diffstat makeinfo
>
> This recent commit tries to hide host applications which should not be used by recipes as they
> should be provided from their -native packages (from DEPENDS) during the build.
> http://cgit.openembedded.org/openembedded-core/commit/?id=fa764a403da34bb0ca9fa3767a9e9dba8d685965
>
> With that change all tools which must be installed on the host need to be present, even if in your
> use case some of them might not be used.
> Did you install the prerequisites?
> http://www.yoctoproject.org/docs/2.2.1/ref-manual/ref-manual.html#required-packages-for-the-host-dev
> elopment-system

The real point is that it's not his host - it's the [remote]
autobuilder tool for OpenEmbedded (IIRC)

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: OpenEmbedded Layer Index autobuild errors

[Daniel Dickinson]

On Sun, 19 Mar 2017 15:39:38 +0100
Gary Thomas <gary@mlbassoc.com> wrote:

> >
> > With that change all tools which must be installed on the host need
> > to be present, even if in your use case some of them might not be
> > used. Did you install the prerequisites?
> > http://www.yoctoproject.org/docs/2.2.1/ref-manual/ref-manual.html#required-packages-for-the-host-dev
> > elopment-system  
> 
> The real point is that it's not his host - it's the [remote]
> autobuilder tool for OpenEmbedded (IIRC)
> 

Yes, thank you.  I'm not sure it's a full autobuilder or just parses
recipes for generating the web pages, but either way it's not my host,
it's an openembedded.org host.  Apparently I'm not good at explaining
that...

Regards,

Daniel

Re: OpenEmbedded Layer Index autobuild errors

[Max Krummenacher]

Am Sonntag, den 19.03.2017, 12:04 -0400 schrieb Daniel Dickinson:
> On Sun, 19 Mar 2017 15:39:38 +0100
> Gary Thomas <gary@mlbassoc.com> wrote:
> 
> > > 
> > > With that change all tools which must be installed on the host need
> > > to be present, even if in your use case some of them might not be
> > > used. Did you install the prerequisites?
> > > http://www.yoctoproject.org/docs/2.2.1/ref-manual/ref-manual.html#required-packages-for-the-ho
> > > st-dev
> > > elopment-system  
> > 
> > The real point is that it's not his host - it's the [remote]
> > autobuilder tool for OpenEmbedded (IIRC)
> > 
> 
> Yes, thank you.  I'm not sure it's a full autobuilder or just parses
> recipes for generating the web pages, but either way it's not my host,
> it's an openembedded.org host.  Apparently I'm not good at explaining
> that...

My bad, sorry.

And it looks like the host fails to update any of the layers, e.g. also openembedded-core:
https://layers.openembedded.org/layerindex/layerupdate/323749/
Adding Paul, as he maybe knows how to get those tools onto the server.

Max

> 
> Regards,
> 
> Daniel

Re: OpenEmbedded Layer Index autobuild errors

[Paul Eggleton]

On Monday, 20 March 2017 10:51:46 AM NZDT Max Krummenacher wrote:
> Am Sonntag, den 19.03.2017, 12:04 -0400 schrieb Daniel Dickinson:
> > On Sun, 19 Mar 2017 15:39:38 +0100
> > 
> > Gary Thomas <gary@mlbassoc.com> wrote:
> > > > With that change all tools which must be installed on the host need
> > > > to be present, even if in your use case some of them might not be
> > > > used. Did you install the prerequisites?
> > > > http://www.yoctoproject.org/docs/2.2.1/ref-manual/ref-manual.html#requ
> > > > ired-packages-for-the-ho st-dev
> > > > elopment-system
> > > 
> > > The real point is that it's not his host - it's the [remote]
> > > autobuilder tool for OpenEmbedded (IIRC)
> > 
> > Yes, thank you.  I'm not sure it's a full autobuilder or just parses
> > recipes for generating the web pages, but either way it's not my host,
> > it's an openembedded.org host.  Apparently I'm not good at explaining
> > that...
> 
> My bad, sorry.
> 
> And it looks like the host fails to update any of the layers, e.g. also
> openembedded-core:
> https://layers.openembedded.org/layerindex/layerupdate/323749/
> Adding Paul, as he maybe knows how to get those tools onto the server.

Ah yes - I think I may just take the cheap way out and clear HOSTTOOLS when 
parsing in the index - we're not doing any actual building, after all, so 
we're not going to be calling any of these tools).

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: OpenEmbedded Layer Index autobuild errors

[Paul Eggleton]

On Monday, 20 March 2017 11:33:16 AM NZDT Paul Eggleton wrote:
> On Monday, 20 March 2017 10:51:46 AM NZDT Max Krummenacher wrote:
> > Am Sonntag, den 19.03.2017, 12:04 -0400 schrieb Daniel Dickinson:
> > > On Sun, 19 Mar 2017 15:39:38 +0100
> > > 
> > > Gary Thomas <gary@mlbassoc.com> wrote:
> > > > > With that change all tools which must be installed on the host need
> > > > > to be present, even if in your use case some of them might not be
> > > > > used. Did you install the prerequisites?
> > > > > http://www.yoctoproject.org/docs/2.2.1/ref-manual/ref-manual.html#re
> > > > > qu
> > > > > ired-packages-for-the-ho st-dev
> > > > > elopment-system
> > > > 
> > > > The real point is that it's not his host - it's the [remote]
> > > > autobuilder tool for OpenEmbedded (IIRC)
> > > 
> > > Yes, thank you.  I'm not sure it's a full autobuilder or just parses
> > > recipes for generating the web pages, but either way it's not my host,
> > > it's an openembedded.org host.  Apparently I'm not good at explaining
> > > that...
> > 
> > My bad, sorry.
> > 
> > And it looks like the host fails to update any of the layers, e.g. also
> > openembedded-core:
> > https://layers.openembedded.org/layerindex/layerupdate/323749/
> > Adding Paul, as he maybe knows how to get those tools onto the server.
> 
> Ah yes - I think I may just take the cheap way out and clear HOSTTOOLS when
> parsing in the index - we're not doing any actual building, after all, so
> we're not going to be calling any of these tools).

Well, perhaps I spoke too soon. We call gcc to check its version just when 
parsing, I suspect there may be others, so maybe the safest thing is to just 
install these tools. Things are rarely as simple as they first appear...

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: OpenEmbedded Layer Index autobuild errors

[Paul Eggleton]

On Monday, 20 March 2017 11:46:06 AM NZDT Paul Eggleton wrote:
> On Monday, 20 March 2017 11:33:16 AM NZDT Paul Eggleton wrote:
> > On Monday, 20 March 2017 10:51:46 AM NZDT Max Krummenacher wrote:
> > > Am Sonntag, den 19.03.2017, 12:04 -0400 schrieb Daniel Dickinson:
> > > > On Sun, 19 Mar 2017 15:39:38 +0100
> > > > 
> > > > Gary Thomas <gary@mlbassoc.com> wrote:
> > > > > > With that change all tools which must be installed on the host
> > > > > > need
> > > > > > to be present, even if in your use case some of them might not be
> > > > > > used. Did you install the prerequisites?
> > > > > > http://www.yoctoproject.org/docs/2.2.1/ref-manual/ref-manual.html#
> > > > > > re
> > > > > > qu
> > > > > > ired-packages-for-the-ho st-dev
> > > > > > elopment-system
> > > > > 
> > > > > The real point is that it's not his host - it's the [remote]
> > > > > autobuilder tool for OpenEmbedded (IIRC)
> > > > 
> > > > Yes, thank you.  I'm not sure it's a full autobuilder or just parses
> > > > recipes for generating the web pages, but either way it's not my host,
> > > > it's an openembedded.org host.  Apparently I'm not good at explaining
> > > > that...
> > > 
> > > My bad, sorry.
> > > 
> > > And it looks like the host fails to update any of the layers, e.g. also
> > > openembedded-core:
> > > https://layers.openembedded.org/layerindex/layerupdate/323749/
> > > Adding Paul, as he maybe knows how to get those tools onto the server.
> > 
> > Ah yes - I think I may just take the cheap way out and clear HOSTTOOLS
> > when
> > parsing in the index - we're not doing any actual building, after all, so
> > we're not going to be calling any of these tools).
> 
> Well, perhaps I spoke too soon. We call gcc to check its version just when
> parsing, I suspect there may be others, so maybe the safest thing is to just
> install these tools. Things are rarely as simple as they first appear...

FYI I have merged a patch for this, just waiting for it to be applied in the 
OE infrastructure now.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: OpenEmbedded Layer Index autobuild errors

[cshored]

> -----Original Message-----
> From: Paul Eggleton [mailto:paul.eggleton@linux.intel.com]
> Sent: March 20, 2017 9:28 PM
> To: openembedded-devel@lists.openembedded.org
> Cc: Max Krummenacher <max.oss.09@gmail.com>; Gary Thomas
> <gary@mlbassoc.com>; Daniel Dickinson <cshored@cshore.thecshore.com>
> Subject: Re: [oe] OpenEmbedded Layer Index autobuild errors
> 
> On Monday, 20 March 2017 11:46:06 AM NZDT Paul Eggleton wrote:
> > On Monday, 20 March 2017 11:33:16 AM NZDT Paul Eggleton wrote:
> > > On Monday, 20 March 2017 10:51:46 AM NZDT Max Krummenacher
> wrote:
> > > > Am Sonntag, den 19.03.2017, 12:04 -0400 schrieb Daniel Dickinson:
> > > > > On Sun, 19 Mar 2017 15:39:38 +0100
> > > > >
> > > > > Gary Thomas <gary@mlbassoc.com> wrote:
> > > > > > > With that change all tools which must be installed on the
> > > > > > > host need to be present, even if in your use case some of
> > > > > > > them might not be used. Did you install the prerequisites?
> > > > > > > http://www.yoctoproject.org/docs/2.2.1/ref-manual/ref-manual
> > > > > > > .html#
> > > > > > > re
> > > > > > > qu
> > > > > > > ired-packages-for-the-ho st-dev elopment-system
> > > > > >
> > > > > > The real point is that it's not his host - it's the [remote]
> > > > > > autobuilder tool for OpenEmbedded (IIRC)
> > > > >
> > > > > Yes, thank you.  I'm not sure it's a full autobuilder or just
> > > > > parses recipes for generating the web pages, but either way it's
> > > > > not my host, it's an openembedded.org host.  Apparently I'm not
> > > > > good at explaining that...
> > > >
> > > > My bad, sorry.
> > > >
> > > > And it looks like the host fails to update any of the layers, e.g.
> > > > also
> > > > openembedded-core:
> > > > https://layers.openembedded.org/layerindex/layerupdate/323749/
> > > > Adding Paul, as he maybe knows how to get those tools onto the
> server.
> > >
> > > Ah yes - I think I may just take the cheap way out and clear
> > > HOSTTOOLS when parsing in the index - we're not doing any actual
> > > building, after all, so we're not going to be calling any of these
> > > tools).
> >
> > Well, perhaps I spoke too soon. We call gcc to check its version just
> > when parsing, I suspect there may be others, so maybe the safest thing
> > is to just install these tools. Things are rarely as simple as they
first appear...
> 
> FYI I have merged a patch for this, just waiting for it to be applied in
the OE
> infrastructure now.

Thank you; it's  been working since about March 28.  Now to prepare an
announcement / RFC message.

Regards,

Daniel