setscene fetch failures are fatal

setscene fetch failures are fatal

[Joshua Watt]

I'm having an issue with the _setscene tasks failing to fetch causing
our builds to fail. For example: we see the following in our CI build
logs:

 WARNING: flac-1.3.1-r0 do_package_write_ipk_setscene: Failed to fetch
URL file://07/sstate:flac:cortexa15hf-neon-poky-linux-gnueabi:1.3.1:r0:cortexa15hf-neon:3:07fe4a22caf5e326a98e397f7923060a_package_write_ipk.tgz.siginfo;downloadfilename=07/sstate:flac:cortexa15hf-neon-poky-linux-gnueabi:1.3.1:r0:cortexa15hf-neon:3:07fe4a22caf5e326a98e397f7923060a_package_write_ipk.tgz.siginfo,
attempting MIRRORS if available
 ERROR: flac-1.3.1-r0 do_package_write_ipk_setscene: Fetcher failure:
Unable to find file
file://07/sstate:flac:cortexa15hf-neon-poky-linux-gnueabi:1.3.1:r0:cortexa15hf-neon:3:07fe4a22caf5e326a98e397f7923060a_package_write_ipk.tgz.siginfo;downloadfilename=07/sstate:flac:cortexa15hf-neon-poky-linux-gnueabi:1.3.1:r0:cortexa15hf-neon:3:07fe4a22caf5e326a98e397f7923060a_package_write_ipk.tgz.siginfo
anywhere. The paths that were searched were:
  /tmp/tmp.S3vCIuI7SE/sstate

The build carries on as normal and even builds everything
successfully, but the final error code returned from bitbake is '1'.
This is particularly annoying because A) this happens a lot and B) the
build did actually produce usable outputs so failing isn't really
accurate.

Near as I can tell, this is caused because the knotty UI treats any
logging.error() message as fatal and will have a non-zero return code
if one occurs.

I also realize this is probably mostly caused by an infrastructure
problem on our end, but I don't seem to be able to get anyone to
magically make that better so that this failure stops happening :(

Does anyone have any ideas for things that could be done to mitigate
this? So far, the only thing I've come up with is to modify the
bitbake Fetch.download() so that an argument can be passed that says
whether to use logger.error() or logger.warning() when reporting an
error, but that feels really ugly.

Thanks,
Joshua Watt

Re: setscene fetch failures are fatal

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2892 bytes --]

There are patches for this on the ML from long time ago. They were rejected
that it's just hiding the real infrastructure issue which should be fixed,
but I keep them in my builds as well because as you say, errors like this
triggering the CI build failures of unrelated changes is quite annoying.

https://github.com/shr-distribution/bitbake/commit/75858b7f4f03137646cb934b8e3cde0807493eed
+
http://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/master&id=dec7674a1cb501ee8e4f864dcb4adc307c10c159

On Sat, Jun 29, 2019 at 12:23 AM Joshua Watt <jpewhacker@gmail.com> wrote:

> I'm having an issue with the _setscene tasks failing to fetch causing
> our builds to fail. For example: we see the following in our CI build
> logs:
>
>  WARNING: flac-1.3.1-r0 do_package_write_ipk_setscene: Failed to fetch
> URL
> file://07/sstate:flac:cortexa15hf-neon-poky-linux-gnueabi:1.3.1:r0:cortexa15hf-neon:3:07fe4a22caf5e326a98e397f7923060a_package_write_ipk.tgz.siginfo;downloadfilename=07/sstate:flac:cortexa15hf-neon-poky-linux-gnueabi:1.3.1:r0:cortexa15hf-neon:3:07fe4a22caf5e326a98e397f7923060a_package_write_ipk.tgz.siginfo,
> attempting MIRRORS if available
>  ERROR: flac-1.3.1-r0 do_package_write_ipk_setscene: Fetcher failure:
> Unable to find file
>
> file://07/sstate:flac:cortexa15hf-neon-poky-linux-gnueabi:1.3.1:r0:cortexa15hf-neon:3:07fe4a22caf5e326a98e397f7923060a_package_write_ipk.tgz.siginfo;downloadfilename=07/sstate:flac:cortexa15hf-neon-poky-linux-gnueabi:1.3.1:r0:cortexa15hf-neon:3:07fe4a22caf5e326a98e397f7923060a_package_write_ipk.tgz.siginfo
> anywhere. The paths that were searched were:
>   /tmp/tmp.S3vCIuI7SE/sstate
>
> The build carries on as normal and even builds everything
> successfully, but the final error code returned from bitbake is '1'.
> This is particularly annoying because A) this happens a lot and B) the
> build did actually produce usable outputs so failing isn't really
> accurate.
>
> Near as I can tell, this is caused because the knotty UI treats any
> logging.error() message as fatal and will have a non-zero return code
> if one occurs.
>
> I also realize this is probably mostly caused by an infrastructure
> problem on our end, but I don't seem to be able to get anyone to
> magically make that better so that this failure stops happening :(
>
> Does anyone have any ideas for things that could be done to mitigate
> this? So far, the only thing I've come up with is to modify the
> bitbake Fetch.download() so that an argument can be passed that says
> whether to use logger.error() or logger.warning() when reporting an
> error, but that feels really ugly.
>
> Thanks,
> Joshua Watt
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>

[-- Attachment #2: Type: text/html, Size: 3753 bytes --]

Re: setscene fetch failures are fatal

[Joshua Watt]

On 6/28/19 5:43 PM, Martin Jansa wrote:
> There are patches for this on the ML from long time ago. They were 
> rejected that it's just hiding the real infrastructure issue which 
> should be fixed, but I keep them in my builds as well because as you 
> say, errors like this triggering the CI build failures of unrelated 
> changes is quite annoying.
>
> https://github.com/shr-distribution/bitbake/commit/75858b7f4f03137646cb934b8e3cde0807493eed
> +
> http://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/master&id=dec7674a1cb501ee8e4f864dcb4adc307c10c159

Thanks. I found those original patches on the mailing list to review the 
arguments. I've proposed a different solution here:

http://lists.openembedded.org/pipermail/bitbake-devel/2019-July/020105.html

with test cases here:

http://lists.openembedded.org/pipermail/openembedded-core/2019-July/284285.html

We'll see if that is a more palatable alternative.

>
> On Sat, Jun 29, 2019 at 12:23 AM Joshua Watt <jpewhacker@gmail.com 
> <mailto:jpewhacker@gmail.com>> wrote:
>
>     I'm having an issue with the _setscene tasks failing to fetch causing
>     our builds to fail. For example: we see the following in our CI build
>     logs:
>
>      WARNING: flac-1.3.1-r0 do_package_write_ipk_setscene: Failed to fetch
>     URL
>     file://07/sstate:flac:cortexa15hf-neon-poky-linux-gnueabi:1.3.1:r0:cortexa15hf-neon:3:07fe4a22caf5e326a98e397f7923060a_package_write_ipk.tgz.siginfo;downloadfilename=07/sstate:flac:cortexa15hf-neon-poky-linux-gnueabi:1.3.1:r0:cortexa15hf-neon:3:07fe4a22caf5e326a98e397f7923060a_package_write_ipk.tgz.siginfo,
>     attempting MIRRORS if available
>      ERROR: flac-1.3.1-r0 do_package_write_ipk_setscene: Fetcher failure:
>     Unable to find file
>     file://07/sstate:flac:cortexa15hf-neon-poky-linux-gnueabi:1.3.1:r0:cortexa15hf-neon:3:07fe4a22caf5e326a98e397f7923060a_package_write_ipk.tgz.siginfo;downloadfilename=07/sstate:flac:cortexa15hf-neon-poky-linux-gnueabi:1.3.1:r0:cortexa15hf-neon:3:07fe4a22caf5e326a98e397f7923060a_package_write_ipk.tgz.siginfo
>     anywhere. The paths that were searched were:
>       /tmp/tmp.S3vCIuI7SE/sstate
>
>     The build carries on as normal and even builds everything
>     successfully, but the final error code returned from bitbake is '1'.
>     This is particularly annoying because A) this happens a lot and B) the
>     build did actually produce usable outputs so failing isn't really
>     accurate.
>
>     Near as I can tell, this is caused because the knotty UI treats any
>     logging.error() message as fatal and will have a non-zero return code
>     if one occurs.
>
>     I also realize this is probably mostly caused by an infrastructure
>     problem on our end, but I don't seem to be able to get anyone to
>     magically make that better so that this failure stops happening :(
>
>     Does anyone have any ideas for things that could be done to mitigate
>     this? So far, the only thing I've come up with is to modify the
>     bitbake Fetch.download() so that an argument can be passed that says
>     whether to use logger.error() or logger.warning() when reporting an
>     error, but that feels really ugly.
>
>     Thanks,
>     Joshua Watt
>     -- 
>     _______________________________________________
>     bitbake-devel mailing list
>     bitbake-devel@lists.openembedded.org
>     <mailto:bitbake-devel@lists.openembedded.org>
>     http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>