Checking sstate mirror is hanging at 100%

Checking sstate mirror is hanging at 100%

[Vineeth Karumanchi]

[-- Attachment #1: Type: text/plain, Size: 638 bytes --]

Hi All,

I am running sstate mirrors on a web server. My network has proxy, 
sstate is sharing well with proxies, .

But, without proxies, bitbake is hanging at "/Checking sstate mirror 
object #######100%/".  Is there a way to change timeout of sstate mirror 
checking.

I did some debugging with "-DD" and I see bitbake is trying to fetch the 
sstate objects, failing and keep on re-trying.


/DEBUG: checkstatus: trying again/


In this case i would expect bitbake to "check for sstate mirrors" for 
some time and rebuild from scratch. Is there any approach to over come 
this ?


Thanks in Advance

Vineeth


[-- Attachment #2: Type: text/html, Size: 1033 bytes --]

Re: Checking sstate mirror is hanging at 100%

[Alexander Kanavin]

On 10/30/2017 07:54 AM, Vineeth Karumanchi wrote:

> In this case i would expect bitbake to "check for sstate mirrors" for 
> some time and rebuild from scratch. Is there any approach to over come 
> this ?

The approach is to fix your infrastructure so that fetching from network 
doesn't fail. We've gotten a similar request recently for enabling 'soft 
sstate cache failure', and RP is firmly against it:

http://lists.openembedded.org/pipermail/openembedded-core/2017-August/141593.html

Alex

Re: Checking sstate mirror is hanging at 100%

[Andre McCurdy]

On Mon, Oct 30, 2017 at 2:37 AM, Alexander Kanavin
<alexander.kanavin@linux.intel.com> wrote:
> On 10/30/2017 07:54 AM, Vineeth Karumanchi wrote:
>
>> In this case i would expect bitbake to "check for sstate mirrors" for some
>> time and rebuild from scratch. Is there any approach to over come this ?
>
> The approach is to fix your infrastructure so that fetching from network
> doesn't fail. We've gotten a similar request recently for enabling 'soft
> sstate cache failure', and RP is firmly against it:
>
> http://lists.openembedded.org/pipermail/openembedded-core/2017-August/141593.html

It looks like the logic in the wget fetcher retries once, so if it's
getting stuck then that seems like a genuine bug.

Vineeth, are you sure it's really continuously retrying? Or are you
just seeing timeouts for many fetches (which will eventually
complete)? Note that you can reduce the wget timeout by setting your
own value for "FETCHCMD_wget". See bitbake/lib/bb/fetch2/wget.py for
the default value - use that as a starting point for your
modification.

Re: Checking sstate mirror is hanging at 100%

[Vineeth Karumanchi]

> From: Andre McCurdy [mailto:armccurdy@gmail.com]
> Sent: Tuesday, October 31, 2017 1:19 AM
> To: Alexander Kanavin <alexander.kanavin@linux.intel.com>
> Cc: Vineeth Karumanchi <VINEETHC@xilinx.com>; Yocto discussion list
> <yocto@yoctoproject.org>
> Subject: Re: [yocto] [Yocto] Checking sstate mirror is hanging at 100%
> 
> On Mon, Oct 30, 2017 at 2:37 AM, Alexander Kanavin
> <alexander.kanavin@linux.intel.com> wrote:
> > On 10/30/2017 07:54 AM, Vineeth Karumanchi wrote:
> >
> >> In this case i would expect bitbake to "check for sstate mirrors" for
> >> some time and rebuild from scratch. Is there any approach to over come
> this ?
> >
> > The approach is to fix your infrastructure so that fetching from
> > network doesn't fail. We've gotten a similar request recently for
> > enabling 'soft sstate cache failure', and RP is firmly against it:
> >
> > http://lists.openembedded.org/pipermail/openembedded-core/2017-
> August/
> > 141593.html
> 
> It looks like the logic in the wget fetcher retries once, so if it's getting stuck
> then that seems like a genuine bug.
> 
> Vineeth, are you sure it's really continuously retrying? 
Yes, I see continuous retry of wget of same sstate files.
With -DD I see the below messages  for every try:

DEBUG: checkstatus: trying again
DEBUG: checkstatus: trying again
DEBUG: checkstatus: trying again
DEBUG: checkstatus: trying again
DEBUG: checkstatus: trying again

>Or are you just seeing
> timeouts for many fetches (which will eventually complete)? Note that you
> can reduce the wget timeout by setting your own value for
> "FETCHCMD_wget". See bitbake/lib/bb/fetch2/wget.py for the default value
> - use that as a starting point for your modification.

No fetch failure error, Just hanging at 100%.

I did some more analysis:
My network needs proxy to access any site.

To validate a URL, bitbake/lib/bb/fetch2/__init__.py:  urllib.parse.urlparse(uri) and checking for netloc.
This is returning true without proxy being set, but the same uri is not establishing connection without proxies.

bitbake is thinking it as a valid  and keep retrying without any errors.

Thanks
Vineeth

Re: Checking sstate mirror is hanging at 100%

[Andre McCurdy]

>>
>> It looks like the logic in the wget fetcher retries once, so if it's getting stuck
>> then that seems like a genuine bug.
>>
>> Vineeth, are you sure it's really continuously retrying?
> Yes, I see continuous retry of wget of same sstate files.

The fetcher explicitly tries to avoid that by retrying each file only
once. See retry logic in:

  http://git.openembedded.org/bitbake/commit/?id=54b1961551511948e0cbd2ac39f19b39b9cee568

However, there is a later commit which may also be related:

  http://git.openembedded.org/bitbake/commit/?id=6fa07752bbd3ac345cd8617da49a70e0b2dd565f

Does the version of bitbake which you are using contain both commits?

Re: Checking sstate mirror is hanging at 100%

[Vineeth Karumanchi]

On 11/1/2017 12:11 AM, Andre McCurdy wrote:
>>> It looks like the logic in the wget fetcher retries once, so if it's getting stuck
>>> then that seems like a genuine bug.
>>>
>>> Vineeth, are you sure it's really continuously retrying?
>> Yes, I see continuous retry of wget of same sstate files.
> The fetcher explicitly tries to avoid that by retrying each file only
> once. See retry logic in:
>
>    http://git.openembedded.org/bitbake/commit/?id=54b1961551511948e0cbd2ac39f19b39b9cee568
>
> However, there is a later commit which may also be related:
>
>    http://git.openembedded.org/bitbake/commit/?id=6fa07752bbd3ac345cd8617da49a70e0b2dd565f
>
> Does the version of bitbake which you are using contain both commits?

Thanks Andre
I am not having later patch. I will check the issue with master branch

Re: Checking sstate mirror is hanging at 100%

[Vineeth Chowdary Karumanchi]

On 11/1/2017 12:11 AM, Andre McCurdy wrote:

> The fetcher explicitly tries to avoid that by retrying each file only
> once. See retry logic in:
> 
>    http://git.openembedded.org/bitbake/commit/?id=54b1961551511948e0cbd2ac39f19b39b9cee568
> 
> However, there is a later commit which may also be related:
> 
>    http://git.openembedded.org/bitbake/commit/?id=6fa07752bbd3ac345cd8617da49a70e0b2dd565f
I didnt have this patch, presently i am on morty branch. I will check 
with master branch

Thanks for the info
Vineeth
> 
> Does the version of bitbake which you are using contain both commits?
>