Is http://downloads.yoctoproject.org/ down?

Is http://downloads.yoctoproject.org/ down?

[severin.zwicker]

[-- Attachment #1: Type: text/plain, Size: 181 bytes --]

I can't reach http://downloads.yoctoproject.org/ to download http://downloads.yoctoproject.org/releases/uninative/2.7/x86_64-nativesdk-libc.tar.xz .

Did somebody know anything?

[-- Attachment #2: Type: text/html, Size: 2267 bytes --]

Re: [yocto] Is http://downloads.yoctoproject.org/ down?

[Paul Barker]

On Tue, 12 May 2020 at 13:36, Zwicker Severin
<severin.zwicker@rittmeyer.com> wrote:
>
> I can’t reach http://downloads.yoctoproject.org/ to download http://downloads.yoctoproject.org/releases/uninative/2.7/x86_64-nativesdk-libc.tar.xz .
>
>
>
> Did somebody know anything?

Yes, it currently appears to be down. The relevant folks will be
looking at it but it's still early in the US timezones.

Re: [yocto] Is http://downloads.yoctoproject.org/ down?

[Denys Dmytriyenko]

On Tue, May 12, 2020 at 02:03:07PM +0100, Paul Barker wrote:
> On Tue, 12 May 2020 at 13:36, Zwicker Severin
> <severin.zwicker@rittmeyer.com> wrote:
> >
> > I can’t reach http://downloads.yoctoproject.org/ to download http://downloads.yoctoproject.org/releases/uninative/2.7/x86_64-nativesdk-libc.tar.xz .
> >
> >
> >
> > Did somebody know anything?
> 
> Yes, it currently appears to be down. The relevant folks will be
> looking at it but it's still early in the US timezones.

Could it be Europe-specific problem?
Seems slow, but still working here in the US:

$ wget http://downloads.yoctoproject.org/releases/uninative/2.7/x86_64-nativesdk-libc.tar.xz
--2020-05-12 09:08:15--  http://downloads.yoctoproject.org/releases/uninative/2.7/x86_64-nativesdk-libc.tar.xz
Resolving downloads.yoctoproject.org... 198.145.29.63
Connecting to downloads.yoctoproject.org|198.145.29.63|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5450224 (5.2M) [application/octet-stream]
Saving to: ‘x86_64-nativesdk-libc.tar.xz’

x86_64-nativesdk-libc.tar.xz           100%[==========================================================================>]   5.20M   540KB/s    in 59s     

2020-05-12 09:09:27 (89.5 KB/s) - ‘x86_64-nativesdk-libc.tar.xz’ saved [5450224/5450224]

Re: Is http://downloads.yoctoproject.org/ down?

[Scott Branden]

[-- Attachment #1: Type: text/plain, Size: 373 bytes --]

With downloads.yoctoproject.org down again this begs the question:

Why does uninative tarball get fetched by default from the internet every time?
Why is it in not cached like every other tarball that is downloaded by recipes?
The default behaviour for the yocto project should be to have the tarball stored in the yocto cache such that it is not fetched every time?

[-- Attachment #2: Type: text/html, Size: 389 bytes --]

Re: [yocto] Is http://downloads.yoctoproject.org/ down?

[Richard Purdie]

On Tue, 2020-08-04 at 14:36 -0700, Scott Branden via
lists.yoctoproject.org wrote:
> With downloads.yoctoproject.org down again this begs the question:
> 
> Why does uninative tarball get fetched by default from the internet
> every time?
> Why is it in not cached like every other tarball that is downloaded
> by recipes?
> The default behaviour for the yocto project should be to have the
> tarball stored in the yocto cache such that it is not fetched every
> time?

Those are good questions. 

As far as I know on my own local builds, the uninative tarball *is*
cached locally in DL_DIR. We don't see repeated downloads on the
autobuilder either so it appears to work there too.

If you're not seeing that it would definitely be something to
investigate.

Cheers,

Richard

Re: [yocto] Is http://downloads.yoctoproject.org/ down?

[Scott Branden]

Hi Richard,

On 2020-08-05 2:50 a.m., Richard Purdie wrote:
> On Tue, 2020-08-04 at 14:36 -0700, Scott Branden via
> lists.yoctoproject.org wrote:
>> With downloads.yoctoproject.org down again this begs the question:
>>
>> Why does uninative tarball get fetched by default from the internet
>> every time?
>> Why is it in not cached like every other tarball that is downloaded
>> by recipes?
>> The default behaviour for the yocto project should be to have the
>> tarball stored in the yocto cache such that it is not fetched every
>> time?
> Those are good questions. 
>
> As far as I know on my own local builds, the uninative tarball *is*
> cached locally in DL_DIR. We don't see repeated downloads on the
> autobuilder either so it appears to work there too.
Yes, the uninative tarball *is* cached but I see repeated downloads.
>
> If you're not seeing that it would definitely be something to
> investigate.
I just did a clone of poky.git and the repeated downloads do NOT happen.
So I now need to investigate what is different when adding layers and conf settings
that is causing the issue.

> Cheers,
>
> Richard
>
Thanks for confirming the uninative caching should work, but it is not in my environment and I will have to investigate.

Scott

Re: [yocto] Is http://downloads.yoctoproject.org/ down?

[Scott Branden]

We found the source of the uninative fetch problem:

SOURCE_MIRROR_URL needs to be set in order for uninative to be picked up from local mirror.

The standard PREMIRRORS mechanism doesn't work for this single tarball.
So every time a clean poky build took place it always went to the internet to fetch the uninative tarball
the first time and store it in the local build download.

I think this should be clearly documented somewhere in the yocto manual?

On 2020-08-05 10:07 a.m., Scott Branden wrote:
> Hi Richard,
>
> On 2020-08-05 2:50 a.m., Richard Purdie wrote:
>> On Tue, 2020-08-04 at 14:36 -0700, Scott Branden via
>> lists.yoctoproject.org wrote:
>>> With downloads.yoctoproject.org down again this begs the question:
>>>
>>> Why does uninative tarball get fetched by default from the internet
>>> every time?
>>> Why is it in not cached like every other tarball that is downloaded
>>> by recipes?
>>> The default behaviour for the yocto project should be to have the
>>> tarball stored in the yocto cache such that it is not fetched every
>>> time?
>> Those are good questions. 
>>
>> As far as I know on my own local builds, the uninative tarball *is*
>> cached locally in DL_DIR. We don't see repeated downloads on the
>> autobuilder either so it appears to work there too.
> Yes, the uninative tarball *is* cached but I see repeated downloads.
>> If you're not seeing that it would definitely be something to
>> investigate.
> I just did a clone of poky.git and the repeated downloads do NOT happen.
> So I now need to investigate what is different when adding layers and conf settings
> that is causing the issue.
>
>> Cheers,
>>
>> Richard
>>
> Thanks for confirming the uninative caching should work, but it is not in my environment and I will have to investigate.
>
> Scott

Re: [yocto] Is http://downloads.yoctoproject.org/ down?

[Richard Purdie]

On Thu, 2020-08-06 at 11:05 -0700, Scott Branden wrote:
> We found the source of the uninative fetch problem:
> 
> SOURCE_MIRROR_URL needs to be set in order for uninative to be picked
> up from local mirror.
> 
> The standard PREMIRRORS mechanism doesn't work for this single
> tarball.
> So every time a clean poky build took place it always went to the
> internet to fetch the uninative tarball
> the first time and store it in the local build download.
> 
> I think this should be clearly documented somewhere in the yocto
> manual?

That definitely explains things. I think its assumed that people would
set appropriate PREMIRRORS to cover uninative in this case but that
isn't documented. I'm not even sure we want for people to have to care
about this :/.

Can you try this change:

diff --git a/meta/classes/uninative.bbclass b/meta/classes/uninative.bbclass
index 70799bbf6d..99072e1326 100644
--- a/meta/classes/uninative.bbclass
+++ b/meta/classes/uninative.bbclass
@@ -56,9 +56,14 @@ python uninative_event_fetchloader() {
             # Our games with path manipulation of DL_DIR mean standard PREMIRRORS don't work
             # and we can't easily put 'chksum' into the url path from a url parameter with
             # the current fetcher url handling
-            ownmirror = d.getVar('SOURCE_MIRROR_URL')
-            if ownmirror:
-                localdata.appendVar("PREMIRRORS", " ${UNINATIVE_URL}${UNINATIVE_TARBALL} ${SOURCE_MIRROR_URL}/uninative/%s/${UNINATIVE_TARBALL}" % chksum)
+            premirrors = bb.fetch2.mirror_from_string(localdata.getVar("PREMIRRORS"))
+            for line in premirrors:
+                try:
+                    (find, replace) = line
+                except ValueError:
+                    continue
+                if find.startswith("http"):
+                    localdata.appendVar("PREMIRRORS", " ${UNINATIVE_URL}${UNINATIVE_TARBALL} %s/uninative/%s/${UNINATIVE_TARBALL}" % (replace, chksum))
 
             srcuri = d.expand("${UNINATIVE_URL}${UNINATIVE_TARBALL};sha256sum=%s" % chksum)
             bb.note("Fetching uninative binary shim from %s" % srcuri)

Hopefully that handles this case more generically.

Cheers,
Richard
 

Re: [yocto] Is http://downloads.yoctoproject.org/ down?

[Scott Branden]

Hi Richard,

On 2020-08-06 1:35 p.m., Richard Purdie wrote:
> On Thu, 2020-08-06 at 11:05 -0700, Scott Branden wrote:
>> We found the source of the uninative fetch problem:
>>
>> SOURCE_MIRROR_URL needs to be set in order for uninative to be picked
>> up from local mirror.
>>
>> The standard PREMIRRORS mechanism doesn't work for this single
>> tarball.
>> So every time a clean poky build took place it always went to the
>> internet to fetch the uninative tarball
>> the first time and store it in the local build download.
>>
>> I think this should be clearly documented somewhere in the yocto
>> manual?
> That definitely explains things. I think its assumed that people would
> set appropriate PREMIRRORS to cover uninative in this case but that
> isn't documented. I'm not even sure we want for people to have to care
> about this :/.
>
> Can you try this change:
>
> diff --git a/meta/classes/uninative.bbclass b/meta/classes/uninative.bbclass
> index 70799bbf6d..99072e1326 100644
> --- a/meta/classes/uninative.bbclass
> +++ b/meta/classes/uninative.bbclass
> @@ -56,9 +56,14 @@ python uninative_event_fetchloader() {
>              # Our games with path manipulation of DL_DIR mean standard PREMIRRORS don't work
>              # and we can't easily put 'chksum' into the url path from a url parameter with
>              # the current fetcher url handling
> -            ownmirror = d.getVar('SOURCE_MIRROR_URL')
> -            if ownmirror:
> -                localdata.appendVar("PREMIRRORS", " ${UNINATIVE_URL}${UNINATIVE_TARBALL} ${SOURCE_MIRROR_URL}/uninative/%s/${UNINATIVE_TARBALL}" % chksum)
> +            premirrors = bb.fetch2.mirror_from_string(localdata.getVar("PREMIRRORS"))
> +            for line in premirrors:
> +                try:
> +                    (find, replace) = line
> +                except ValueError:
> +                    continue
> +                if find.startswith("http"):
> +                    localdata.appendVar("PREMIRRORS", " ${UNINATIVE_URL}${UNINATIVE_TARBALL} %s/uninative/%s/${UNINATIVE_TARBALL}" % (replace, chksum))
>  
>              srcuri = d.expand("${UNINATIVE_URL}${UNINATIVE_TARBALL};sha256sum=%s" % chksum)
>              bb.note("Fetching uninative binary shim from %s" % srcuri)
Note is missleading as it is going to fetch from PREMIRRORS.
>
> Hopefully that handles this case more generically.
Yes, this works.  uninative is now fetched from the PREMIRRORS like all the other downloads even though it prints the note about Fetching uninative binary shim from the URL.
>
> Cheers,
> Richard
>  
>
>
>

Re: [yocto] Is http://downloads.yoctoproject.org/ down?

[Richard Purdie]

On Thu, 2020-08-06 at 15:31 -0700, Scott Branden wrote:
> > diff --git a/meta/classes/uninative.bbclass
> > b/meta/classes/uninative.bbclass
> > index 70799bbf6d..99072e1326 100644
> > --- a/meta/classes/uninative.bbclass
> > +++ b/meta/classes/uninative.bbclass
> > @@ -56,9 +56,14 @@ python uninative_event_fetchloader() {
> >              # Our games with path manipulation of DL_DIR mean
> > standard PREMIRRORS don't work
> >              # and we can't easily put 'chksum' into the url path
> > from a url parameter with
> >              # the current fetcher url handling
> > -            ownmirror = d.getVar('SOURCE_MIRROR_URL')
> > -            if ownmirror:
> > -                localdata.appendVar("PREMIRRORS", "
> > ${UNINATIVE_URL}${UNINATIVE_TARBALL}
> > ${SOURCE_MIRROR_URL}/uninative/%s/${UNINATIVE_TARBALL}" % chksum)
> > +            premirrors =
> > bb.fetch2.mirror_from_string(localdata.getVar("PREMIRRORS"))
> > +            for line in premirrors:
> > +                try:
> > +                    (find, replace) = line
> > +                except ValueError:
> > +                    continue
> > +                if find.startswith("http"):
> > +                    localdata.appendVar("PREMIRRORS", "
> > ${UNINATIVE_URL}${UNINATIVE_TARBALL}
> > %s/uninative/%s/${UNINATIVE_TARBALL}" % (replace, chksum))
> >  
> >              srcuri =
> > d.expand("${UNINATIVE_URL}${UNINATIVE_TARBALL};sha256sum=%s" %
> > chksum)
> >              bb.note("Fetching uninative binary shim from %s" %
> > srcuri)
>
> Note is missleading as it is going to fetch from PREMIRRORS.
>
> > Hopefully that handles this case more generically.
> Yes, this works.  uninative is now fetched from the PREMIRRORS like
> all the other downloads even though it prints the note about Fetching
> uninative binary shim from the URL.

Thanks, I've tweaked the patch to improve the note and submitted it.
Good to get to the bottom of this!

Cheers,

Richard