gitlab-ci: amd64-opensuse-leap-container job failing

gitlab-ci: amd64-opensuse-leap-container job failing

[Philippe Mathieu-Daudé]

Hi,

FYI the OpenSUSE job is failing since few days, i.e.:
https://gitlab.com/qemu-project/qemu/-/jobs/1622345026

  Retrieving repository 'Main Repository' metadata
[..........................................................error]
  Repository 'Main Repository' is invalid.

[repo-oss|http://download.opensuse.org/distribution/leap/15.2/repo/oss/]
Valid metadata not found at specified URL
  History:
   - Download (curl) error for
'http://download.opensuse.org/distribution/leap/15.2/repo/oss/repodata/repomd.xml':
  Error code: Curl error 56
  Error message: Recv failure: Connection reset by peer
   - Can't provide /repodata/repomd.xml
  Please check if the URIs defined for this repository are pointing to a
valid repository.
  Warning: Skipping repository 'Main Repository' because of the above error.

I tried to run 'zypper ref' with:

-- >8 --
--- a/tests/docker/dockerfiles/opensuse-leap.docker
+++ b/tests/docker/dockerfiles/opensuse-leap.docker
@@ -109,5 +109,7 @@ ENV PACKAGES \
     zlib-devel
 ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3.6

-RUN zypper update -y && zypper --non-interactive install -y $PACKAGES
+RUN zypper refresh && \
+    zypper update -y && \
+    zypper --non-interactive install -y $PACKAGES
 RUN rpm -q $PACKAGES | sort > /packages.txt
---

but no luck: https://gitlab.com/philmd/qemu/-/jobs/1623554962

Should we temporarily disable to job and its dependencies?

Regards,

Phil.

Re: gitlab-ci: amd64-opensuse-leap-container job failing

[Daniel P. Berrangé]

On Sun, Sep 26, 2021 at 07:23:56PM +0200, Philippe Mathieu-Daudé wrote:
> Hi,
> 
> FYI the OpenSUSE job is failing since few days, i.e.:
> https://gitlab.com/qemu-project/qemu/-/jobs/1622345026
> 
>   Retrieving repository 'Main Repository' metadata
> [..........................................................error]
>   Repository 'Main Repository' is invalid.
> 
> [repo-oss|http://download.opensuse.org/distribution/leap/15.2/repo/oss/]
> Valid metadata not found at specified URL
>   History:
>    - Download (curl) error for
> 'http://download.opensuse.org/distribution/leap/15.2/repo/oss/repodata/repomd.xml':
>   Error code: Curl error 56
>   Error message: Recv failure: Connection reset by peer
>    - Can't provide /repodata/repomd.xml
>   Please check if the URIs defined for this repository are pointing to a
> valid repository.
>   Warning: Skipping repository 'Main Repository' because of the above error.
> 
> I tried to run 'zypper ref' with:

It isn't confined to only SuSE. In libvirt we've had similar problems
with several other jobs, though are suse jobs are the worst affected.

GitLab have finally acknowledged it is an general infra issue affecting
many things:

   https://status.gitlab.com/
   https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5590

> -- >8 --
> --- a/tests/docker/dockerfiles/opensuse-leap.docker
> +++ b/tests/docker/dockerfiles/opensuse-leap.docker
> @@ -109,5 +109,7 @@ ENV PACKAGES \
>      zlib-devel
>  ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3.6
> 
> -RUN zypper update -y && zypper --non-interactive install -y $PACKAGES
> +RUN zypper refresh && \
> +    zypper update -y && \
> +    zypper --non-interactive install -y $PACKAGES
>  RUN rpm -q $PACKAGES | sort > /packages.txt
> ---
> 
> but no luck: https://gitlab.com/philmd/qemu/-/jobs/1623554962
> 
> Should we temporarily disable to job and its dependencies?

Given it is believed to be a gitlab infra issue, rather than a problem
of ours, or something we're using, I think best to wait a little longer
to see if they get fix the infra.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: gitlab-ci: amd64-opensuse-leap-container job failing

[Al Cho]

On Mon, 2021-09-27 at 09:35 +0100, Daniel P. Berrangé wrote:
> On Sun, Sep 26, 2021 at 07:23:56PM +0200, Philippe Mathieu-Daudé wrote:
> > Hi,
> > 
> > FYI the OpenSUSE job is failing since few days, i.e.:
> > https://gitlab.com/qemu-project/qemu/-/jobs/1622345026
> > 
> >   Retrieving repository 'Main Repository' metadata
> > [..........................................................error]
> >   Repository 'Main Repository' is invalid.
> > 
> > [repo-
> > oss|http://download.opensuse.org/distribution/leap/15.2/repo/oss/]
> > Valid metadata not found at specified URL
> >   History:
> >    - Download (curl) error for
> > '
> > http://download.opensuse.org/distribution/leap/15.2/repo/oss/repodata/repomd.xml'
> > :
> >   Error code: Curl error 56
> >   Error message: Recv failure: Connection reset by peer
> >    - Can't provide /repodata/repomd.xml
> >   Please check if the URIs defined for this repository are pointing
> > to a
> > valid repository.
> >   Warning: Skipping repository 'Main Repository' because of the above
> > error.
> > 
> > I tried to run 'zypper ref' with:
> 
> It isn't confined to only SuSE. In libvirt we've had similar problems
> with several other jobs, though are suse jobs are the worst affected.
> 
> GitLab have finally acknowledged it is an general infra issue affecting
> many things:
> 
>    https://status.gitlab.com/
>    https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5590
> 
> > -- >8 --
> > --- a/tests/docker/dockerfiles/opensuse-leap.docker
> > +++ b/tests/docker/dockerfiles/opensuse-leap.docker
> > @@ -109,5 +109,7 @@ ENV PACKAGES \
> >      zlib-devel
> >  ENV QEMU_CONFIGURE_OPTS --python=/usr/bin/python3.6
> > 
> > -RUN zypper update -y && zypper --non-interactive install -y
> > $PACKAGES
> > +RUN zypper refresh && \
> > +    zypper update -y && \
> > +    zypper --non-interactive install -y $PACKAGES
> >  RUN rpm -q $PACKAGES | sort > /packages.txt
> > ---
> > 
> > but no luck: https://gitlab.com/philmd/qemu/-/jobs/1623554962
> > 
> > Should we temporarily disable to job and its dependencies?
> 
> Given it is believed to be a gitlab infra issue, rather than a problem
> of ours, or something we're using, I think best to wait a little longer
> to see if they get fix the infra.
> 

agree, and I am also checking the status of it.
for now the http://download.opensuse.org/distribution/leap/15.2 and the
repo works.
Will follow up it.

Cheers,
      AL

Re: gitlab-ci: amd64-opensuse-leap-container job failing

[Philippe Mathieu-Daudé]

On 9/27/21 10:35, Daniel P. Berrangé wrote:
> On Sun, Sep 26, 2021 at 07:23:56PM +0200, Philippe Mathieu-Daudé wrote:
>> Hi,
>>
>> FYI the OpenSUSE job is failing since few days, i.e.:
>> https://gitlab.com/qemu-project/qemu/-/jobs/1622345026

> It isn't confined to only SuSE. In libvirt we've had similar problems
> with several other jobs, though are suse jobs are the worst affected.
> 
> GitLab have finally acknowledged it is an general infra issue affecting
> many things:
> 
>    https://status.gitlab.com/
>    https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5590

>> Should we temporarily disable to job and its dependencies?
> 
> Given it is believed to be a gitlab infra issue, rather than a problem
> of ours, or something we're using, I think best to wait a little longer
> to see if they get fix the infra.

OK (I checked the status page during Saturday and Sunday morning and it
was all green).

Thanks for investigating,

Phil.

Re: gitlab-ci: amd64-opensuse-leap-container job failing

[Daniel P. Berrangé]

On Mon, Sep 27, 2021 at 09:35:22AM +0100, Daniel P. Berrangé wrote:
> On Sun, Sep 26, 2021 at 07:23:56PM +0200, Philippe Mathieu-Daudé wrote:
> > Hi,
> > 
> > FYI the OpenSUSE job is failing since few days, i.e.:
> > https://gitlab.com/qemu-project/qemu/-/jobs/1622345026
> > 
> >   Retrieving repository 'Main Repository' metadata
> > [..........................................................error]
> >   Repository 'Main Repository' is invalid.
> > 
> > [repo-oss|http://download.opensuse.org/distribution/leap/15.2/repo/oss/]
> > Valid metadata not found at specified URL
> >   History:
> >    - Download (curl) error for
> > 'http://download.opensuse.org/distribution/leap/15.2/repo/oss/repodata/repomd.xml':
> >   Error code: Curl error 56
> >   Error message: Recv failure: Connection reset by peer
> >    - Can't provide /repodata/repomd.xml
> >   Please check if the URIs defined for this repository are pointing to a
> > valid repository.
> >   Warning: Skipping repository 'Main Repository' because of the above error.
> > 
> > I tried to run 'zypper ref' with:
> 
> It isn't confined to only SuSE. In libvirt we've had similar problems
> with several other jobs, though are suse jobs are the worst affected.
> 
> GitLab have finally acknowledged it is an general infra issue affecting
> many things:
> 
>    https://status.gitlab.com/
>    https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5590

Setting GitLab CI env var works around the problem temporarily:

 FF_NETWORK_PER_BUILD=true

You can set it for all repos under a group eg

  https://gitlab.com/groups/qemu-project/-/settings/ci_cd

or per repo eg

  https://gitlab.com/berrange/libvirt/-/settings/ci_cd


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: gitlab-ci: amd64-opensuse-leap-container job failing

[Philippe Mathieu-Daudé]

On 9/27/21 15:47, Daniel P. Berrangé wrote:
> On Mon, Sep 27, 2021 at 09:35:22AM +0100, Daniel P. Berrangé wrote:
>> On Sun, Sep 26, 2021 at 07:23:56PM +0200, Philippe Mathieu-Daudé wrote:
>>> Hi,
>>>
>>> FYI the OpenSUSE job is failing since few days, i.e.:
>>> https://gitlab.com/qemu-project/qemu/-/jobs/1622345026
>>>
>>>   Retrieving repository 'Main Repository' metadata
>>> [..........................................................error]
>>>   Repository 'Main Repository' is invalid.
>>>
>>> [repo-oss|http://download.opensuse.org/distribution/leap/15.2/repo/oss/]
>>> Valid metadata not found at specified URL
>>>   History:
>>>    - Download (curl) error for
>>> 'http://download.opensuse.org/distribution/leap/15.2/repo/oss/repodata/repomd.xml':
>>>   Error code: Curl error 56
>>>   Error message: Recv failure: Connection reset by peer
>>>    - Can't provide /repodata/repomd.xml
>>>   Please check if the URIs defined for this repository are pointing to a
>>> valid repository.
>>>   Warning: Skipping repository 'Main Repository' because of the above error.
>>>
>>> I tried to run 'zypper ref' with:
>>
>> It isn't confined to only SuSE. In libvirt we've had similar problems
>> with several other jobs, though are suse jobs are the worst affected.
>>
>> GitLab have finally acknowledged it is an general infra issue affecting
>> many things:
>>
>>    https://status.gitlab.com/
>>    https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5590
> 
> Setting GitLab CI env var works around the problem temporarily:
> 
>  FF_NETWORK_PER_BUILD=true

Do you know if we need to recreate the pipeline?
(It didn't work on already failing one, I'm going to test
a freshly created one now).

> You can set it for all repos under a group eg
> 
>   https://gitlab.com/groups/qemu-project/-/settings/ci_cd
> 
> or per repo eg
> 
>   https://gitlab.com/berrange/libvirt/-/settings/ci_cd
> 
> 
> Regards,
> Daniel
> 

Re: gitlab-ci: amd64-opensuse-leap-container job failing

[Daniel P. Berrangé]

On Mon, Sep 27, 2021 at 04:35:04PM +0200, Philippe Mathieu-Daudé wrote:
> On 9/27/21 15:47, Daniel P. Berrangé wrote:
> > On Mon, Sep 27, 2021 at 09:35:22AM +0100, Daniel P. Berrangé wrote:
> >> On Sun, Sep 26, 2021 at 07:23:56PM +0200, Philippe Mathieu-Daudé wrote:
> >>> Hi,
> >>>
> >>> FYI the OpenSUSE job is failing since few days, i.e.:
> >>> https://gitlab.com/qemu-project/qemu/-/jobs/1622345026
> >>>
> >>>   Retrieving repository 'Main Repository' metadata
> >>> [..........................................................error]
> >>>   Repository 'Main Repository' is invalid.
> >>>
> >>> [repo-oss|http://download.opensuse.org/distribution/leap/15.2/repo/oss/]
> >>> Valid metadata not found at specified URL
> >>>   History:
> >>>    - Download (curl) error for
> >>> 'http://download.opensuse.org/distribution/leap/15.2/repo/oss/repodata/repomd.xml':
> >>>   Error code: Curl error 56
> >>>   Error message: Recv failure: Connection reset by peer
> >>>    - Can't provide /repodata/repomd.xml
> >>>   Please check if the URIs defined for this repository are pointing to a
> >>> valid repository.
> >>>   Warning: Skipping repository 'Main Repository' because of the above error.
> >>>
> >>> I tried to run 'zypper ref' with:
> >>
> >> It isn't confined to only SuSE. In libvirt we've had similar problems
> >> with several other jobs, though are suse jobs are the worst affected.
> >>
> >> GitLab have finally acknowledged it is an general infra issue affecting
> >> many things:
> >>
> >>    https://status.gitlab.com/
> >>    https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5590
> > 
> > Setting GitLab CI env var works around the problem temporarily:
> > 
> >  FF_NETWORK_PER_BUILD=true
> 
> Do you know if we need to recreate the pipeline?
> (It didn't work on already failing one, I'm going to test
> a freshly created one now).

You shoudln't need to re-create the pipeline, just retry the job.

If setting the variable in the web UI, make sure "Protect variable"
is *not* set. Only the "master" branch is protected by default
so other branches won't get protected variables set in their jobs.


> 
> > You can set it for all repos under a group eg
> > 
> >   https://gitlab.com/groups/qemu-project/-/settings/ci_cd
> > 
> > or per repo eg
> > 
> >   https://gitlab.com/berrange/libvirt/-/settings/ci_cd
> > 
> > 
> > Regards,
> > Daniel
> > 
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: gitlab-ci: amd64-opensuse-leap-container job failing

[Daniel P. Berrangé]

On Mon, Sep 27, 2021 at 09:35:22AM +0100, Daniel P. Berrangé wrote:
> On Sun, Sep 26, 2021 at 07:23:56PM +0200, Philippe Mathieu-Daudé wrote:
> > Hi,
> > 
> > FYI the OpenSUSE job is failing since few days, i.e.:
> > https://gitlab.com/qemu-project/qemu/-/jobs/1622345026
> > 
> >   Retrieving repository 'Main Repository' metadata
> > [..........................................................error]
> >   Repository 'Main Repository' is invalid.
> > 
> > [repo-oss|http://download.opensuse.org/distribution/leap/15.2/repo/oss/]
> > Valid metadata not found at specified URL
> >   History:
> >    - Download (curl) error for
> > 'http://download.opensuse.org/distribution/leap/15.2/repo/oss/repodata/repomd.xml':
> >   Error code: Curl error 56
> >   Error message: Recv failure: Connection reset by peer
> >    - Can't provide /repodata/repomd.xml
> >   Please check if the URIs defined for this repository are pointing to a
> > valid repository.
> >   Warning: Skipping repository 'Main Repository' because of the above error.
> > 
> > I tried to run 'zypper ref' with:
> 
> It isn't confined to only SuSE. In libvirt we've had similar problems
> with several other jobs, though are suse jobs are the worst affected.
> 
> GitLab have finally acknowledged it is an general infra issue affecting
> many things:
> 
>    https://status.gitlab.com/
>    https://gitlab.com/gitlab-com/gl-infra/production/-/issues/5590

As it turned out, it wasn't gitlab's fault at all. Google cloud had
some networking problems, and these were resolved a short while ago.

So gitlab CI jobs sould be back to normal, and the FF_NETWORK_PER_BUILD
env variable workaround should no longer be needed either.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|