Re: [PATCH v6 00/25] xl / libxl: named PCI pass-through devices

Re: [PATCH v6 00/25] xl / libxl: named PCI pass-through devices

[Stefano Stabellini]

The pipeline failed because the "fedora-gcc-debug" build failed with a
timeout: 

ERROR: Job failed: execution took longer than 1h0m0s seconds

given that all the other jobs passed (including the other Fedora job), I
take this failed because the gitlab-ci x86 runners were overloaded?


On Tue, 8 Dec 2020, no-reply@patchew.org wrote:
> Hi,
> 
> Patchew automatically ran gitlab-ci pipeline with this patch (series) applied, but the job failed. Maybe there's a bug in the patches?
> 
> You can find the link to the pipeline in the beginning of the report below:
> 
> Type: series
> Message-id: 20201208193033.11306-1-paul@xen.org
> Subject: [PATCH v6 00/25] xl / libxl: named PCI pass-through devices
> 
> === TEST SCRIPT BEGIN ===
> #!/bin/bash
> sleep 10
> patchew gitlab-pipeline-check -p xen-project/patchew/xen
> === TEST SCRIPT END ===
> 
> warning: redirecting to https://gitlab.com/xen-project/patchew/xen.git/
> From https://gitlab.com/xen-project/patchew/xen
>    5e666356a9..4b0e0db861  master     -> master
> warning: redirecting to https://gitlab.com/xen-project/patchew/xen.git/
> From https://gitlab.com/xen-project/patchew/xen
>  * [new tag]               patchew/20201208193033.11306-1-paul@xen.org -> patchew/20201208193033.11306-1-paul@xen.org
> Switched to a new branch 'test'
> 6c78dcb6d3 libxl / libxlu: support 'xl pci-attach/detach' by name
> 117f736c8b docs/man: modify xl-pci-configuration(5) to add 'name' field to PCI_SPEC_STRING
> 38e63698d6 xl: support naming of assignable devices
> 32b064a4a2 libxl: introduce libxl_pci_bdf_assignable_add/remove/list/list_free(), ...
> 830b6fa734 libxl: convert internal functions in libxl_pci.c...
> d5d5d08e3b docs/man: modify xl(1) in preparation for naming of assignable devices
> bb4cbf5856 libxlu: introduce xlu_pci_parse_spec_string()
> 62f09b89d2 libxl: introduce 'libxl_pci_bdf' in the idl...
> eb3c3ecef6 docs/man: fix xl(1) documentation for 'pci' operations
> cab74a871d docs/man: improve documentation of PCI_SPEC_STRING...
> da45af2de8 docs/man: extract documentation of PCI_SPEC_STRING from the xl.cfg manpage...
> 797b0fd3d4 libxl: use COMPARE_PCI() macro is_pci_in_array()...
> 2c0d9b579f libxl: add libxl_device_pci_assignable_list_free()...
> 1d4d73044e libxl: make sure callers of libxl_device_pci_list() free the list after use
> 24150e4156 libxl: remove get_all_assigned_devices() from libxl_pci.c
> a3d908d5a2 libxl: remove unnecessary check from libxl__device_pci_add()
> ada8e55b23 libxl: generalise 'driver_path' xenstore access functions in libxl_pci.c
> a38482aa96 libxl: stop using aodev->device_config in libxl__device_pci_add()...
> d115527623 libxl: remove extraneous arguments to do_pci_remove() in libxl_pci.c
> b1369310e6 libxl: s/detatched/detached in libxl_pci.c
> 4ccef90ca8 libxl: add/recover 'rdm_policy' to/from PCI backend in xenstore
> 09d3adddb4 libxl: Make sure devices added by pci-attach are reflected in the config
> e2feb1c29b libxl: make libxl__device_list() work correctly for LIBXL__DEVICE_KIND_PCI...
> 8599a6a85e xl: s/pcidev/pci where possible
> 4648bbbb01 libxl: s/pcidev/pci and remove DEFINE_DEVICE_TYPE_STRUCT_X
> 
> === OUTPUT BEGIN ===
> [2020-12-08 20:09:14] Looking up pipeline...
> [2020-12-08 20:09:14] Found pipeline 226993561:
> 
> https://gitlab.com/xen-project/patchew/xen/-/pipelines/226993561
> 
> [2020-12-08 20:09:14] Waiting for pipeline to finish...
> [2020-12-08 20:24:18] Still waiting...
> [2020-12-08 20:39:23] Still waiting...
> [2020-12-08 20:54:28] Still waiting...
> [2020-12-08 21:09:32] Still waiting...
> [2020-12-08 21:24:36] Still waiting...
> [2020-12-08 21:39:41] Still waiting...
> [2020-12-08 21:54:45] Still waiting...
> [2020-12-08 21:54:46] Pipeline failed
> [2020-12-08 21:54:46] Job 'qemu-smoke-x86-64-clang-pvh' in stage 'test' is skipped
> [2020-12-08 21:54:46] Job 'qemu-smoke-x86-64-gcc-pvh' in stage 'test' is skipped
> [2020-12-08 21:54:46] Job 'qemu-smoke-x86-64-clang' in stage 'test' is skipped
> [2020-12-08 21:54:46] Job 'qemu-smoke-x86-64-gcc' in stage 'test' is skipped
> [2020-12-08 21:54:46] Job 'build-each-commit-gcc' in stage 'test' is skipped
> === OUTPUT END ===
> 
> Test command exited with code: 1

Re: [PATCH v6 00/25] xl / libxl: named PCI pass-through devices

[Wei Liu]

On Tue, Dec 08, 2020 at 05:02:50PM -0800, Stefano Stabellini wrote:
> The pipeline failed because the "fedora-gcc-debug" build failed with a
> timeout: 
> 
> ERROR: Job failed: execution took longer than 1h0m0s seconds
> 
> given that all the other jobs passed (including the other Fedora job), I
> take this failed because the gitlab-ci x86 runners were overloaded?
> 

The CI system is configured to auto-scale as the number of jobs grows.
The limit is set to 10 (VMs) at the moment.

https://gitlab.com/xen-project/xen-gitlab-ci/-/commit/832bfd72ea3a227283bf3df88b418a9aae95a5a4

I haven't looked at the log, but the number of build jobs looks rather
larger than when we get started. Maybe the limit of 10 is not good
enough?

Wei.

Re: [PATCH v6 00/25] xl / libxl: named PCI pass-through devices

[Stefano Stabellini]

On Wed, 9 Dec 2020, Wei Liu wrote:
> On Tue, Dec 08, 2020 at 05:02:50PM -0800, Stefano Stabellini wrote:
> > The pipeline failed because the "fedora-gcc-debug" build failed with a
> > timeout: 
> > 
> > ERROR: Job failed: execution took longer than 1h0m0s seconds
> > 
> > given that all the other jobs passed (including the other Fedora job), I
> > take this failed because the gitlab-ci x86 runners were overloaded?
> > 
> 
> The CI system is configured to auto-scale as the number of jobs grows.
> The limit is set to 10 (VMs) at the moment.
> 
> https://gitlab.com/xen-project/xen-gitlab-ci/-/commit/832bfd72ea3a227283bf3df88b418a9aae95a5a4
> 
> I haven't looked at the log, but the number of build jobs looks rather
> larger than when we get started. Maybe the limit of 10 is not good
> enough?

Interesting! That's only for the x86 runners, not the ARM runners (we
only have 1 ARM64 runner), is that right?

If we could increase the number of VMs for x86 I think that would be
helpful because we have very many x86 jobs.

Re: [PATCH v6 00/25] xl / libxl: named PCI pass-through devices

[Stefano Stabellini]

On Wed, 9 Dec 2020, Stefano Stabellini wrote:
> On Wed, 9 Dec 2020, Wei Liu wrote:
> > On Tue, Dec 08, 2020 at 05:02:50PM -0800, Stefano Stabellini wrote:
> > > The pipeline failed because the "fedora-gcc-debug" build failed with a
> > > timeout: 
> > > 
> > > ERROR: Job failed: execution took longer than 1h0m0s seconds
> > > 
> > > given that all the other jobs passed (including the other Fedora job), I
> > > take this failed because the gitlab-ci x86 runners were overloaded?
> > > 
> > 
> > The CI system is configured to auto-scale as the number of jobs grows.
> > The limit is set to 10 (VMs) at the moment.
> > 
> > https://gitlab.com/xen-project/xen-gitlab-ci/-/commit/832bfd72ea3a227283bf3df88b418a9aae95a5a4
> > 
> > I haven't looked at the log, but the number of build jobs looks rather
> > larger than when we get started. Maybe the limit of 10 is not good
> > enough?
> 
> Interesting! That's only for the x86 runners, not the ARM runners (we
> only have 1 ARM64 runner), is that right?
> 
> If we could increase the number of VMs for x86 I think that would be
> helpful because we have very many x86 jobs.

I don't know what is going on but at the moment there seems to be only
one x86 build active
(https://gitlab.com/xen-project/patchew/xen/-/pipelines/227280736).
Should there be at least 3 of them?

Re: [PATCH v6 00/25] xl / libxl: named PCI pass-through devices

[Wei Liu]

On Wed, Dec 09, 2020 at 06:41:03PM -0800, Stefano Stabellini wrote:
> On Wed, 9 Dec 2020, Stefano Stabellini wrote:
> > On Wed, 9 Dec 2020, Wei Liu wrote:
> > > On Tue, Dec 08, 2020 at 05:02:50PM -0800, Stefano Stabellini wrote:
> > > > The pipeline failed because the "fedora-gcc-debug" build failed with a
> > > > timeout: 
> > > > 
> > > > ERROR: Job failed: execution took longer than 1h0m0s seconds
> > > > 
> > > > given that all the other jobs passed (including the other Fedora job), I
> > > > take this failed because the gitlab-ci x86 runners were overloaded?
> > > > 
> > > 
> > > The CI system is configured to auto-scale as the number of jobs grows.
> > > The limit is set to 10 (VMs) at the moment.
> > > 
> > > https://gitlab.com/xen-project/xen-gitlab-ci/-/commit/832bfd72ea3a227283bf3df88b418a9aae95a5a4
> > > 
> > > I haven't looked at the log, but the number of build jobs looks rather
> > > larger than when we get started. Maybe the limit of 10 is not good
> > > enough?
> > 
> > Interesting! That's only for the x86 runners, not the ARM runners (we
> > only have 1 ARM64 runner), is that right?
> > 
> > If we could increase the number of VMs for x86 I think that would be
> > helpful because we have very many x86 jobs.
> 
> I don't know what is going on but at the moment there seems to be only
> one x86 build active
> (https://gitlab.com/xen-project/patchew/xen/-/pipelines/227280736).
> Should there be at least 3 of them?

Not sure what you meant here. That pipeline is green.

It may take some time for the CI to scale up if it is "cold". By default
there is only 1 standby runner to reduce cost.

Wei.

gitlab-docker-machine-oyster failure, Was: [PATCH v6 00/25] xl / libxl: named PCI pass-through devices

[Stefano Stabellini]

Hi Doug,

After chatting with Wei on IRC, it became obvious that the issue is that
gitlab-docker-machine-oyster failed, see its grey status here under
"Runners":

https://gitlab.com/xen-project/patchew/xen/-/settings/ci_cd

Maybe it is just a matter of rebooting the VM? Doug, could you give it a
try?

Thank you!

Cheers,

Stefano



On Thu, 10 Dec 2020, Wei Liu wrote:
> On Wed, Dec 09, 2020 at 06:41:03PM -0800, Stefano Stabellini wrote:
> > On Wed, 9 Dec 2020, Stefano Stabellini wrote:
> > > On Wed, 9 Dec 2020, Wei Liu wrote:
> > > > On Tue, Dec 08, 2020 at 05:02:50PM -0800, Stefano Stabellini wrote:
> > > > > The pipeline failed because the "fedora-gcc-debug" build failed with a
> > > > > timeout: 
> > > > > 
> > > > > ERROR: Job failed: execution took longer than 1h0m0s seconds
> > > > > 
> > > > > given that all the other jobs passed (including the other Fedora job), I
> > > > > take this failed because the gitlab-ci x86 runners were overloaded?
> > > > > 
> > > > 
> > > > The CI system is configured to auto-scale as the number of jobs grows.
> > > > The limit is set to 10 (VMs) at the moment.
> > > > 
> > > > https://gitlab.com/xen-project/xen-gitlab-ci/-/commit/832bfd72ea3a227283bf3df88b418a9aae95a5a4
> > > > 
> > > > I haven't looked at the log, but the number of build jobs looks rather
> > > > larger than when we get started. Maybe the limit of 10 is not good
> > > > enough?
> > > 
> > > Interesting! That's only for the x86 runners, not the ARM runners (we
> > > only have 1 ARM64 runner), is that right?
> > > 
> > > If we could increase the number of VMs for x86 I think that would be
> > > helpful because we have very many x86 jobs.
> > 
> > I don't know what is going on but at the moment there seems to be only
> > one x86 build active
> > (https://gitlab.com/xen-project/patchew/xen/-/pipelines/227280736).
> > Should there be at least 3 of them?
> 
> Not sure what you meant here. That pipeline is green.
> 
> It may take some time for the CI to scale up if it is "cold". By default
> there is only 1 standby runner to reduce cost.