* [PATCH 0/2] gitlab: avoid timeouts in windows cross builds
@ 2021-08-10 14:06 Daniel P. Berrangé
2021-08-10 14:06 ` [PATCH 1/2] gitlab: exclude sparc-softmmu and riscv32-softmmu from " Daniel P. Berrangé
` (2 more replies)
0 siblings, 3 replies; 13+ messages in thread
From: Daniel P. Berrangé @ 2021-08-10 14:06 UTC (permalink / raw)
To: qemu-devel
Cc: Thomas Huth, Daniel P. Berrangé, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Willian Rampazzo, Alex Bennée
The win64 cross build is quite often hitting the job timeout, despite
having an elevated timeout of 80 minutes. A typical time is 75-78
minutes in my tests, which leaves little headroom.
I'm not inclined to increase the timeout beyond 80 minutes as this is
already unreasonably long.
Interestingly the win32 job doesn't timeout in the same way. In local
tests I've found the mingw32 gcc is as much as 50% faster than
mingw64 gcc in building QEMU. This explains at least why we only see
win64 gitlab builds timeout normally.
None the less, this series thus cuts down on the amount of stuff built
in both win32 and win64 jobs, as well as cross build jobs more
generally.
With this applied:
- win32 job runs in 40 minutes instead of 55 minutes
- win64 job runs in 50 minutes instead of 75 minutes
- other cross build jobs are a few minutes faster
this should give us plenty of headroom to avoid hitting the job
timeouts.
Daniel P. Berrangé (2):
gitlab: exclude sparc-softmmu and riscv32-softmmu from cross builds
gitlab: skip many more targets in windows cross builds
.gitlab-ci.d/crossbuild-template.yml | 3 ++-
.gitlab-ci.d/crossbuilds.yml | 2 ++
2 files changed, 4 insertions(+), 1 deletion(-)
--
2.31.1
^ permalink raw reply [flat|nested] 13+ messages in thread* [PATCH 1/2] gitlab: exclude sparc-softmmu and riscv32-softmmu from cross builds
2021-08-10 14:06 [PATCH 0/2] gitlab: avoid timeouts in windows cross builds Daniel P. Berrangé
@ 2021-08-10 14:06 ` Daniel P. Berrangé
2021-08-10 14:56 ` Willian Rampazzo
` (2 more replies)
2021-08-10 14:06 ` [PATCH 2/2] gitlab: skip many more targets in windows " Daniel P. Berrangé
2021-08-11 5:53 ` [PATCH 0/2] gitlab: avoid timeouts " Thomas Huth
2 siblings, 3 replies; 13+ messages in thread
From: Daniel P. Berrangé @ 2021-08-10 14:06 UTC (permalink / raw)
To: qemu-devel
Cc: Thomas Huth, Daniel P. Berrangé, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Willian Rampazzo, Alex Bennée
We need to cut down compile time by excluding more targets. Both these
targets still have their 64-bit variant enabled, so the loss of coverage
is mitigated to some degree.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
.gitlab-ci.d/crossbuild-template.yml | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml
index 7d3ad00a1e..cfb576b54c 100644
--- a/.gitlab-ci.d/crossbuild-template.yml
+++ b/.gitlab-ci.d/crossbuild-template.yml
@@ -9,7 +9,8 @@
../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
--disable-user --target-list-exclude="arm-softmmu cris-softmmu
i386-softmmu microblaze-softmmu mips-softmmu mipsel-softmmu
- mips64-softmmu ppc-softmmu sh4-softmmu xtensa-softmmu"
+ mips64-softmmu ppc-softmmu riscv32-softmmu sh4-softmmu
+ sparc-softmmu xtensa-softmmu"
- make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
- if grep -q "EXESUF=.exe" config-host.mak;
then make installer;
--
2.31.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH 1/2] gitlab: exclude sparc-softmmu and riscv32-softmmu from cross builds
2021-08-10 14:06 ` [PATCH 1/2] gitlab: exclude sparc-softmmu and riscv32-softmmu from " Daniel P. Berrangé
@ 2021-08-10 14:56 ` Willian Rampazzo
2021-08-11 6:24 ` Philippe Mathieu-Daudé
2021-08-18 9:15 ` Mark Cave-Ayland
2 siblings, 0 replies; 13+ messages in thread
From: Willian Rampazzo @ 2021-08-10 14:56 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Thomas Huth, Alex Bennée, qemu-devel,
Wainer dos Santos Moschetta, Philippe Mathieu-Daudé
On Tue, Aug 10, 2021 at 11:07 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> We need to cut down compile time by excluding more targets. Both these
> targets still have their 64-bit variant enabled, so the loss of coverage
> is mitigated to some degree.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> .gitlab-ci.d/crossbuild-template.yml | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 1/2] gitlab: exclude sparc-softmmu and riscv32-softmmu from cross builds
2021-08-10 14:06 ` [PATCH 1/2] gitlab: exclude sparc-softmmu and riscv32-softmmu from " Daniel P. Berrangé
2021-08-10 14:56 ` Willian Rampazzo
@ 2021-08-11 6:24 ` Philippe Mathieu-Daudé
2021-08-18 9:15 ` Mark Cave-Ayland
2 siblings, 0 replies; 13+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-08-11 6:24 UTC (permalink / raw)
To: Daniel P. Berrangé, qemu-devel, Mark Cave-Ayland, Artyom Tarasenko
Cc: Willian Rampazzo, Thomas Huth, Alex Bennée,
Wainer dos Santos Moschetta
+Mark/Artyom
On 8/10/21 4:06 PM, Daniel P. Berrangé wrote:
> We need to cut down compile time by excluding more targets. Both these
> targets still have their 64-bit variant enabled, so the loss of coverage
> is mitigated to some degree.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> .gitlab-ci.d/crossbuild-template.yml | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml
> index 7d3ad00a1e..cfb576b54c 100644
> --- a/.gitlab-ci.d/crossbuild-template.yml
> +++ b/.gitlab-ci.d/crossbuild-template.yml
> @@ -9,7 +9,8 @@
> ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
> --disable-user --target-list-exclude="arm-softmmu cris-softmmu
> i386-softmmu microblaze-softmmu mips-softmmu mipsel-softmmu
> - mips64-softmmu ppc-softmmu sh4-softmmu xtensa-softmmu"
> + mips64-softmmu ppc-softmmu riscv32-softmmu sh4-softmmu
> + sparc-softmmu xtensa-softmmu"
I disagree for sparc, as there are some differences (sparc64 is not
a superset of it). However I'm not against this patch.
> - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
> - if grep -q "EXESUF=.exe" config-host.mak;
> then make installer;
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 1/2] gitlab: exclude sparc-softmmu and riscv32-softmmu from cross builds
2021-08-10 14:06 ` [PATCH 1/2] gitlab: exclude sparc-softmmu and riscv32-softmmu from " Daniel P. Berrangé
2021-08-10 14:56 ` Willian Rampazzo
2021-08-11 6:24 ` Philippe Mathieu-Daudé
@ 2021-08-18 9:15 ` Mark Cave-Ayland
2021-08-18 9:29 ` Daniel P. Berrangé
2 siblings, 1 reply; 13+ messages in thread
From: Mark Cave-Ayland @ 2021-08-18 9:15 UTC (permalink / raw)
To: Daniel P. Berrangé, qemu-devel
Cc: Willian Rampazzo, Thomas Huth, Alex Bennée,
Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta
On 10/08/2021 15:06, Daniel P. Berrangé wrote:
> We need to cut down compile time by excluding more targets. Both these
> targets still have their 64-bit variant enabled, so the loss of coverage
> is mitigated to some degree.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> .gitlab-ci.d/crossbuild-template.yml | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml
> index 7d3ad00a1e..cfb576b54c 100644
> --- a/.gitlab-ci.d/crossbuild-template.yml
> +++ b/.gitlab-ci.d/crossbuild-template.yml
> @@ -9,7 +9,8 @@
> ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
> --disable-user --target-list-exclude="arm-softmmu cris-softmmu
> i386-softmmu microblaze-softmmu mips-softmmu mipsel-softmmu
> - mips64-softmmu ppc-softmmu sh4-softmmu xtensa-softmmu"
> + mips64-softmmu ppc-softmmu riscv32-softmmu sh4-softmmu
> + sparc-softmmu xtensa-softmmu"
> - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
> - if grep -q "EXESUF=.exe" config-host.mak;
> then make installer;
I'd prefer to keep sparc-softmmu if possible, simply because my everyday platform is
Linux and so having a cross-build for Windows will catch things that I may miss on a
day-to-day basis. Is sparc-softmmu currently enabled as part of the native MINGW64 build?
If I go to my Gitlab QEMU fork Settings -> CI/CD -> Variables there is an option to
set variables that can be used in job scripts. Perhaps this could be used so that I
can configure my personal QEMU fork to always run sparc-softmmu builds when preparing
PRs even if they aren't enabled for everyone by default? At least this would then
allow me to spot any breakage before sending a final PR to Peter.
ATB,
Mark.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 1/2] gitlab: exclude sparc-softmmu and riscv32-softmmu from cross builds
2021-08-18 9:15 ` Mark Cave-Ayland
@ 2021-08-18 9:29 ` Daniel P. Berrangé
2021-08-18 9:45 ` Mark Cave-Ayland
0 siblings, 1 reply; 13+ messages in thread
From: Daniel P. Berrangé @ 2021-08-18 9:29 UTC (permalink / raw)
To: Mark Cave-Ayland
Cc: Thomas Huth, qemu-devel, Wainer dos Santos Moschetta,
Philippe Mathieu-Daudé,
Willian Rampazzo, Alex Bennée
On Wed, Aug 18, 2021 at 10:15:47AM +0100, Mark Cave-Ayland wrote:
> On 10/08/2021 15:06, Daniel P. Berrangé wrote:
>
> > We need to cut down compile time by excluding more targets. Both these
> > targets still have their 64-bit variant enabled, so the loss of coverage
> > is mitigated to some degree.
> >
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> > .gitlab-ci.d/crossbuild-template.yml | 3 ++-
> > 1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml
> > index 7d3ad00a1e..cfb576b54c 100644
> > --- a/.gitlab-ci.d/crossbuild-template.yml
> > +++ b/.gitlab-ci.d/crossbuild-template.yml
> > @@ -9,7 +9,8 @@
> > ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
> > --disable-user --target-list-exclude="arm-softmmu cris-softmmu
> > i386-softmmu microblaze-softmmu mips-softmmu mipsel-softmmu
> > - mips64-softmmu ppc-softmmu sh4-softmmu xtensa-softmmu"
> > + mips64-softmmu ppc-softmmu riscv32-softmmu sh4-softmmu
> > + sparc-softmmu xtensa-softmmu"
> > - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
> > - if grep -q "EXESUF=.exe" config-host.mak;
> > then make installer;
>
> I'd prefer to keep sparc-softmmu if possible, simply because my everyday
> platform is Linux and so having a cross-build for Windows will catch things
> that I may miss on a day-to-day basis. Is sparc-softmmu currently enabled as
> part of the native MINGW64 build?
Note the builds still include 'sparc64-softmmu', so we're only loosing
cover in places where it diverges fromthe 64-bit build, but this will
sstil get coverage in native builds.
> If I go to my Gitlab QEMU fork Settings -> CI/CD -> Variables there is an
> option to set variables that can be used in job scripts. Perhaps this could
> be used so that I can configure my personal QEMU fork to always run
> sparc-softmmu builds when preparing PRs even if they aren't enabled for
> everyone by default? At least this would then allow me to spot any breakage
> before sending a final PR to Peter.
Separately from this I'm doing some work to make things more configurable
https://lists.gnu.org/archive/html/qemu-devel/2021-08/msg02102.html
but not on this level of granularity.
Rather than globally excluding from all cross builds, I think we ought to
split it up more equitably acrss the different builds in some way.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 1/2] gitlab: exclude sparc-softmmu and riscv32-softmmu from cross builds
2021-08-18 9:29 ` Daniel P. Berrangé
@ 2021-08-18 9:45 ` Mark Cave-Ayland
2021-08-18 10:23 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 13+ messages in thread
From: Mark Cave-Ayland @ 2021-08-18 9:45 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Thomas Huth, qemu-devel, Wainer dos Santos Moschetta,
Philippe Mathieu-Daudé,
Willian Rampazzo, Alex Bennée
On 18/08/2021 10:29, Daniel P. Berrangé wrote:
> On Wed, Aug 18, 2021 at 10:15:47AM +0100, Mark Cave-Ayland wrote:
>> On 10/08/2021 15:06, Daniel P. Berrangé wrote:
>>
>>> We need to cut down compile time by excluding more targets. Both these
>>> targets still have their 64-bit variant enabled, so the loss of coverage
>>> is mitigated to some degree.
>>>
>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>> ---
>>> .gitlab-ci.d/crossbuild-template.yml | 3 ++-
>>> 1 file changed, 2 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml
>>> index 7d3ad00a1e..cfb576b54c 100644
>>> --- a/.gitlab-ci.d/crossbuild-template.yml
>>> +++ b/.gitlab-ci.d/crossbuild-template.yml
>>> @@ -9,7 +9,8 @@
>>> ../configure --enable-werror --disable-docs $QEMU_CONFIGURE_OPTS
>>> --disable-user --target-list-exclude="arm-softmmu cris-softmmu
>>> i386-softmmu microblaze-softmmu mips-softmmu mipsel-softmmu
>>> - mips64-softmmu ppc-softmmu sh4-softmmu xtensa-softmmu"
>>> + mips64-softmmu ppc-softmmu riscv32-softmmu sh4-softmmu
>>> + sparc-softmmu xtensa-softmmu"
>>> - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
>>> - if grep -q "EXESUF=.exe" config-host.mak;
>>> then make installer;
>>
>> I'd prefer to keep sparc-softmmu if possible, simply because my everyday
>> platform is Linux and so having a cross-build for Windows will catch things
>> that I may miss on a day-to-day basis. Is sparc-softmmu currently enabled as
>> part of the native MINGW64 build?
>
> Note the builds still include 'sparc64-softmmu', so we're only loosing
> cover in places where it diverges fromthe 64-bit build, but this will
> sstil get coverage in native builds.
Okay I see. From my perspective sparc-softmmu covers more corner cases simply because
everything is wired up as cMMIO peripherals on a custum bus compared with
sparc64-softmmu which is a PCI machine and so there is already a lot of overlap
there. However...
>> If I go to my Gitlab QEMU fork Settings -> CI/CD -> Variables there is an
>> option to set variables that can be used in job scripts. Perhaps this could
>> be used so that I can configure my personal QEMU fork to always run
>> sparc-softmmu builds when preparing PRs even if they aren't enabled for
>> everyone by default? At least this would then allow me to spot any breakage
>> before sending a final PR to Peter.
>
> Separately from this I'm doing some work to make things more configurable
>
> https://lists.gnu.org/archive/html/qemu-devel/2021-08/msg02102.html
>
> but not on this level of granularity.
>
> Rather than globally excluding from all cross builds, I think we ought to
> split it up more equitably acrss the different builds in some way.
From my perspective as long as I have something in Gitlab that I can push and then
get a green CI build that covers both sparc platforms on Linux/Windows then I'm not
too concerned about the implementation details. As long as the combination is covered
in a job somewhere in your latest set of build changes then that's okay with me.
ATB,
Mark.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 1/2] gitlab: exclude sparc-softmmu and riscv32-softmmu from cross builds
2021-08-18 9:45 ` Mark Cave-Ayland
@ 2021-08-18 10:23 ` Philippe Mathieu-Daudé
0 siblings, 0 replies; 13+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-08-18 10:23 UTC (permalink / raw)
To: Mark Cave-Ayland, Daniel P. Berrangé
Cc: Willian Rampazzo, Thomas Huth, Alex Bennée, qemu-devel,
Wainer dos Santos Moschetta
On 8/18/21 11:45 AM, Mark Cave-Ayland wrote:
> On 18/08/2021 10:29, Daniel P. Berrangé wrote:
>> On Wed, Aug 18, 2021 at 10:15:47AM +0100, Mark Cave-Ayland wrote:
>>> On 10/08/2021 15:06, Daniel P. Berrangé wrote:
>>>
>>>> We need to cut down compile time by excluding more targets. Both these
>>>> targets still have their 64-bit variant enabled, so the loss of
>>>> coverage
>>>> is mitigated to some degree.
>>>>
>>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>>> ---
>>>> .gitlab-ci.d/crossbuild-template.yml | 3 ++-
>>>> 1 file changed, 2 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/.gitlab-ci.d/crossbuild-template.yml
>>>> b/.gitlab-ci.d/crossbuild-template.yml
>>>> index 7d3ad00a1e..cfb576b54c 100644
>>>> --- a/.gitlab-ci.d/crossbuild-template.yml
>>>> +++ b/.gitlab-ci.d/crossbuild-template.yml
>>>> @@ -9,7 +9,8 @@
>>>> ../configure --enable-werror --disable-docs
>>>> $QEMU_CONFIGURE_OPTS
>>>> --disable-user --target-list-exclude="arm-softmmu
>>>> cris-softmmu
>>>> i386-softmmu microblaze-softmmu mips-softmmu
>>>> mipsel-softmmu
>>>> - mips64-softmmu ppc-softmmu sh4-softmmu xtensa-softmmu"
>>>> + mips64-softmmu ppc-softmmu riscv32-softmmu sh4-softmmu
>>>> + sparc-softmmu xtensa-softmmu"
>>>> - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
>>>> - if grep -q "EXESUF=.exe" config-host.mak;
>>>> then make installer;
>>>
>>> I'd prefer to keep sparc-softmmu if possible, simply because my everyday
>>> platform is Linux and so having a cross-build for Windows will catch
>>> things
>>> that I may miss on a day-to-day basis. Is sparc-softmmu currently
>>> enabled as
>>> part of the native MINGW64 build?
>>
>> Note the builds still include 'sparc64-softmmu', so we're only loosing
>> cover in places where it diverges fromthe 64-bit build, but this will
>> sstil get coverage in native builds.
>
> Okay I see. From my perspective sparc-softmmu covers more corner cases
> simply because everything is wired up as cMMIO peripherals on a custum
> bus compared with sparc64-softmmu which is a PCI machine and so there is
> already a lot of overlap there. However...
>
>>> If I go to my Gitlab QEMU fork Settings -> CI/CD -> Variables there
>>> is an
>>> option to set variables that can be used in job scripts. Perhaps this
>>> could
>>> be used so that I can configure my personal QEMU fork to always run
>>> sparc-softmmu builds when preparing PRs even if they aren't enabled for
>>> everyone by default? At least this would then allow me to spot any
>>> breakage
>>> before sending a final PR to Peter.
>>
>> Separately from this I'm doing some work to make things more configurable
>>
>> https://lists.gnu.org/archive/html/qemu-devel/2021-08/msg02102.html
>>
>> but not on this level of granularity.
>>
>> Rather than globally excluding from all cross builds, I think we ought to
>> split it up more equitably acrss the different builds in some way.
>
> From my perspective as long as I have something in Gitlab that I can
> push and then get a green CI build that covers both sparc platforms on
> Linux/Windows then I'm not too concerned about the implementation
> details. As long as the combination is covered in a job somewhere in
> your latest set of build changes then that's okay with me.
Your use case seems similar to mine: I overload the current mainstream
pipeline with extra jobs. See:
https://docs.gitlab.com/ee/ci/pipelines/settings.html#specify-a-custom-cicd-configuration-file
My namespace CI points at a custom config in some branch:
https://gitlab.com/philmd/qemu/raw/my_gci_config/.gitlab-ci.d/philmd.yml
This files simply contains:
include:
# Keep mainstream config ...
- local: '/.gitlab-ci.d/qemu-project.yml'
# ... but include my extra jobs.
-
'https://gitlab.com/philmd/qemu/raw/my_gci_config/.gitlab-ci.d/nvme_wip.yml'
-
'https://gitlab.com/philmd/qemu/raw/my_gci_config/.gitlab-ci.d/mips_slow.yml'
-
'https://gitlab.com/philmd/qemu/raw/my_gci_config/.gitlab-ci.d/philmd_runners.yml'
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH 2/2] gitlab: skip many more targets in windows cross builds
2021-08-10 14:06 [PATCH 0/2] gitlab: avoid timeouts in windows cross builds Daniel P. Berrangé
2021-08-10 14:06 ` [PATCH 1/2] gitlab: exclude sparc-softmmu and riscv32-softmmu from " Daniel P. Berrangé
@ 2021-08-10 14:06 ` Daniel P. Berrangé
2021-08-10 14:59 ` Willian Rampazzo
2021-08-11 6:20 ` Philippe Mathieu-Daudé
2021-08-11 5:53 ` [PATCH 0/2] gitlab: avoid timeouts " Thomas Huth
2 siblings, 2 replies; 13+ messages in thread
From: Daniel P. Berrangé @ 2021-08-10 14:06 UTC (permalink / raw)
To: qemu-devel
Cc: Thomas Huth, Daniel P. Berrangé, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta, Willian Rampazzo, Alex Bennée
The windows cross builds still take way too long in gitlab CI, so need
more targets to be skipped. We don't want to hurt coverage of other
cross builds more though, so we let jobs fine tune with a new env
variale $CROSS_SKIP_TARGETS.
We take the set of targets that are considered relatively niche or
very old architectures, and skip approx half of them in win32 builds
and the other half of them in win64.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
.gitlab-ci.d/crossbuild-template.yml | 2 +-
.gitlab-ci.d/crossbuilds.yml | 2 ++
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml
index cfb576b54c..10d22dcf6c 100644
--- a/.gitlab-ci.d/crossbuild-template.yml
+++ b/.gitlab-ci.d/crossbuild-template.yml
@@ -10,7 +10,7 @@
--disable-user --target-list-exclude="arm-softmmu cris-softmmu
i386-softmmu microblaze-softmmu mips-softmmu mipsel-softmmu
mips64-softmmu ppc-softmmu riscv32-softmmu sh4-softmmu
- sparc-softmmu xtensa-softmmu"
+ sparc-softmmu xtensa-softmmu $CROSS_SKIP_TARGETS"
- make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
- if grep -q "EXESUF=.exe" config-host.mak;
then make installer;
diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
index 4ff3aa3cfc..a5f9dbcbeb 100644
--- a/.gitlab-ci.d/crossbuilds.yml
+++ b/.gitlab-ci.d/crossbuilds.yml
@@ -160,6 +160,7 @@ cross-win32-system:
job: win32-fedora-cross-container
variables:
IMAGE: fedora-win32-cross
+ CROSS_SKIP_TARGETS: or1k-softmmu rx-softmmu sh4eb-softmmu sparc64-softmmu tricore-softmmu xtensaeb-softmmu
artifacts:
paths:
- build/qemu-setup*.exe
@@ -170,6 +171,7 @@ cross-win64-system:
job: win64-fedora-cross-container
variables:
IMAGE: fedora-win64-cross
+ CROSS_SKIP_TARGETS: alpha-softmmu avr-softmmu hppa-softmmu m68k-softmmu microblazeel-softmmu mips64el-softmmu nios2-softmmu
artifacts:
paths:
- build/qemu-setup*.exe
--
2.31.1
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH 2/2] gitlab: skip many more targets in windows cross builds
2021-08-10 14:06 ` [PATCH 2/2] gitlab: skip many more targets in windows " Daniel P. Berrangé
@ 2021-08-10 14:59 ` Willian Rampazzo
2021-08-11 6:20 ` Philippe Mathieu-Daudé
1 sibling, 0 replies; 13+ messages in thread
From: Willian Rampazzo @ 2021-08-10 14:59 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Thomas Huth, Alex Bennée, qemu-devel,
Wainer dos Santos Moschetta, Philippe Mathieu-Daudé
On Tue, Aug 10, 2021 at 11:07 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> The windows cross builds still take way too long in gitlab CI, so need
> more targets to be skipped. We don't want to hurt coverage of other
> cross builds more though, so we let jobs fine tune with a new env
> variale $CROSS_SKIP_TARGETS.
s/variale/variable
>
> We take the set of targets that are considered relatively niche or
> very old architectures, and skip approx half of them in win32 builds
> and the other half of them in win64.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> .gitlab-ci.d/crossbuild-template.yml | 2 +-
> .gitlab-ci.d/crossbuilds.yml | 2 ++
> 2 files changed, 3 insertions(+), 1 deletion(-)
>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 2/2] gitlab: skip many more targets in windows cross builds
2021-08-10 14:06 ` [PATCH 2/2] gitlab: skip many more targets in windows " Daniel P. Berrangé
2021-08-10 14:59 ` Willian Rampazzo
@ 2021-08-11 6:20 ` Philippe Mathieu-Daudé
2021-08-11 8:43 ` Daniel P. Berrangé
1 sibling, 1 reply; 13+ messages in thread
From: Philippe Mathieu-Daudé @ 2021-08-11 6:20 UTC (permalink / raw)
To: Daniel P. Berrangé, qemu-devel
Cc: Willian Rampazzo, Thomas Huth, Alex Bennée,
Wainer dos Santos Moschetta
On 8/10/21 4:06 PM, Daniel P. Berrangé wrote:
> The windows cross builds still take way too long in gitlab CI, so need
> more targets to be skipped. We don't want to hurt coverage of other
> cross builds more though, so we let jobs fine tune with a new env
> variale $CROSS_SKIP_TARGETS.
>
> We take the set of targets that are considered relatively niche or
> very old architectures, and skip approx half of them in win32 builds
> and the other half of them in win64.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> .gitlab-ci.d/crossbuild-template.yml | 2 +-
> .gitlab-ci.d/crossbuilds.yml | 2 ++
> 2 files changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml
> index cfb576b54c..10d22dcf6c 100644
> --- a/.gitlab-ci.d/crossbuild-template.yml
> +++ b/.gitlab-ci.d/crossbuild-template.yml
> @@ -10,7 +10,7 @@
> --disable-user --target-list-exclude="arm-softmmu cris-softmmu
> i386-softmmu microblaze-softmmu mips-softmmu mipsel-softmmu
> mips64-softmmu ppc-softmmu riscv32-softmmu sh4-softmmu
> - sparc-softmmu xtensa-softmmu"
> + sparc-softmmu xtensa-softmmu $CROSS_SKIP_TARGETS"
> - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
> - if grep -q "EXESUF=.exe" config-host.mak;
> then make installer;
> diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
> index 4ff3aa3cfc..a5f9dbcbeb 100644
> --- a/.gitlab-ci.d/crossbuilds.yml
> +++ b/.gitlab-ci.d/crossbuilds.yml
> @@ -160,6 +160,7 @@ cross-win32-system:
> job: win32-fedora-cross-container
> variables:
> IMAGE: fedora-win32-cross
> + CROSS_SKIP_TARGETS: or1k-softmmu rx-softmmu sh4eb-softmmu sparc64-softmmu tricore-softmmu xtensaeb-softmmu
> artifacts:
> paths:
> - build/qemu-setup*.exe
> @@ -170,6 +171,7 @@ cross-win64-system:
> job: win64-fedora-cross-container
> variables:
> IMAGE: fedora-win64-cross
> + CROSS_SKIP_TARGETS: alpha-softmmu avr-softmmu hppa-softmmu m68k-softmmu microblazeel-softmmu mips64el-softmmu nios2-softmmu
It looks you inverted the lists. We expect more Win64 users, and these
are the targets of interest. I'd keep here (skipping in Win32):
alpha-softmmu avr-softmmu hppa-softmmu m68k-softmmu microblazeel-softmmu
mips64el-softmmu sparc64-softmmu
And skip (keep them in win32):
or1k-softmmu rx-softmmu sh4eb-softmmu nios2-softmmu tricore-softmmu
xtensaeb-softmmu
> artifacts:
> paths:
> - build/qemu-setup*.exe
>
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 2/2] gitlab: skip many more targets in windows cross builds
2021-08-11 6:20 ` Philippe Mathieu-Daudé
@ 2021-08-11 8:43 ` Daniel P. Berrangé
0 siblings, 0 replies; 13+ messages in thread
From: Daniel P. Berrangé @ 2021-08-11 8:43 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: Willian Rampazzo, Thomas Huth, Alex Bennée, qemu-devel,
Wainer dos Santos Moschetta
On Wed, Aug 11, 2021 at 08:20:37AM +0200, Philippe Mathieu-Daudé wrote:
> On 8/10/21 4:06 PM, Daniel P. Berrangé wrote:
> > The windows cross builds still take way too long in gitlab CI, so need
> > more targets to be skipped. We don't want to hurt coverage of other
> > cross builds more though, so we let jobs fine tune with a new env
> > variale $CROSS_SKIP_TARGETS.
> >
> > We take the set of targets that are considered relatively niche or
> > very old architectures, and skip approx half of them in win32 builds
> > and the other half of them in win64.
> >
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> > .gitlab-ci.d/crossbuild-template.yml | 2 +-
> > .gitlab-ci.d/crossbuilds.yml | 2 ++
> > 2 files changed, 3 insertions(+), 1 deletion(-)
> >
> > diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml
> > index cfb576b54c..10d22dcf6c 100644
> > --- a/.gitlab-ci.d/crossbuild-template.yml
> > +++ b/.gitlab-ci.d/crossbuild-template.yml
> > @@ -10,7 +10,7 @@
> > --disable-user --target-list-exclude="arm-softmmu cris-softmmu
> > i386-softmmu microblaze-softmmu mips-softmmu mipsel-softmmu
> > mips64-softmmu ppc-softmmu riscv32-softmmu sh4-softmmu
> > - sparc-softmmu xtensa-softmmu"
> > + sparc-softmmu xtensa-softmmu $CROSS_SKIP_TARGETS"
> > - make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
> > - if grep -q "EXESUF=.exe" config-host.mak;
> > then make installer;
> > diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
> > index 4ff3aa3cfc..a5f9dbcbeb 100644
> > --- a/.gitlab-ci.d/crossbuilds.yml
> > +++ b/.gitlab-ci.d/crossbuilds.yml
> > @@ -160,6 +160,7 @@ cross-win32-system:
> > job: win32-fedora-cross-container
> > variables:
> > IMAGE: fedora-win32-cross
> > + CROSS_SKIP_TARGETS: or1k-softmmu rx-softmmu sh4eb-softmmu sparc64-softmmu tricore-softmmu xtensaeb-softmmu
> > artifacts:
> > paths:
> > - build/qemu-setup*.exe
> > @@ -170,6 +171,7 @@ cross-win64-system:
> > job: win64-fedora-cross-container
> > variables:
> > IMAGE: fedora-win64-cross
> > + CROSS_SKIP_TARGETS: alpha-softmmu avr-softmmu hppa-softmmu m68k-softmmu microblazeel-softmmu mips64el-softmmu nios2-softmmu
>
> It looks you inverted the lists. We expect more Win64 users, and these
> are the targets of interest. I'd keep here (skipping in Win32):
I wouldn't say inverted, because the split was entirely arbitrary. While
32-bit vs 64-bit could conceivably show a difference, in practice those
differences are likely to already be caught by non-windows builds. These
jobs are most important for catching UNIX vs Windows differences, so in
that view whether a target is built for win32 vs win64 doesn't matter.
>
> alpha-softmmu avr-softmmu hppa-softmmu m68k-softmmu microblazeel-softmmu
> mips64el-softmmu sparc64-softmmu
>
> And skip (keep them in win32):
>
> or1k-softmmu rx-softmmu sh4eb-softmmu nios2-softmmu tricore-softmmu
> xtensaeb-softmmu
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 0/2] gitlab: avoid timeouts in windows cross builds
2021-08-10 14:06 [PATCH 0/2] gitlab: avoid timeouts in windows cross builds Daniel P. Berrangé
2021-08-10 14:06 ` [PATCH 1/2] gitlab: exclude sparc-softmmu and riscv32-softmmu from " Daniel P. Berrangé
2021-08-10 14:06 ` [PATCH 2/2] gitlab: skip many more targets in windows " Daniel P. Berrangé
@ 2021-08-11 5:53 ` Thomas Huth
2 siblings, 0 replies; 13+ messages in thread
From: Thomas Huth @ 2021-08-11 5:53 UTC (permalink / raw)
To: Daniel P. Berrangé, qemu-devel
Cc: Willian Rampazzo, Alex Bennée, Philippe Mathieu-Daudé,
Wainer dos Santos Moschetta
On 10/08/2021 16.06, Daniel P. Berrangé wrote:
> The win64 cross build is quite often hitting the job timeout, despite
> having an elevated timeout of 80 minutes. A typical time is 75-78
> minutes in my tests, which leaves little headroom.
>
> I'm not inclined to increase the timeout beyond 80 minutes as this is
> already unreasonably long.
>
> Interestingly the win32 job doesn't timeout in the same way. In local
> tests I've found the mingw32 gcc is as much as 50% faster than
> mingw64 gcc in building QEMU. This explains at least why we only see
> win64 gitlab builds timeout normally.
I've had similar experiences in the past, and once even already sent a patch:
https://patchwork.ozlabs.org/project/qemu-devel/patch/20210414081907.871437-4-thuth@redhat.com/
But your apprach is looking way nicer, indeed!
Queued to my testing-next branch now:
https://gitlab.com/thuth/qemu/-/commits/testing-next/
Thanks,
Thomas
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2021-08-18 10:24 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-10 14:06 [PATCH 0/2] gitlab: avoid timeouts in windows cross builds Daniel P. Berrangé
2021-08-10 14:06 ` [PATCH 1/2] gitlab: exclude sparc-softmmu and riscv32-softmmu from " Daniel P. Berrangé
2021-08-10 14:56 ` Willian Rampazzo
2021-08-11 6:24 ` Philippe Mathieu-Daudé
2021-08-18 9:15 ` Mark Cave-Ayland
2021-08-18 9:29 ` Daniel P. Berrangé
2021-08-18 9:45 ` Mark Cave-Ayland
2021-08-18 10:23 ` Philippe Mathieu-Daudé
2021-08-10 14:06 ` [PATCH 2/2] gitlab: skip many more targets in windows " Daniel P. Berrangé
2021-08-10 14:59 ` Willian Rampazzo
2021-08-11 6:20 ` Philippe Mathieu-Daudé
2021-08-11 8:43 ` Daniel P. Berrangé
2021-08-11 5:53 ` [PATCH 0/2] gitlab: avoid timeouts " Thomas Huth
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.