qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* [Qemu-devel] more automated/public CI for QEMU pullreqs
@ 2019-08-16 18:16 Peter Maydell
  2019-08-22 10:49 ` Stefan Hajnoczi
  2019-08-22 11:47 ` [Qemu-devel] " Daniel P. Berrangé
  0 siblings, 2 replies; 14+ messages in thread
From: Peter Maydell @ 2019-08-16 18:16 UTC (permalink / raw)
  To: QEMU Developers
  Cc: Samuel Ortiz, Kashyap Chamarthy, Philippe Mathieu-Daudé,
	Richard Henderson, Stefan Hajnoczi, Paolo Bonzini,
	Alex Bennée

We had a conversation some months back about ways we might switch
away from the current handling of pull requests which I do via some
hand-curated scripts and personal access to machines, to a more
automated system that could be operated by a wider range of people.
Unfortunately that conversation took place off-list (largely my fault
for forgetting a cc: at the beginning of the email chain), and in any
case it sort of fizzled out.  So let's restart it, on the mailing
list this time.

Here's a summary of stuff from the old thread and general
explanation of the problem:

My current setup is mostly just running the equivalent of
"make && make check" on a bunch of machines and configs
on the merge commit before I push it to master. I also do
a 'make check-tcg' on one of the builds and run a variant
of the 'linux-user-test' tarball of 'ls' binaries.
The scripts do some simple initial checks which mostly are
preventing problems seen in the past:
 * don't allow submodules to be updated unless I kick the
   merge off with a command line option saying submodule updates
   are OK here (this catches accidental misdriving of git by
   a submaintainer folding a submodule change into a patch
   during a rebase)
 * check we aren't trying to merge after tagging the final
   release but before doing the 'reopen development tree'
   commit that bumps the VERSION file
 * check for bogus "author is qemu-devel@nongnu.org" commits
 * check for UTF-8 mangling
 * check the gpg signature on the pullreq
A human needs to also eyeball the commits and the diffstat
for weird stuff (usually cursory for frequent pullreq submitters,
and more carefully for new submaintainers).

I have this semi-automated with some hacky scripts.  The major thing we
need for a replacement is the coverage of different host
architectures and operating systems, which is a poor match to most of
the cloud-CI services out there (including Travis).  We also want the
tests to run in a reasonably short wall-clock time from being kicked
off.

Awkward bonus extra requirement: it would be useful to be
able to do a merge CI run "privately", eg because the thing
being tested is a fix for a security bug that's not yet
public. But that's rare so we can probably do it by hand.

There are some other parts to this, like getting some kind of
project-role-account access to machines where that's OK, or finding
replacements where the machines really are my personal ones or
otherwise not OK for project access.  But I think that should be
fairly easy to resolve so let's keep this thread to the
automating-the-CI part.

The two major contenders suggested were:

(1) GitLab CI, which supports custom 'runners' which we can set
up to run builds and tests on machines we have project access to

(2) Patchew, which can handle running tests on multiple machines (eg
we do s390 testing today for all patches on list), and which we could
enhance to provide support for the release-manager to do their work

Advantages of Gitlab CI:
 * somebody else is doing the development and maintainance of the
   CI tool -- bigger 'bus factor' than patchew
 * already does (more or less) what we want without needing
   extra coding work

Advantages of Patchew:
 * we're already using it for patch submissions, so we know it's
   not going to go away
 * it's very easy to deploy to a new host
 * no dependencies except Python, so works anywhere we expect
   to be able to build QEMU (whereas gitlab CI's runner is
   written in Go, and there seem to be ongoing issues with getting
   it actually to compile for other architectures than x86)

I don't have an opinion really, but I think it would be good to
make a choice and start working forwards towards getting this
a bit less hacky and a bit more offloadable to other people.

Perhaps a good first step would be to keep the 'simple checks
of broken commits' part done as a local script but have the
CI done via "push proposed merge commit to $SOMEWHERE to
kick off the CI".

Input, opinions, recommendations, offers to do some of the work? :-)

thanks
-- PMM


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Qemu-devel] more automated/public CI for QEMU pullreqs
  2019-08-16 18:16 [Qemu-devel] more automated/public CI for QEMU pullreqs Peter Maydell
@ 2019-08-22 10:49 ` Stefan Hajnoczi
  2019-08-22 11:04   ` Peter Maydell
  2019-09-30 12:53   ` Peter Maydell
  2019-08-22 11:47 ` [Qemu-devel] " Daniel P. Berrangé
  1 sibling, 2 replies; 14+ messages in thread
From: Stefan Hajnoczi @ 2019-08-22 10:49 UTC (permalink / raw)
  To: Peter Maydell
  Cc: Samuel Ortiz, Kashyap Chamarthy, Philippe Mathieu-Daudé,
	Richard Henderson, QEMU Developers, Paolo Bonzini,
	Alex Bennée

[-- Attachment #1: Type: text/plain, Size: 5107 bytes --]

On Fri, Aug 16, 2019 at 07:16:55PM +0100, Peter Maydell wrote:
> We had a conversation some months back about ways we might switch
> away from the current handling of pull requests which I do via some
> hand-curated scripts and personal access to machines, to a more
> automated system that could be operated by a wider range of people.
> Unfortunately that conversation took place off-list (largely my fault
> for forgetting a cc: at the beginning of the email chain), and in any
> case it sort of fizzled out.  So let's restart it, on the mailing
> list this time.
> 
> Here's a summary of stuff from the old thread and general
> explanation of the problem:
> 
> My current setup is mostly just running the equivalent of
> "make && make check" on a bunch of machines and configs
> on the merge commit before I push it to master. I also do
> a 'make check-tcg' on one of the builds and run a variant
> of the 'linux-user-test' tarball of 'ls' binaries.
> The scripts do some simple initial checks which mostly are
> preventing problems seen in the past:
>  * don't allow submodules to be updated unless I kick the
>    merge off with a command line option saying submodule updates
>    are OK here (this catches accidental misdriving of git by
>    a submaintainer folding a submodule change into a patch
>    during a rebase)
>  * check we aren't trying to merge after tagging the final
>    release but before doing the 'reopen development tree'
>    commit that bumps the VERSION file
>  * check for bogus "author is qemu-devel@nongnu.org" commits
>  * check for UTF-8 mangling
>  * check the gpg signature on the pullreq
> A human needs to also eyeball the commits and the diffstat
> for weird stuff (usually cursory for frequent pullreq submitters,
> and more carefully for new submaintainers).
> 
> I have this semi-automated with some hacky scripts.  The major thing we
> need for a replacement is the coverage of different host
> architectures and operating systems, which is a poor match to most of
> the cloud-CI services out there (including Travis).  We also want the
> tests to run in a reasonably short wall-clock time from being kicked
> off.
> 
> Awkward bonus extra requirement: it would be useful to be
> able to do a merge CI run "privately", eg because the thing
> being tested is a fix for a security bug that's not yet
> public. But that's rare so we can probably do it by hand.
> 
> There are some other parts to this, like getting some kind of
> project-role-account access to machines where that's OK, or finding
> replacements where the machines really are my personal ones or
> otherwise not OK for project access.  But I think that should be
> fairly easy to resolve so let's keep this thread to the
> automating-the-CI part.
> 
> The two major contenders suggested were:
> 
> (1) GitLab CI, which supports custom 'runners' which we can set
> up to run builds and tests on machines we have project access to
> 
> (2) Patchew, which can handle running tests on multiple machines (eg
> we do s390 testing today for all patches on list), and which we could
> enhance to provide support for the release-manager to do their work
> 
> Advantages of Gitlab CI:
>  * somebody else is doing the development and maintainance of the
>    CI tool -- bigger 'bus factor' than patchew
>  * already does (more or less) what we want without needing
>    extra coding work
> 
> Advantages of Patchew:
>  * we're already using it for patch submissions, so we know it's
>    not going to go away
>  * it's very easy to deploy to a new host
>  * no dependencies except Python, so works anywhere we expect
>    to be able to build QEMU (whereas gitlab CI's runner is
>    written in Go, and there seem to be ongoing issues with getting
>    it actually to compile for other architectures than x86)
> 
> I don't have an opinion really, but I think it would be good to
> make a choice and start working forwards towards getting this
> a bit less hacky and a bit more offloadable to other people.
> 
> Perhaps a good first step would be to keep the 'simple checks
> of broken commits' part done as a local script but have the
> CI done via "push proposed merge commit to $SOMEWHERE to
> kick off the CI".
> 
> Input, opinions, recommendations, offers to do some of the work? :-)

Thanks for writing up this summary!  The requirements and possible
solutions are now clear.

We need someone willing to set up and maintain the CI.

One-off tasks:

1. Create CI runners that offer similar cross-architecture coverage to
   Peter's current setup.  qemu.org has some x86, ppc, and s390 server
   resources available.  I'm not sure about ARM and other architectures.

2. Write CI configuration to run Peter's "make && make check && make
   check-tcg && linux-user-test".

3. Document the CI on wiki.qemu.org.

Ongoing responsibilities:

1. Triage failures that the qemu.git maintainer thinks are related to CI
   runners.

2. Keep the CI up-to-date and runners online.

Any volunteers?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Qemu-devel] more automated/public CI for QEMU pullreqs
  2019-08-22 10:49 ` Stefan Hajnoczi
@ 2019-08-22 11:04   ` Peter Maydell
  2019-08-22 11:14     ` Paolo Bonzini
  2019-09-30 12:53   ` Peter Maydell
  1 sibling, 1 reply; 14+ messages in thread
From: Peter Maydell @ 2019-08-22 11:04 UTC (permalink / raw)
  To: Stefan Hajnoczi
  Cc: Samuel Ortiz, Kashyap Chamarthy, Philippe Mathieu-Daudé,
	Richard Henderson, QEMU Developers, Paolo Bonzini,
	Alex Bennée

On Thu, 22 Aug 2019 at 11:50, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> One-off tasks:
>
> 1. Create CI runners that offer similar cross-architecture coverage to
>    Peter's current setup.  qemu.org has some x86, ppc, and s390 server
>    resources available.  I'm not sure about ARM and other architectures.

Quick list of what I currently have:
 * freebsd/openbsd/netbsd -- these are just running the tests/vm stuff
 * an x86-64 machine doing the w32/w64 crossbuilds
 * aarch64
 * aarch32 (done via chroot on an aarch64 box)
 * osx
 * ppc64 (on a gcc compile farm machine)
 * s390 (on a machine Christian arranged for project use)
 * x86-64 various configs
 * there's also a sparc64 box in the gcc compile farm, but I am
   not currently using it because 'make check' hangs intermittently
   in a weird way that last time I tried to dig into it made me
   suspect a bug in the host kernel (we were doing a syscall with
   what seemed like a valid timeout and never timing out)

Of those the s390 is OK for project use. I'm not sure the gcc
compile farm stuff is recommended for fully automatic CI, so
if we can avoid that it would be better. Linaro can probably
provide some arm-server resources but I haven't checked/asked yet.

> 2. Write CI configuration to run Peter's "make && make check && make
>    check-tcg && linux-user-test".

My local scripting for all this is archived here:
https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree

thanks
-- PMM


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Qemu-devel] more automated/public CI for QEMU pullreqs
  2019-08-22 11:04   ` Peter Maydell
@ 2019-08-22 11:14     ` Paolo Bonzini
  0 siblings, 0 replies; 14+ messages in thread
From: Paolo Bonzini @ 2019-08-22 11:14 UTC (permalink / raw)
  To: Peter Maydell, Stefan Hajnoczi
  Cc: Samuel Ortiz, Kashyap Chamarthy, Alex Bennée,
	Richard Henderson, QEMU Developers, Philippe Mathieu-Daudé

On 22/08/19 13:04, Peter Maydell wrote:
> On Thu, 22 Aug 2019 at 11:50, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>> One-off tasks:
>>
>> 1. Create CI runners that offer similar cross-architecture coverage to
>>    Peter's current setup.  qemu.org has some x86, ppc, and s390 server
>>    resources available.  I'm not sure about ARM and other architectures.
> 
> Quick list of what I currently have:
>  * freebsd/openbsd/netbsd -- these are just running the tests/vm stuff
>  * an x86-64 machine doing the w32/w64 crossbuilds
>  * aarch64
>  * aarch32 (done via chroot on an aarch64 box)
>  * osx

I was thinking of setting up a Macincloud account with a Patchew runner
for this.

>  * ppc64 (on a gcc compile farm machine)
>  * s390 (on a machine Christian arranged for project use)

Great, we could also move the Patchew tester off the Fedora machine that
is getting a bit crowded.

David, any chance you could add Docker support to ppc64 builds?  I don't
feel too comfortable doing untrusted builds inside the RH network
without any protection from containers etc.

> Of those the s390 is OK for project use. I'm not sure the gcc
> compile farm stuff is recommended for fully automatic CI, so
> if we can avoid that it would be better. Linaro can probably
> provide some arm-server resources but I haven't checked/asked yet.

Same as above for Docker support.  If ARM builds can be containerized, I
can also look for resources inside Red Hat.

Paolo


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Qemu-devel] more automated/public CI for QEMU pullreqs
  2019-08-16 18:16 [Qemu-devel] more automated/public CI for QEMU pullreqs Peter Maydell
  2019-08-22 10:49 ` Stefan Hajnoczi
@ 2019-08-22 11:47 ` Daniel P. Berrangé
  2019-08-22 16:31   ` Dr. David Alan Gilbert
  1 sibling, 1 reply; 14+ messages in thread
From: Daniel P. Berrangé @ 2019-08-22 11:47 UTC (permalink / raw)
  To: Peter Maydell
  Cc: Samuel Ortiz, Kashyap Chamarthy, Alex Bennée,
	Richard Henderson, QEMU Developers, Stefan Hajnoczi,
	Paolo Bonzini, Philippe Mathieu-Daudé

On Fri, Aug 16, 2019 at 07:16:55PM +0100, Peter Maydell wrote:
> The two major contenders suggested were:
> 
> (1) GitLab CI, which supports custom 'runners' which we can set
> up to run builds and tests on machines we have project access to
> 
> (2) Patchew, which can handle running tests on multiple machines (eg
> we do s390 testing today for all patches on list), and which we could
> enhance to provide support for the release-manager to do their work
> 
> Advantages of Gitlab CI:
>  * somebody else is doing the development and maintainance of the
>    CI tool -- bigger 'bus factor' than patchew
>  * already does (more or less) what we want without needing
>    extra coding work
> 
> Advantages of Patchew:
>  * we're already using it for patch submissions, so we know it's
>    not going to go away
>  * it's very easy to deploy to a new host
>  * no dependencies except Python, so works anywhere we expect
>    to be able to build QEMU (whereas gitlab CI's runner is
>    written in Go, and there seem to be ongoing issues with getting
>    it actually to compile for other architectures than x86)

IMHO the development tools/processes chosen should enable the project
contributors to maximise the time they can put into developing useful
features for QEMU itself. Time we spend writing CI systems like patchew
is time we're taking away from writing QEMU features, unless the new CI
system offers major efficiency benefits over other existing solutions.

A second important aspect is that to enable a smooth/shallow onramp
for new contributors it is useful to have our development processes
and tools be familiar to those with general open source dev experience
outside QEMU.

With both these points in mind, I think it is  pretty hard sell to
say we should write & maintain a custom CI system just for QEMU
unless it is offering major compelling functionality we can't do
without.

IOW, I'd be biased in favour of GitLab CI.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Qemu-devel] more automated/public CI for QEMU pullreqs
  2019-08-22 11:47 ` [Qemu-devel] " Daniel P. Berrangé
@ 2019-08-22 16:31   ` Dr. David Alan Gilbert
  2019-08-22 16:48     ` Paolo Bonzini
  0 siblings, 1 reply; 14+ messages in thread
From: Dr. David Alan Gilbert @ 2019-08-22 16:31 UTC (permalink / raw)
  To: Daniel P. Berrangé
  Cc: Peter Maydell, Samuel Ortiz, Kashyap Chamarthy,
	Philippe Mathieu-Daudé,
	Richard Henderson, QEMU Developers, Stefan Hajnoczi,
	Paolo Bonzini, Alex Bennée

* Daniel P. Berrangé (berrange@redhat.com) wrote:
> On Fri, Aug 16, 2019 at 07:16:55PM +0100, Peter Maydell wrote:
> > The two major contenders suggested were:
> > 
> > (1) GitLab CI, which supports custom 'runners' which we can set
> > up to run builds and tests on machines we have project access to
> > 
> > (2) Patchew, which can handle running tests on multiple machines (eg
> > we do s390 testing today for all patches on list), and which we could
> > enhance to provide support for the release-manager to do their work
> > 
> > Advantages of Gitlab CI:
> >  * somebody else is doing the development and maintainance of the
> >    CI tool -- bigger 'bus factor' than patchew
> >  * already does (more or less) what we want without needing
> >    extra coding work
> > 
> > Advantages of Patchew:
> >  * we're already using it for patch submissions, so we know it's
> >    not going to go away
> >  * it's very easy to deploy to a new host
> >  * no dependencies except Python, so works anywhere we expect
> >    to be able to build QEMU (whereas gitlab CI's runner is
> >    written in Go, and there seem to be ongoing issues with getting
> >    it actually to compile for other architectures than x86)
> 
> IMHO the development tools/processes chosen should enable the project
> contributors to maximise the time they can put into developing useful
> features for QEMU itself. Time we spend writing CI systems like patchew
> is time we're taking away from writing QEMU features, unless the new CI
> system offers major efficiency benefits over other existing solutions.
> 
> A second important aspect is that to enable a smooth/shallow onramp
> for new contributors it is useful to have our development processes
> and tools be familiar to those with general open source dev experience
> outside QEMU.
> 
> With both these points in mind, I think it is  pretty hard sell to
> say we should write & maintain a custom CI system just for QEMU
> unless it is offering major compelling functionality we can't do
> without.
> 
> IOW, I'd be biased in favour of GitLab CI.

I'd agree; and I'd also find it useful to have runners setup for
Gitlab CI for related things (it would be useful for the virtio-fs
stuff);  if there are problems on other architectures then we should
find some go wranglers to go fix it.

Dave

> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Qemu-devel] more automated/public CI for QEMU pullreqs
  2019-08-22 16:31   ` Dr. David Alan Gilbert
@ 2019-08-22 16:48     ` Paolo Bonzini
  2019-08-22 16:50       ` Dr. David Alan Gilbert
  2019-08-22 17:03       ` Daniel P. Berrangé
  0 siblings, 2 replies; 14+ messages in thread
From: Paolo Bonzini @ 2019-08-22 16:48 UTC (permalink / raw)
  To: Dr. David Alan Gilbert, Daniel P. Berrangé
  Cc: Peter Maydell, Samuel Ortiz, Kashyap Chamarthy,
	Philippe Mathieu-Daudé,
	Richard Henderson, QEMU Developers, Stefan Hajnoczi,
	Alex Bennée

On 22/08/19 18:31, Dr. David Alan Gilbert wrote:
>> With both these points in mind, I think it is  pretty hard sell to
>> say we should write & maintain a custom CI system just for QEMU
>> unless it is offering major compelling functionality we can't do
>> without.

In theory I agree.

In practice, the major compelling functionality is portability.  If it
is true that setting up runners is problematic even on aarch64, frankly
GitLab CI is dead on arrival.  If it is not true, then I'd be very happy
to use GitLab CI too.

Paolo

> I'd agree; and I'd also find it useful to have runners setup for
> Gitlab CI for related things (it would be useful for the virtio-fs
> stuff);  if there are problems on other architectures then we should
> find some go wranglers to go fix it.


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Qemu-devel] more automated/public CI for QEMU pullreqs
  2019-08-22 16:48     ` Paolo Bonzini
@ 2019-08-22 16:50       ` Dr. David Alan Gilbert
  2019-08-22 17:05         ` Paolo Bonzini
  2019-08-22 17:03       ` Daniel P. Berrangé
  1 sibling, 1 reply; 14+ messages in thread
From: Dr. David Alan Gilbert @ 2019-08-22 16:50 UTC (permalink / raw)
  To: Paolo Bonzini
  Cc: Peter Maydell, Daniel P. Berrangé,
	Samuel Ortiz, Kashyap Chamarthy, Philippe Mathieu-Daudé,
	Richard Henderson, QEMU Developers, Stefan Hajnoczi,
	Alex Bennée

* Paolo Bonzini (pbonzini@redhat.com) wrote:
> On 22/08/19 18:31, Dr. David Alan Gilbert wrote:
> >> With both these points in mind, I think it is  pretty hard sell to
> >> say we should write & maintain a custom CI system just for QEMU
> >> unless it is offering major compelling functionality we can't do
> >> without.

(That was Dan's comment)

> In theory I agree.
> 
> In practice, the major compelling functionality is portability.  If it
> is true that setting up runners is problematic even on aarch64, frankly
> GitLab CI is dead on arrival.  If it is not true, then I'd be very happy
> to use GitLab CI too.

IMHO if for some weird reason Gitlab has problems on aarch64 then we just need to get that
fixed.

Dave

> Paolo
> 
> > I'd agree; and I'd also find it useful to have runners setup for
> > Gitlab CI for related things (it would be useful for the virtio-fs
> > stuff);  if there are problems on other architectures then we should
> > find some go wranglers to go fix it.
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Qemu-devel] more automated/public CI for QEMU pullreqs
  2019-08-22 16:48     ` Paolo Bonzini
  2019-08-22 16:50       ` Dr. David Alan Gilbert
@ 2019-08-22 17:03       ` Daniel P. Berrangé
  1 sibling, 0 replies; 14+ messages in thread
From: Daniel P. Berrangé @ 2019-08-22 17:03 UTC (permalink / raw)
  To: Paolo Bonzini
  Cc: Peter Maydell, Samuel Ortiz, Kashyap Chamarthy,
	Philippe Mathieu-Daudé,
	Richard Henderson, Dr. David Alan Gilbert, QEMU Developers,
	Stefan Hajnoczi, Alex Bennée

On Thu, Aug 22, 2019 at 06:48:08PM +0200, Paolo Bonzini wrote:
> On 22/08/19 18:31, Dr. David Alan Gilbert wrote:
> >> With both these points in mind, I think it is  pretty hard sell to
> >> say we should write & maintain a custom CI system just for QEMU
> >> unless it is offering major compelling functionality we can't do
> >> without.
> 
> In theory I agree.
> 
> In practice, the major compelling functionality is portability.  If it
> is true that setting up runners is problematic even on aarch64, frankly
> GitLab CI is dead on arrival.  If it is not true, then I'd be very happy
> to use GitLab CI too.

Sure, I did qualify my statement in the text before that quoted here
to say that the patchew would have to offer some compelling feature
over GitLab. If GitLab can't be made to work with non-x86 runners,
then clearly non-x86 portability is the compelling feature that would
make Patchew a sensible option for QEMU.

I don't know what the problems are wrt to aarch64 & GitLab but the
scheme for connecting new runners to GitLab CI looks pretty flexible
when I examined it, so I'd really hope non-x86 is not a show stopper.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Qemu-devel] more automated/public CI for QEMU pullreqs
  2019-08-22 16:50       ` Dr. David Alan Gilbert
@ 2019-08-22 17:05         ` Paolo Bonzini
  2019-08-23 13:15           ` Philippe Mathieu-Daudé
  0 siblings, 1 reply; 14+ messages in thread
From: Paolo Bonzini @ 2019-08-22 17:05 UTC (permalink / raw)
  To: Dr. David Alan Gilbert
  Cc: Peter Maydell, Daniel P. Berrangé,
	Samuel Ortiz, Kashyap Chamarthy, Philippe Mathieu-Daudé,
	Richard Henderson, QEMU Developers, Stefan Hajnoczi,
	Alex Bennée

On 22/08/19 18:50, Dr. David Alan Gilbert wrote:
> * Paolo Bonzini (pbonzini@redhat.com) wrote:
>> On 22/08/19 18:31, Dr. David Alan Gilbert wrote:
>>>> With both these points in mind, I think it is  pretty hard sell to
>>>> say we should write & maintain a custom CI system just for QEMU
>>>> unless it is offering major compelling functionality we can't do
>>>> without.
> 
> (That was Dan's comment)
> 
>> In theory I agree.
>>
>> In practice, the major compelling functionality is portability.  If it
>> is true that setting up runners is problematic even on aarch64, frankly
>> GitLab CI is dead on arrival.  If it is not true, then I'd be very happy
>> to use GitLab CI too.
> 
> IMHO if for some weird reason Gitlab has problems on aarch64 then we
> just need to get that fixed.

I'm sure it's just some packaging or deployment issue.  But
https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/725 has been
open for more than one year; the last two messages are:

* 1 month ago: "I hope we will be able to merge it soon"

* 3 weeks ago: "Today I tried use gitlab-runner on my arm64 box, however
it kept mysteriously failing"

So the question is simply who does the work.

Paolo

> Dave
> 
>> Paolo
>>
>>> I'd agree; and I'd also find it useful to have runners setup for
>>> Gitlab CI for related things (it would be useful for the virtio-fs
>>> stuff);  if there are problems on other architectures then we should
>>> find some go wranglers to go fix it.
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Qemu-devel] more automated/public CI for QEMU pullreqs
  2019-08-22 17:05         ` Paolo Bonzini
@ 2019-08-23 13:15           ` Philippe Mathieu-Daudé
  2019-08-24  7:44             ` Alex Bennée
  0 siblings, 1 reply; 14+ messages in thread
From: Philippe Mathieu-Daudé @ 2019-08-23 13:15 UTC (permalink / raw)
  To: Paolo Bonzini, Dr. David Alan Gilbert
  Cc: Peter Maydell, Daniel P. Berrangé,
	Samuel Ortiz, Kashyap Chamarthy, Richard Henderson,
	QEMU Developers, Stefan Hajnoczi, Alex Bennée

On 8/22/19 7:05 PM, Paolo Bonzini wrote:
> On 22/08/19 18:50, Dr. David Alan Gilbert wrote:
>> * Paolo Bonzini (pbonzini@redhat.com) wrote:
>>> On 22/08/19 18:31, Dr. David Alan Gilbert wrote:
>>>>> With both these points in mind, I think it is  pretty hard sell to
>>>>> say we should write & maintain a custom CI system just for QEMU
>>>>> unless it is offering major compelling functionality we can't do
>>>>> without.
>>
>> (That was Dan's comment)
>>
>>> In theory I agree.
>>>
>>> In practice, the major compelling functionality is portability.  If it
>>> is true that setting up runners is problematic even on aarch64, frankly
>>> GitLab CI is dead on arrival.  If it is not true, then I'd be very happy
>>> to use GitLab CI too.
>>
>> IMHO if for some weird reason Gitlab has problems on aarch64 then we
>> just need to get that fixed.
> 
> I'm sure it's just some packaging or deployment issue.  But
> https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/725 has been
> open for more than one year; the last two messages are:
> 
> * 1 month ago: "I hope we will be able to merge it soon"
> 
> * 3 weeks ago: "Today I tried use gitlab-runner on my arm64 box, however
> it kept mysteriously failing"
> 
> So the question is simply who does the work.

IIRC Samuel Ortiz told he was using GitLab with Aarch64 runners around
Nov 2018, but "compiling from source". Alex Bennée tried building it on
our Packet server during early 2019.
Later an (unattended?) Ubuntu upgrade installed a package that does not
work anymore with current GitLab server. I noticed this few months ago,
built it again and tested it, then looked at what was wrong with the
upstream MR. The Aarch64 packaging succeed when cross-building on x86_64
host, but fails when building natively... Since part of it is "built or
tested in the cloud" and involving Go, I simply let a comment:
https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/725#note_183470145

So to confirm what Paolo said, GitLab runners work on Aarch64
(and we have it well tested), however there is a packaging issue,
so it does not work "out of the box".


Related to:

  Runner compiled with Go 1.8.7 seems to not work properly with
  multiarch support. Executing the binary built with Go 1.8.7
  results with an error [...]

There has been 1 recent fix for the go runner:
https://bugs.launchpad.net/qemu/+bug/1838946/comments/1

And there is an ongoing discussion about "patch to swap SIGRTMIN + 1
and SIGRTMAX - 1".


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [Qemu-devel] more automated/public CI for QEMU pullreqs
  2019-08-23 13:15           ` Philippe Mathieu-Daudé
@ 2019-08-24  7:44             ` Alex Bennée
  0 siblings, 0 replies; 14+ messages in thread
From: Alex Bennée @ 2019-08-24  7:44 UTC (permalink / raw)
  To: Philippe Mathieu-Daudé
  Cc: Peter Maydell, Daniel P. Berrangé,
	Samuel Ortiz, Kashyap Chamarthy, Richard Henderson,
	Dr. David Alan Gilbert, QEMU Developers, Stefan Hajnoczi,
	Paolo Bonzini


Philippe Mathieu-Daudé <philmd@redhat.com> writes:

> On 8/22/19 7:05 PM, Paolo Bonzini wrote:
>> On 22/08/19 18:50, Dr. David Alan Gilbert wrote:
>>> * Paolo Bonzini (pbonzini@redhat.com) wrote:
>>>> On 22/08/19 18:31, Dr. David Alan Gilbert wrote:
>>>>>> With both these points in mind, I think it is  pretty hard sell to
>>>>>> say we should write & maintain a custom CI system just for QEMU
>>>>>> unless it is offering major compelling functionality we can't do
>>>>>> without.
>>>
>>> (That was Dan's comment)
>>>
>>>> In theory I agree.
>>>>
>>>> In practice, the major compelling functionality is portability.  If it
>>>> is true that setting up runners is problematic even on aarch64, frankly
>>>> GitLab CI is dead on arrival.  If it is not true, then I'd be very happy
>>>> to use GitLab CI too.
>>>
>>> IMHO if for some weird reason Gitlab has problems on aarch64 then we
>>> just need to get that fixed.
>>
>> I'm sure it's just some packaging or deployment issue.  But
>> https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/725 has been
>> open for more than one year; the last two messages are:
>>
>> * 1 month ago: "I hope we will be able to merge it soon"
>>
>> * 3 weeks ago: "Today I tried use gitlab-runner on my arm64 box, however
>> it kept mysteriously failing"
>>
>> So the question is simply who does the work.
>
> IIRC Samuel Ortiz told he was using GitLab with Aarch64 runners around
> Nov 2018, but "compiling from source". Alex Bennée tried building it on
> our Packet server during early 2019.
> Later an (unattended?) Ubuntu upgrade installed a package that does not
> work anymore with current GitLab server. I noticed this few months ago,
> built it again and tested it, then looked at what was wrong with the
> upstream MR. The Aarch64 packaging succeed when cross-building on x86_64
> host, but fails when building natively... Since part of it is "built or
> tested in the cloud" and involving Go, I simply let a comment:
> https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/725#note_183470145

I need to have another look at this but from memory it is because the
"cross-build" approach they use is to try and build all arches with a
qemu-user controlled docker build. However if the host architecture is
the target you are aiming for that should run normally - however it
failed on qemu-test because you can't build the armhf binaries there as
not all AArch64 machines support AArch32. There is a bug in the Debian
binfmt_misc scripts that I raised but needs proper fixing.

I think the easiest thing to do would be to document how to build
exactly 1 architecture at a time so you don't have to succeed in
building everything at once and can build something natively without
jumping through hoops.

>
> So to confirm what Paolo said, GitLab runners work on Aarch64
> (and we have it well tested), however there is a packaging issue,
> so it does not work "out of the box".
>
>
> Related to:
>
>   Runner compiled with Go 1.8.7 seems to not work properly with
>   multiarch support. Executing the binary built with Go 1.8.7
>   results with an error [...]
>
> There has been 1 recent fix for the go runner:
> https://bugs.launchpad.net/qemu/+bug/1838946/comments/1
>
> And there is an ongoing discussion about "patch to swap SIGRTMIN + 1
> and SIGRTMAX - 1".

Much as I champion qemu-user for solving build problems I think being
able to build natively should be the quicker and easier fix (not that we
shouldn't fix qemu-user bugs as well).

--
Alex Bennée


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: more automated/public CI for QEMU pullreqs
  2019-08-22 10:49 ` Stefan Hajnoczi
  2019-08-22 11:04   ` Peter Maydell
@ 2019-09-30 12:53   ` Peter Maydell
  2019-10-04  9:31     ` Stefan Hajnoczi
  1 sibling, 1 reply; 14+ messages in thread
From: Peter Maydell @ 2019-09-30 12:53 UTC (permalink / raw)
  To: Stefan Hajnoczi
  Cc: Samuel Ortiz, Kashyap Chamarthy, Philippe Mathieu-Daudé,
	Richard Henderson, QEMU Developers, Paolo Bonzini,
	Alex Bennée

On Thu, 22 Aug 2019 at 11:50, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> Thanks for writing up this summary!  The requirements and possible
> solutions are now clear.
>
> We need someone willing to set up and maintain the CI.
>
> One-off tasks:
>
> 1. Create CI runners that offer similar cross-architecture coverage to
>    Peter's current setup.  qemu.org has some x86, ppc, and s390 server
>    resources available.  I'm not sure about ARM and other architectures.
>
> 2. Write CI configuration to run Peter's "make && make check && make
>    check-tcg && linux-user-test".
>
> 3. Document the CI on wiki.qemu.org.
>
> Ongoing responsibilities:
>
> 1. Triage failures that the qemu.git maintainer thinks are related to CI
>    runners.
>
> 2. Keep the CI up-to-date and runners online.
>
> Any volunteers?

I see this call for volunteers didn't attract any offers :-/

Thread consensus seems to be toward using gitlab CI -- assuming
we can hand-build the gitlab runner for the architectures
we need without having to block on gitlab fixing their weird
"build for all architectures at once" packaging ?

thanks
-- PMM


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: more automated/public CI for QEMU pullreqs
  2019-09-30 12:53   ` Peter Maydell
@ 2019-10-04  9:31     ` Stefan Hajnoczi
  0 siblings, 0 replies; 14+ messages in thread
From: Stefan Hajnoczi @ 2019-10-04  9:31 UTC (permalink / raw)
  To: Peter Maydell
  Cc: Samuel Ortiz, Kashyap Chamarthy, Philippe Mathieu-Daudé,
	Richard Henderson, QEMU Developers, Paolo Bonzini,
	Alex Bennée

[-- Attachment #1: Type: text/plain, Size: 1446 bytes --]

On Mon, Sep 30, 2019 at 01:53:16PM +0100, Peter Maydell wrote:
> On Thu, 22 Aug 2019 at 11:50, Stefan Hajnoczi <stefanha@redhat.com> wrote:
> > Thanks for writing up this summary!  The requirements and possible
> > solutions are now clear.
> >
> > We need someone willing to set up and maintain the CI.
> >
> > One-off tasks:
> >
> > 1. Create CI runners that offer similar cross-architecture coverage to
> >    Peter's current setup.  qemu.org has some x86, ppc, and s390 server
> >    resources available.  I'm not sure about ARM and other architectures.
> >
> > 2. Write CI configuration to run Peter's "make && make check && make
> >    check-tcg && linux-user-test".
> >
> > 3. Document the CI on wiki.qemu.org.
> >
> > Ongoing responsibilities:
> >
> > 1. Triage failures that the qemu.git maintainer thinks are related to CI
> >    runners.
> >
> > 2. Keep the CI up-to-date and runners online.
> >
> > Any volunteers?
> 
> I see this call for volunteers didn't attract any offers :-/
> 
> Thread consensus seems to be toward using gitlab CI -- assuming
> we can hand-build the gitlab runner for the architectures
> we need without having to block on gitlab fixing their weird
> "build for all architectures at once" packaging ?

Let's tackle the issue of CI at KVM Forum.  The main issue is to get
people.

I will talk to colleagues beforehand to see if anyone is available for
this role.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2019-10-04  9:32 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-08-16 18:16 [Qemu-devel] more automated/public CI for QEMU pullreqs Peter Maydell
2019-08-22 10:49 ` Stefan Hajnoczi
2019-08-22 11:04   ` Peter Maydell
2019-08-22 11:14     ` Paolo Bonzini
2019-09-30 12:53   ` Peter Maydell
2019-10-04  9:31     ` Stefan Hajnoczi
2019-08-22 11:47 ` [Qemu-devel] " Daniel P. Berrangé
2019-08-22 16:31   ` Dr. David Alan Gilbert
2019-08-22 16:48     ` Paolo Bonzini
2019-08-22 16:50       ` Dr. David Alan Gilbert
2019-08-22 17:05         ` Paolo Bonzini
2019-08-23 13:15           ` Philippe Mathieu-Daudé
2019-08-24  7:44             ` Alex Bennée
2019-08-22 17:03       ` Daniel P. Berrangé

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).