QEMU-Devel Archive on lore.kernel.org
 help / color / Atom feed
From: Stefan Hajnoczi <stefanha@redhat.com>
To: Peter Maydell <peter.maydell@linaro.org>
Cc: "Samuel Ortiz" <sameo@linux.intel.com>,
	"Kashyap Chamarthy" <kchamart@redhat.com>,
	"Philippe Mathieu-Daudé" <philmd@redhat.com>,
	"Richard Henderson" <richard.henderson@linaro.org>,
	"QEMU Developers" <qemu-devel@nongnu.org>,
	"Paolo Bonzini" <pbonzini@redhat.com>,
	"Alex Bennée" <alex.bennee@linaro.org>
Subject: Re: [Qemu-devel] more automated/public CI for QEMU pullreqs
Date: Thu, 22 Aug 2019 11:49:57 +0100
Message-ID: <20190822104957.GE20491@stefanha-x1.localdomain> (raw)
In-Reply-To: <CAFEAcA8kEKVcRu62+VGDkzRj2J87QPxzjg05dCHszeBC6X76pg@mail.gmail.com>

[-- Attachment #1: Type: text/plain, Size: 5107 bytes --]

On Fri, Aug 16, 2019 at 07:16:55PM +0100, Peter Maydell wrote:
> We had a conversation some months back about ways we might switch
> away from the current handling of pull requests which I do via some
> hand-curated scripts and personal access to machines, to a more
> automated system that could be operated by a wider range of people.
> Unfortunately that conversation took place off-list (largely my fault
> for forgetting a cc: at the beginning of the email chain), and in any
> case it sort of fizzled out.  So let's restart it, on the mailing
> list this time.
> 
> Here's a summary of stuff from the old thread and general
> explanation of the problem:
> 
> My current setup is mostly just running the equivalent of
> "make && make check" on a bunch of machines and configs
> on the merge commit before I push it to master. I also do
> a 'make check-tcg' on one of the builds and run a variant
> of the 'linux-user-test' tarball of 'ls' binaries.
> The scripts do some simple initial checks which mostly are
> preventing problems seen in the past:
>  * don't allow submodules to be updated unless I kick the
>    merge off with a command line option saying submodule updates
>    are OK here (this catches accidental misdriving of git by
>    a submaintainer folding a submodule change into a patch
>    during a rebase)
>  * check we aren't trying to merge after tagging the final
>    release but before doing the 'reopen development tree'
>    commit that bumps the VERSION file
>  * check for bogus "author is qemu-devel@nongnu.org" commits
>  * check for UTF-8 mangling
>  * check the gpg signature on the pullreq
> A human needs to also eyeball the commits and the diffstat
> for weird stuff (usually cursory for frequent pullreq submitters,
> and more carefully for new submaintainers).
> 
> I have this semi-automated with some hacky scripts.  The major thing we
> need for a replacement is the coverage of different host
> architectures and operating systems, which is a poor match to most of
> the cloud-CI services out there (including Travis).  We also want the
> tests to run in a reasonably short wall-clock time from being kicked
> off.
> 
> Awkward bonus extra requirement: it would be useful to be
> able to do a merge CI run "privately", eg because the thing
> being tested is a fix for a security bug that's not yet
> public. But that's rare so we can probably do it by hand.
> 
> There are some other parts to this, like getting some kind of
> project-role-account access to machines where that's OK, or finding
> replacements where the machines really are my personal ones or
> otherwise not OK for project access.  But I think that should be
> fairly easy to resolve so let's keep this thread to the
> automating-the-CI part.
> 
> The two major contenders suggested were:
> 
> (1) GitLab CI, which supports custom 'runners' which we can set
> up to run builds and tests on machines we have project access to
> 
> (2) Patchew, which can handle running tests on multiple machines (eg
> we do s390 testing today for all patches on list), and which we could
> enhance to provide support for the release-manager to do their work
> 
> Advantages of Gitlab CI:
>  * somebody else is doing the development and maintainance of the
>    CI tool -- bigger 'bus factor' than patchew
>  * already does (more or less) what we want without needing
>    extra coding work
> 
> Advantages of Patchew:
>  * we're already using it for patch submissions, so we know it's
>    not going to go away
>  * it's very easy to deploy to a new host
>  * no dependencies except Python, so works anywhere we expect
>    to be able to build QEMU (whereas gitlab CI's runner is
>    written in Go, and there seem to be ongoing issues with getting
>    it actually to compile for other architectures than x86)
> 
> I don't have an opinion really, but I think it would be good to
> make a choice and start working forwards towards getting this
> a bit less hacky and a bit more offloadable to other people.
> 
> Perhaps a good first step would be to keep the 'simple checks
> of broken commits' part done as a local script but have the
> CI done via "push proposed merge commit to $SOMEWHERE to
> kick off the CI".
> 
> Input, opinions, recommendations, offers to do some of the work? :-)

Thanks for writing up this summary!  The requirements and possible
solutions are now clear.

We need someone willing to set up and maintain the CI.

One-off tasks:

1. Create CI runners that offer similar cross-architecture coverage to
   Peter's current setup.  qemu.org has some x86, ppc, and s390 server
   resources available.  I'm not sure about ARM and other architectures.

2. Write CI configuration to run Peter's "make && make check && make
   check-tcg && linux-user-test".

3. Document the CI on wiki.qemu.org.

Ongoing responsibilities:

1. Triage failures that the qemu.git maintainer thinks are related to CI
   runners.

2. Keep the CI up-to-date and runners online.

Any volunteers?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

  reply index

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-08-16 18:16 Peter Maydell
2019-08-22 10:49 ` Stefan Hajnoczi [this message]
2019-08-22 11:04   ` Peter Maydell
2019-08-22 11:14     ` Paolo Bonzini
2019-09-30 12:53   ` Peter Maydell
2019-10-04  9:31     ` Stefan Hajnoczi
2019-08-22 11:47 ` [Qemu-devel] " Daniel P. Berrangé
2019-08-22 16:31   ` Dr. David Alan Gilbert
2019-08-22 16:48     ` Paolo Bonzini
2019-08-22 16:50       ` Dr. David Alan Gilbert
2019-08-22 17:05         ` Paolo Bonzini
2019-08-23 13:15           ` Philippe Mathieu-Daudé
2019-08-24  7:44             ` Alex Bennée
2019-08-22 17:03       ` Daniel P. Berrangé

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20190822104957.GE20491@stefanha-x1.localdomain \
    --to=stefanha@redhat.com \
    --cc=alex.bennee@linaro.org \
    --cc=kchamart@redhat.com \
    --cc=pbonzini@redhat.com \
    --cc=peter.maydell@linaro.org \
    --cc=philmd@redhat.com \
    --cc=qemu-devel@nongnu.org \
    --cc=richard.henderson@linaro.org \
    --cc=sameo@linux.intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

QEMU-Devel Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/qemu-devel/0 qemu-devel/git/0.git
	git clone --mirror https://lore.kernel.org/qemu-devel/1 qemu-devel/git/1.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 qemu-devel qemu-devel/ https://lore.kernel.org/qemu-devel \
		qemu-devel@nongnu.org qemu-devel@archiver.kernel.org
	public-inbox-index qemu-devel

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.nongnu.qemu-devel


AGPL code for this site: git clone https://public-inbox.org/ public-inbox