On Fri, Feb 07, 2020 at 04:26:53PM +0000, Peter Maydell wrote:
> On Fri, 7 Feb 2020 at 08:37, Thomas Huth <thuth@redhat.com> wrote:
> > Question to Peter/Alex/Stefan/Howevermergespullreqsinthefuture:
> >
> > Should the above jobs really be skipped for pull requests, or would it
> > be ok to include them there, too? (in the latter case, the above changes
> > could just be dropped)
> 
> I don't mind, as long as the CI run doesn't take more than (say)
> 1h to 1h30 elapsed time to complete from kicking it off to getting
> all the results back. The specific set of x86 configs tested don't
> really worry me (as long as we do have a reasonable spread):
> the thing I really care about is that we get the multiple
> host architectures and the BSDs into the test setup. (We already
> have about five different ways of doing CI testing of x86 Linux
> hosts, which is the least likely setup to break. It's the
> other host configs that I'm really keen to see progress on
> automation of, because that's what's really blocking us from
> being able to move off my hand-coded scripting.)
>

I can imagine how important the average runtime of those checks is to
Howevermergespullreqsnoworinthefuture.  Now, being a bit more selfish
or paranoid, I see the importance of separating those different types
of checks (pre-PR and others) until we achieve a stable environment
for the former.  I do see an extended amount of testing at every
stage as one of the later goals of this effort, but we have to match
that with both computing and human capacity.

In short, I'd suggest keeping them separate *for now*.

> In the long run we should probably aim for being consistent about
> what we test between the pull-request tests and whatever the
> 'public-facing CI' part is.
>

Right.  I think this will be an exercise in capacity planning, and
ideally, if there are no resource constraints, all checks could be
performed at all times.

> > > +ubuntu-18.04.3-x86_64-notcg:
> > > + tags:
> > > + - ubuntu_18.04.3
> > > + - x86_64
> > > + rules:
> > > + - if: '$CI_COMMIT_REF_NAME == "staging"'
> > > + script:
> > > + # https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/remake-merge-builds#n35
> > > + - ./configure --disable-libssh --enable-debug --disable-tcg
> > > + # https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/pull-buildtest#n35
> > > + - make
> > > + # https://git.linaro.org/people/peter.maydell/misc-scripts.git/tree/pull-buildtest#n39
> > > + # Question: check is disabled on the original script, because the machine
> > > + # is said to be running VirtualBox.  Should this be dropped or should the
> > > + # machine be tweaked or substituted?
> > > + - make check V=1
> >
> > Without TCG, you definitely need a host that can do KVM for running make
> > check.
> > Question for Peter: Would it be ok to drop this job and simply always
> > use the "build-tcg-disabled" job that is already available in
> > .gitlab-ci.yml ?
> 
> If we have a CI setup where KVM reliably works then we should
> ideally test a --disable-tcg setup somehow. Right now my pullreq
> tests don't test that because I run them on my work desktop box
> and (as the config says) sometimes I'm running VirtualBox which
> causes KVM to fail -- but that should be irrelevant to our CI
> runners...
>

You got me confused here Peter.  Do you intend to reuse the same
machines you're using now (say your work desktop box) or is there an
expectation for different machines to be introduced and used for these
jobs?

> thanks
> -- PMM
> 

Thanks for the feedback,
- Cleber.