[Qemu-devel] Who is running QEMU automated tests, and when?

[Qemu-devel] Who is running QEMU automated tests, and when?

[Eduardo Habkost]

(Starting a new thread, for more visibility)

(This was: Re: [Qemu-devel] [RFC PATCH] tests/device-introspect: Test
devices with all machines, not only with "none")

On Thu, Apr 26, 2018 at 01:54:43PM +0200, Markus Armbruster wrote:
> Thomas Huth <thuth@redhat.com> writes:
> > On 17.04.2018 14:12, Markus Armbruster wrote:
> >> Thomas Huth <thuth@redhat.com> writes:
> >> 
> >>> Many device introspection crashes only happen if you are using a
> >>> certain machine, e.g.:
> >>>
> >>> $ ppc-softmmu/qemu-system-ppc -S -M ref405ep,accel=qtest -qmp stdio
> >>> {"QMP": {"version": {"qemu": {"micro": 50, "minor": 11, "major": 2},
> >>>  "package": "build-all"}, "capabilities": []}}
> >>> { 'execute': 'qmp_capabilities' }
> >>> {"return": {}}
> >>> { 'execute': 'device-list-properties',
> >>>   'arguments': {'typename': 'macio-newworld'}}
> >>> Unexpected error in qemu_chr_fe_init() at chardev/char-fe.c:222:
> >>> Device 'serial0' is in use
> >>> Aborted (core dumped)
> >>>
> >>> To be able to catch these problems, let's extend the device-introspect
> >>> test to check the devices on all machine types. Since this is a rather
> >>> slow operation, the test is only run in "SPEED=slow" mode.
> >> 
> >> If the device works with one machine type, it has a decent chance to
> >> work with others, too.  Thus, testing each device with every machine
> >> type is overkill.  I appreciate having overkill as an option :)
> >> 
> >> What I'd like to see for a quick "make check" is testing each device
> >> once.  That should flush out most bugs.  
> >
> > That's already done with the "none" machine.
> 
> I was too terse.  We test each device with -machine none for every
> target.  Fine if that's quick enough.  If not, we might want to reduce
> redundancy there.
> 
> Actually, a worse offender in the "waste everybody's time via redunancy"
> department could be qom-test.
> 
> > Anyway, do you think my patch here is useful and has a chance of getting
> > included? I.e. shall I re-spin this as a non-RFC patch? Or shall we
> > rather wait for Eduardo's python-based tests to get included into the
> > repository?
> 
> I don't mind having make check SPEED=slow run more extensive tests.
> Assuming we actually run them at least once in a while, which seems
> doubtful.

We probably don't do that, but we really must be running a more
extensive (and slower) test set at least once before every
release.

Maybe some people are running SPEED=slow tests, or even more
extensive test suites like avocado-vt once in a while, but we
need to know who is running them, and when.

Today, the only test set I know people really run and would
surely block a release is "make check [SPEED=quick]".

So, for anybody that runs automated QEMU tests once in a while,
can we know:

* What test cases are you running?  Where can we get more
  information about the tests you run?
* When do you run them?  What triggers a new test run?

Peter, do you have additional tests you run before merging a pull
request?  Additional test sets run before tagging a release?

-- 
Eduardo

Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[Peter Maydell]

On 26 April 2018 at 14:57, Eduardo Habkost <ehabkost@redhat.com> wrote:
> Peter, do you have additional tests you run before merging a pull
> request?  Additional test sets run before tagging a release?

I run make && make check, on a variety of hosts, before merging
any pull request. That's the only testing I do, and I don't
do anything more before I do a tag.

thanks
-- PMm

Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[Michael Clark]

On Fri, Apr 27, 2018 at 2:04 AM, Peter Maydell <peter.maydell@linaro.org>
wrote:

> On 26 April 2018 at 14:57, Eduardo Habkost <ehabkost@redhat.com> wrote:
> > Peter, do you have additional tests you run before merging a pull
> > request?  Additional test sets run before tagging a release?
>
> I run make && make check, on a variety of hosts, before merging
> any pull request. That's the only testing I do, and I don't
> do anything more before I do a tag.
>

This is what I do on riscv, however it is not yet automated:

- QEMU upstream `make check` tests are run in Travis and they should all
pass.
- The RISC-V ISA compatibility tests from the `riscv-tests` repository are
  run and currently all of the tests pass.
- Fedora Linux and Debian Linux are used for running self-hosted GCC
bootstraps
  using the latest Linux kernel and GLIBC user-space in the `virt` machine.
  A parallel GCC bootstrap is used for stress testing MTTCG, SMP and the
  interrupt controllers in a multi-core RISC-V virtual machine.
- Linux 4.6.2 with bbl and busybox are used for backward compatibility
testing
  in the `spike_v1.9.1` machine.
- Linux 4.15.0 with bbl and busybox are used for forward compatibility
testing
  in the `spike_v1.10` machine.
- HiFive1 binaries from the `freedom-u-sdk` are used for testing the
  `sifive_e` machine.

I only run the 24-hour type tests if I am changing anything related to
interrupts. We are working on interrupt controller emulation and the SMP
PLIC used by the 'virt' machine was authored by me, however Stefan O'Rear
contributed to the interrupt handling code and I've been very careful to
test that I haven't broken and to fully understand the logic and atomicity
of interrupts. We are working on a new interrupt controller and are
specifically looking at the possibility of adding a GPIO mock that could be
combined with some sort of replay mechanism to test the interrupt
controllers.

We could bring a snapshot of riscv-tests into the QEMU tree however it
would be best if they retained the BSD license so we can share locally
added tests with upstream as upstream is adding tests periodically.

We have recently made changes to the riscv page walker where we have
documented it in such a way that each clause represents a descriprition for
a test case. Our goal is that we would like the code to read similarly to
the RISC-V Privileged ISA specification where possible.

-
https://github.com/riscv/riscv-qemu/blob/qemu-2.13-for-upstream/target/riscv/cpu_helper.c#L210-L243

BTW Andrew Waterman reviewed my update to the page walker and caught a bug.
Linux testing doesn't exercise every dimension of the page permission
checks. We effectively need malware type tests. Multi-core PTE update
atomicity tests will also be interesting. The SiFive verification team have
been working on PMP tests however I am not sure whether these will be made
open source. The RISC-V QEMU PMP code needs testing. It also needs to
implement recent changes to the specification for coarse grain PMPs (4K
granularity) given we use the TLB page size so that PMP can be implemented
with reasonable performance.

We would like to automate all of this and we are working towards that goal.
One of the dependencies for docker tests are the riscv toolchain packages
and we are working on this...

Adding port specific tests, like riscv tests to QEMU not only benefits the
particular port but also exercises a log of generic code such as the TCG,
the IO system, IO mutex, Virt IO and many other shared code paths.

Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[Alex Bennée]

Michael Clark <mjc@sifive.com> writes:

> On Fri, Apr 27, 2018 at 2:04 AM, Peter Maydell <peter.maydell@linaro.org>
> wrote:
>
>> On 26 April 2018 at 14:57, Eduardo Habkost <ehabkost@redhat.com> wrote:
>> > Peter, do you have additional tests you run before merging a pull
>> > request?  Additional test sets run before tagging a release?
>>
>> I run make && make check, on a variety of hosts, before merging
>> any pull request. That's the only testing I do, and I don't
>> do anything more before I do a tag.
>>
>
> This is what I do on riscv, however it is not yet automated:
>
<snip>
>
> We could bring a snapshot of riscv-tests into the QEMU tree however it
> would be best if they retained the BSD license so we can share locally
> added tests with upstream as upstream is adding tests periodically.

Certainly it would be worth considering some smoke tests for
tests/tcg/riscv that at least verify the basics CPU features work as
expected.

I've got some ideas for packaging more extensive test suites like LTP
and kvm-unit-tests but would like to get simple linux-user stuff up and
running first.

<snip>
>
> We would like to automate all of this and we are working towards that goal.
> One of the dependencies for docker tests are the riscv toolchain packages
> and we are working on this...

I got riscv64 working with the SID toolchain:

  https://github.com/stsquad/qemu/commit/db0a108b9a0efdeeb2f63e670595c5e28e236a97

Once I've fixed it up to use snapshots.debian.org it will be in v4 of
the TCG tests.

> Adding port specific tests, like riscv tests to QEMU not only benefits the
> particular port but also exercises a log of generic code such as the TCG,
> the IO system, IO mutex, Virt IO and many other shared code paths.

I agree it would be worthwhile testing.

System tests are a bit more complicated in terms of code required
although we have some for CRIS/MIPS/Xtensa/lm32 which need re-enabling -
although toolchains for the more esoteric architectures is a pain.

--
Alex Bennée

Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[Thomas Huth]

On 09.05.2018 17:53, Alex Bennée wrote:
> 
> Michael Clark <mjc@sifive.com> writes:
> 
>> On Fri, Apr 27, 2018 at 2:04 AM, Peter Maydell <peter.maydell@linaro.org>
>> wrote:
>>
>>> On 26 April 2018 at 14:57, Eduardo Habkost <ehabkost@redhat.com> wrote:
>>>> Peter, do you have additional tests you run before merging a pull
>>>> request?  Additional test sets run before tagging a release?
>>>
>>> I run make && make check, on a variety of hosts, before merging
>>> any pull request. That's the only testing I do, and I don't
>>> do anything more before I do a tag.
>>>
>>
>> This is what I do on riscv, however it is not yet automated:
>>
> <snip>
>>
>> We could bring a snapshot of riscv-tests into the QEMU tree however it
>> would be best if they retained the BSD license so we can share locally
>> added tests with upstream as upstream is adding tests periodically.
> 
> Certainly it would be worth considering some smoke tests for
> tests/tcg/riscv that at least verify the basics CPU features work as
> expected.

 Hi Michael,

has one of the riscv machine an UART that can be programmed with some
few lines of assembler? In that case we could extend
tests/boot-serial-test.c to also check riscv. That's just a very, very
simple test, but it's better than nothing and indeed helped to find a
couple of bugs in the past already.

 Thomas

Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[Thomas Huth]

On 26.04.2018 15:57, Eduardo Habkost wrote:
> (Starting a new thread, for more visibility)
> 
> (This was: Re: [Qemu-devel] [RFC PATCH] tests/device-introspect: Test
> devices with all machines, not only with "none")
> 
> On Thu, Apr 26, 2018 at 01:54:43PM +0200, Markus Armbruster wrote:
[...]
>> I don't mind having make check SPEED=slow run more extensive tests.
>> Assuming we actually run them at least once in a while, which seems
>> doubtful.
> 
> We probably don't do that, but we really must be running a more
> extensive (and slower) test set at least once before every
> release.
> 
> Maybe some people are running SPEED=slow tests, or even more
> extensive test suites like avocado-vt once in a while, but we
> need to know who is running them, and when.

At least I am running "make check SPEED=slow" manually from time to
time, especially when we enter the hard freeze period.

> Today, the only test set I know people really run and would
> surely block a release is "make check [SPEED=quick]".

Same is true for the iotests, though they are run a little bit more
often than "check SPEED=slow", I think.

 Thomas


> So, for anybody that runs automated QEMU tests once in a while,
> can we know:
> 
> * What test cases are you running?  Where can we get more
>   information about the tests you run?
> * When do you run them?  What triggers a new test run?
> 
> Peter, do you have additional tests you run before merging a pull
> request?  Additional test sets run before tagging a release?
> 

Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[Daniel P. Berrangé]

On Thu, Apr 26, 2018 at 04:09:55PM +0200, Thomas Huth wrote:
> On 26.04.2018 15:57, Eduardo Habkost wrote:
> > (Starting a new thread, for more visibility)
> > 
> > (This was: Re: [Qemu-devel] [RFC PATCH] tests/device-introspect: Test
> > devices with all machines, not only with "none")
> > 
> > On Thu, Apr 26, 2018 at 01:54:43PM +0200, Markus Armbruster wrote:
> [...]
> >> I don't mind having make check SPEED=slow run more extensive tests.
> >> Assuming we actually run them at least once in a while, which seems
> >> doubtful.
> > 
> > We probably don't do that, but we really must be running a more
> > extensive (and slower) test set at least once before every
> > release.
> > 
> > Maybe some people are running SPEED=slow tests, or even more
> > extensive test suites like avocado-vt once in a while, but we
> > need to know who is running them, and when.
> 
> At least I am running "make check SPEED=slow" manually from time to
> time, especially when we enter the hard freeze period.

Hmm, we could get this done by travis. It has the concept of "cron jobs"
for scheduling builds separately from pushes.

So we could keep the current travis jobs unchanged, but then add an
use of SPEED=slow when TRAVIS_EVENT_TYPE == "cron" in the travis.yml,
so we can get SPEED=slow run once a day. Just have to be careful which
jobs we make slow so we don't hit the 50 minute timeout.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[Eduardo Habkost]

On Thu, Apr 26, 2018 at 03:14:00PM +0100, Daniel P. Berrangé wrote:
> On Thu, Apr 26, 2018 at 04:09:55PM +0200, Thomas Huth wrote:
> > On 26.04.2018 15:57, Eduardo Habkost wrote:
> > > (Starting a new thread, for more visibility)
> > > 
> > > (This was: Re: [Qemu-devel] [RFC PATCH] tests/device-introspect: Test
> > > devices with all machines, not only with "none")
> > > 
> > > On Thu, Apr 26, 2018 at 01:54:43PM +0200, Markus Armbruster wrote:
> > [...]
> > >> I don't mind having make check SPEED=slow run more extensive tests.
> > >> Assuming we actually run them at least once in a while, which seems
> > >> doubtful.
> > > 
> > > We probably don't do that, but we really must be running a more
> > > extensive (and slower) test set at least once before every
> > > release.
> > > 
> > > Maybe some people are running SPEED=slow tests, or even more
> > > extensive test suites like avocado-vt once in a while, but we
> > > need to know who is running them, and when.
> > 
> > At least I am running "make check SPEED=slow" manually from time to
> > time, especially when we enter the hard freeze period.
> 
> Hmm, we could get this done by travis. It has the concept of "cron jobs"
> for scheduling builds separately from pushes.
> 
> So we could keep the current travis jobs unchanged, but then add an
> use of SPEED=slow when TRAVIS_EVENT_TYPE == "cron" in the travis.yml,
> so we can get SPEED=slow run once a day. Just have to be careful which
> jobs we make slow so we don't hit the 50 minute timeout.

I expect SPEED=slow tests to eventually take longer than 50
minutes, but we can try this and see what happens.

-- 
Eduardo

Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[Daniel P. Berrangé]

On Thu, Apr 26, 2018 at 11:22:36AM -0300, Eduardo Habkost wrote:
> On Thu, Apr 26, 2018 at 03:14:00PM +0100, Daniel P. Berrangé wrote:
> > On Thu, Apr 26, 2018 at 04:09:55PM +0200, Thomas Huth wrote:
> > > On 26.04.2018 15:57, Eduardo Habkost wrote:
> > > > (Starting a new thread, for more visibility)
> > > > 
> > > > (This was: Re: [Qemu-devel] [RFC PATCH] tests/device-introspect: Test
> > > > devices with all machines, not only with "none")
> > > > 
> > > > On Thu, Apr 26, 2018 at 01:54:43PM +0200, Markus Armbruster wrote:
> > > [...]
> > > >> I don't mind having make check SPEED=slow run more extensive tests.
> > > >> Assuming we actually run them at least once in a while, which seems
> > > >> doubtful.
> > > > 
> > > > We probably don't do that, but we really must be running a more
> > > > extensive (and slower) test set at least once before every
> > > > release.
> > > > 
> > > > Maybe some people are running SPEED=slow tests, or even more
> > > > extensive test suites like avocado-vt once in a while, but we
> > > > need to know who is running them, and when.
> > > 
> > > At least I am running "make check SPEED=slow" manually from time to
> > > time, especially when we enter the hard freeze period.
> > 
> > Hmm, we could get this done by travis. It has the concept of "cron jobs"
> > for scheduling builds separately from pushes.
> > 
> > So we could keep the current travis jobs unchanged, but then add an
> > use of SPEED=slow when TRAVIS_EVENT_TYPE == "cron" in the travis.yml,
> > so we can get SPEED=slow run once a day. Just have to be careful which
> > jobs we make slow so we don't hit the 50 minute timeout.
> 
> I expect SPEED=slow tests to eventually take longer than 50
> minutes, but we can try this and see what happens.

We can run many jobs though so if qtests take longer than 50 mins, we could
create 1 job per target arch.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[Thomas Huth]

On 26.04.2018 16:24, Daniel P. Berrangé wrote:
> On Thu, Apr 26, 2018 at 11:22:36AM -0300, Eduardo Habkost wrote:
>> On Thu, Apr 26, 2018 at 03:14:00PM +0100, Daniel P. Berrangé wrote:
>>> On Thu, Apr 26, 2018 at 04:09:55PM +0200, Thomas Huth wrote:
>>>> On 26.04.2018 15:57, Eduardo Habkost wrote:
>>>>> (Starting a new thread, for more visibility)
>>>>>
>>>>> (This was: Re: [Qemu-devel] [RFC PATCH] tests/device-introspect: Test
>>>>> devices with all machines, not only with "none")
>>>>>
>>>>> On Thu, Apr 26, 2018 at 01:54:43PM +0200, Markus Armbruster wrote:
>>>> [...]
>>>>>> I don't mind having make check SPEED=slow run more extensive tests.
>>>>>> Assuming we actually run them at least once in a while, which seems
>>>>>> doubtful.
>>>>>
>>>>> We probably don't do that, but we really must be running a more
>>>>> extensive (and slower) test set at least once before every
>>>>> release.
>>>>>
>>>>> Maybe some people are running SPEED=slow tests, or even more
>>>>> extensive test suites like avocado-vt once in a while, but we
>>>>> need to know who is running them, and when.
>>>>
>>>> At least I am running "make check SPEED=slow" manually from time to
>>>> time, especially when we enter the hard freeze period.
>>>
>>> Hmm, we could get this done by travis. It has the concept of "cron jobs"
>>> for scheduling builds separately from pushes.
>>>
>>> So we could keep the current travis jobs unchanged, but then add an
>>> use of SPEED=slow when TRAVIS_EVENT_TYPE == "cron" in the travis.yml,
>>> so we can get SPEED=slow run once a day. Just have to be careful which
>>> jobs we make slow so we don't hit the 50 minute timeout.
>>
>> I expect SPEED=slow tests to eventually take longer than 50
>> minutes, but we can try this and see what happens.
> 
> We can run many jobs though so if qtests take longer than 50 mins, we could
> create 1 job per target arch.

That could work... individual "make check-qtest-xxx" should be short
enough, I hope.

 Thomas

Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[Daniel P. Berrangé]

On Fri, Apr 27, 2018 at 05:58:44AM +0200, Thomas Huth wrote:
> On 26.04.2018 16:24, Daniel P. Berrangé wrote:
> > On Thu, Apr 26, 2018 at 11:22:36AM -0300, Eduardo Habkost wrote:
> >> On Thu, Apr 26, 2018 at 03:14:00PM +0100, Daniel P. Berrangé wrote:
> >>> On Thu, Apr 26, 2018 at 04:09:55PM +0200, Thomas Huth wrote:
> >>>> On 26.04.2018 15:57, Eduardo Habkost wrote:
> >>>>> (Starting a new thread, for more visibility)
> >>>>>
> >>>>> (This was: Re: [Qemu-devel] [RFC PATCH] tests/device-introspect: Test
> >>>>> devices with all machines, not only with "none")
> >>>>>
> >>>>> On Thu, Apr 26, 2018 at 01:54:43PM +0200, Markus Armbruster wrote:
> >>>> [...]
> >>>>>> I don't mind having make check SPEED=slow run more extensive tests.
> >>>>>> Assuming we actually run them at least once in a while, which seems
> >>>>>> doubtful.
> >>>>>
> >>>>> We probably don't do that, but we really must be running a more
> >>>>> extensive (and slower) test set at least once before every
> >>>>> release.
> >>>>>
> >>>>> Maybe some people are running SPEED=slow tests, or even more
> >>>>> extensive test suites like avocado-vt once in a while, but we
> >>>>> need to know who is running them, and when.
> >>>>
> >>>> At least I am running "make check SPEED=slow" manually from time to
> >>>> time, especially when we enter the hard freeze period.
> >>>
> >>> Hmm, we could get this done by travis. It has the concept of "cron jobs"
> >>> for scheduling builds separately from pushes.
> >>>
> >>> So we could keep the current travis jobs unchanged, but then add an
> >>> use of SPEED=slow when TRAVIS_EVENT_TYPE == "cron" in the travis.yml,
> >>> so we can get SPEED=slow run once a day. Just have to be careful which
> >>> jobs we make slow so we don't hit the 50 minute timeout.
> >>
> >> I expect SPEED=slow tests to eventually take longer than 50
> >> minutes, but we can try this and see what happens.
> > 
> > We can run many jobs though so if qtests take longer than 50 mins, we could
> > create 1 job per target arch.
> 
> That could work... individual "make check-qtest-xxx" should be short
> enough, I hope.

We would also have saved alot of time on the actual build phase before
that by only building 1 target.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[Thomas Huth]

On 26.04.2018 16:22, Eduardo Habkost wrote:
> On Thu, Apr 26, 2018 at 03:14:00PM +0100, Daniel P. Berrangé wrote:
>> On Thu, Apr 26, 2018 at 04:09:55PM +0200, Thomas Huth wrote:
>>> On 26.04.2018 15:57, Eduardo Habkost wrote:
>>>> (Starting a new thread, for more visibility)
>>>>
>>>> (This was: Re: [Qemu-devel] [RFC PATCH] tests/device-introspect: Test
>>>> devices with all machines, not only with "none")
>>>>
>>>> On Thu, Apr 26, 2018 at 01:54:43PM +0200, Markus Armbruster wrote:
>>> [...]
>>>>> I don't mind having make check SPEED=slow run more extensive tests.
>>>>> Assuming we actually run them at least once in a while, which seems
>>>>> doubtful.
>>>>
>>>> We probably don't do that, but we really must be running a more
>>>> extensive (and slower) test set at least once before every
>>>> release.
>>>>
>>>> Maybe some people are running SPEED=slow tests, or even more
>>>> extensive test suites like avocado-vt once in a while, but we
>>>> need to know who is running them, and when.
>>>
>>> At least I am running "make check SPEED=slow" manually from time to
>>> time, especially when we enter the hard freeze period.
>>
>> Hmm, we could get this done by travis. It has the concept of "cron jobs"
>> for scheduling builds separately from pushes.
>>
>> So we could keep the current travis jobs unchanged, but then add an
>> use of SPEED=slow when TRAVIS_EVENT_TYPE == "cron" in the travis.yml,
>> so we can get SPEED=slow run once a day. Just have to be careful which
>> jobs we make slow so we don't hit the 50 minute timeout.
> 
> I expect SPEED=slow tests to eventually take longer than 50
> minutes, but we can try this and see what happens.

I already had to move some of the slow tests to SPEED=slow mode in the
past due to the travis timeouts (which are hit when the machines are
very loaded), so my guess is that SPEED=slow will hardly work with travis.

 Thomas

Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[Cornelia Huck]

On Thu, 26 Apr 2018 16:25:40 +0200
Thomas Huth <thuth@redhat.com> wrote:

> On 26.04.2018 16:22, Eduardo Habkost wrote:
> > On Thu, Apr 26, 2018 at 03:14:00PM +0100, Daniel P. Berrangé wrote:  
> >> On Thu, Apr 26, 2018 at 04:09:55PM +0200, Thomas Huth wrote:  
> >>> On 26.04.2018 15:57, Eduardo Habkost wrote:  
> >>>> (Starting a new thread, for more visibility)
> >>>>
> >>>> (This was: Re: [Qemu-devel] [RFC PATCH] tests/device-introspect: Test
> >>>> devices with all machines, not only with "none")
> >>>>
> >>>> On Thu, Apr 26, 2018 at 01:54:43PM +0200, Markus Armbruster wrote:  
> >>> [...]  
> >>>>> I don't mind having make check SPEED=slow run more extensive tests.
> >>>>> Assuming we actually run them at least once in a while, which seems
> >>>>> doubtful.  
> >>>>
> >>>> We probably don't do that, but we really must be running a more
> >>>> extensive (and slower) test set at least once before every
> >>>> release.
> >>>>
> >>>> Maybe some people are running SPEED=slow tests, or even more
> >>>> extensive test suites like avocado-vt once in a while, but we
> >>>> need to know who is running them, and when.  
> >>>
> >>> At least I am running "make check SPEED=slow" manually from time to
> >>> time, especially when we enter the hard freeze period.  
> >>
> >> Hmm, we could get this done by travis. It has the concept of "cron jobs"
> >> for scheduling builds separately from pushes.
> >>
> >> So we could keep the current travis jobs unchanged, but then add an
> >> use of SPEED=slow when TRAVIS_EVENT_TYPE == "cron" in the travis.yml,
> >> so we can get SPEED=slow run once a day. Just have to be careful which
> >> jobs we make slow so we don't hit the 50 minute timeout.  
> > 
> > I expect SPEED=slow tests to eventually take longer than 50
> > minutes, but we can try this and see what happens.  
> 
> I already had to move some of the slow tests to SPEED=slow mode in the
> past due to the travis timeouts (which are hit when the machines are
> very loaded), so my guess is that SPEED=slow will hardly work with travis.

Do we have another automation infrastructure for that, preferably with
multiple host architectures? I usually run SPEED=slow before I send a
pull request, on s390x and x86 -- unless I forget it. (I really should
script my procedure. Some time.)

Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[Fam Zheng]

On Thu, 04/26 10:57, Eduardo Habkost wrote:
> So, for anybody that runs automated QEMU tests once in a while,
> can we know:
> 
> * What test cases are you running?  Where can we get more
>   information about the tests you run?

These currently run on patchew:

 * checkpatch
 * ./configure && make (s390 host)
 * make docker-test-mingw@fedora
 * make docker-test-quick@centos6
 * make docker-test-build@min-glib


> * When do you run them?  What triggers a new test run?

They are run against each series posted to qemu-devel@nongnu.org. All errors are
sent as reply to the cover letter with the concrete commands, env info and log.

(Should we create a wiki page to maintain information collected by this thread?)

Fam

Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[Eduardo Habkost]

On Fri, Apr 27, 2018 at 11:10:54AM +0800, Fam Zheng wrote:
> On Thu, 04/26 10:57, Eduardo Habkost wrote:
> > So, for anybody that runs automated QEMU tests once in a while,
> > can we know:
> > 
> > * What test cases are you running?  Where can we get more
> >   information about the tests you run?
> 
> These currently run on patchew:
> 
>  * checkpatch
>  * ./configure && make (s390 host)
>  * make docker-test-mingw@fedora
>  * make docker-test-quick@centos6
>  * make docker-test-build@min-glib
> 
> 
> > * When do you run them?  What triggers a new test run?
> 
> They are run against each series posted to qemu-devel@nongnu.org. All errors are
> sent as reply to the cover letter with the concrete commands, env info and log.
> 
> (Should we create a wiki page to maintain information collected by this thread?)

I will collect this data and publish it somewhere.

(I sure yet if a wiki page would be the most practical way to
maintain it up to date.  I'm using a Google spreadsheet to keep
my notes right now.)

-- 
Eduardo

Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[Anthony PERARD]

On Thu, Apr 26, 2018 at 10:57:55AM -0300, Eduardo Habkost wrote:
> So, for anybody that runs automated QEMU tests once in a while,
> can we know:
> 
> * What test cases are you running?  Where can we get more
>   information about the tests you run?
> * When do you run them?  What triggers a new test run?

FYI, we do test QEMU with Xen. But that mostly integration test. I guess
not much of QEMU is tested.

osstest run with the upstream QEMU tree about once a day, if the tree as
changed or if the last test run failed. The test result is sent to
xen-devel ml.

As for what is runned, it mostly starts Xen guests, do migrate, with
different configuration.

Regards,

-- 
Anthony PERARD

Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[Lukáš Doktor]

[-- Attachment #1: Type: text/plain, Size: 4965 bytes --]

Dne 26.4.2018 v 15:57 Eduardo Habkost napsal(a):
> (Starting a new thread, for more visibility)
> 

Hello guys,

what a nice topic. My Jenkins runs following weekly/daily upstream checks using RHEL.7 as host:

ppc64
=====

frequency: weekly
host:
    - ppc64
    - ppc64le
tests:
    - make    # building all targets
    - SPEED=slow make check
    - kvm-unit-tests:
        - ACCEL=kvm,kvm-type=HV
        - ACCEL=kvm,kvm-type=PR
        - ACCEL=tcg
    - qemu-iotests:
        - ALL_TESTS -qcow2 -file
        - ALL_TESTS -qcow2 -nbd    # quite broken
        - ALL_TESTS -raw -file
        - ALL_TESTS -raw -nbd    # quite broken
        - 059 -vmdk
        - 064 vhdx
        - 070 vhdx
        - 075 cloop
        - 076 parallels
        - 078 -bochs
        - 084 -vdi
        - 088 -vpc
        - 116 -qed
        - 131 -parallels
        - 135 -vpc
        - 146 -vpc
        # I tried -nfs but it seems broken and I'm still waiting for feedback
    - functonal:    # using Avocado-vt
        guest:
            - ppc64
            - ppc64le
        tests:
            - various RHEL.7 install jobs
            - migration between various tagged qemu revisions

s390x
=====

frequency: daily
tests:
    - make     # building all targets
    - SPEED=slow make check
    - kvm-unit-tests
    - functonal:    # using Avocado-vt
        - various RHEL.7 install jobs
        - migration using the latest qemu only

Unfortunately I'm currently changing the setup so for the past 1-2 months it's semi-broken. When I finish the transition I plan to add aarch64 upstream checks as well (as I am already running similar downstream suite there), but now I'm struggling with Jenkins and the new setup (which should have simplified things).

Kind regards,
Lukáš


> (This was: Re: [Qemu-devel] [RFC PATCH] tests/device-introspect: Test
> devices with all machines, not only with "none")
> 
> On Thu, Apr 26, 2018 at 01:54:43PM +0200, Markus Armbruster wrote:
>> Thomas Huth <thuth@redhat.com> writes:
>>> On 17.04.2018 14:12, Markus Armbruster wrote:
>>>> Thomas Huth <thuth@redhat.com> writes:
>>>>
>>>>> Many device introspection crashes only happen if you are using a
>>>>> certain machine, e.g.:
>>>>>
>>>>> $ ppc-softmmu/qemu-system-ppc -S -M ref405ep,accel=qtest -qmp stdio
>>>>> {"QMP": {"version": {"qemu": {"micro": 50, "minor": 11, "major": 2},
>>>>>  "package": "build-all"}, "capabilities": []}}
>>>>> { 'execute': 'qmp_capabilities' }
>>>>> {"return": {}}
>>>>> { 'execute': 'device-list-properties',
>>>>>   'arguments': {'typename': 'macio-newworld'}}
>>>>> Unexpected error in qemu_chr_fe_init() at chardev/char-fe.c:222:
>>>>> Device 'serial0' is in use
>>>>> Aborted (core dumped)
>>>>>
>>>>> To be able to catch these problems, let's extend the device-introspect
>>>>> test to check the devices on all machine types. Since this is a rather
>>>>> slow operation, the test is only run in "SPEED=slow" mode.
>>>>
>>>> If the device works with one machine type, it has a decent chance to
>>>> work with others, too.  Thus, testing each device with every machine
>>>> type is overkill.  I appreciate having overkill as an option :)
>>>>
>>>> What I'd like to see for a quick "make check" is testing each device
>>>> once.  That should flush out most bugs.  
>>>
>>> That's already done with the "none" machine.
>>
>> I was too terse.  We test each device with -machine none for every
>> target.  Fine if that's quick enough.  If not, we might want to reduce
>> redundancy there.
>>
>> Actually, a worse offender in the "waste everybody's time via redunancy"
>> department could be qom-test.
>>
>>> Anyway, do you think my patch here is useful and has a chance of getting
>>> included? I.e. shall I re-spin this as a non-RFC patch? Or shall we
>>> rather wait for Eduardo's python-based tests to get included into the
>>> repository?
>>
>> I don't mind having make check SPEED=slow run more extensive tests.
>> Assuming we actually run them at least once in a while, which seems
>> doubtful.
> 
> We probably don't do that, but we really must be running a more
> extensive (and slower) test set at least once before every
> release.
> 
> Maybe some people are running SPEED=slow tests, or even more
> extensive test suites like avocado-vt once in a while, but we
> need to know who is running them, and when.
> 
> Today, the only test set I know people really run and would
> surely block a release is "make check [SPEED=quick]".
> 
> So, for anybody that runs automated QEMU tests once in a while,
> can we know:
> 
> * What test cases are you running?  Where can we get more
>   information about the tests you run?
> * When do you run them?  What triggers a new test run?
> 
> Peter, do you have additional tests you run before merging a pull
> request?  Additional test sets run before tagging a release?
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3538 bytes --]

On Thu, Apr 26, 2018 at 10:57:55AM -0300, Eduardo Habkost wrote:
> (Starting a new thread, for more visibility)
> 
> (This was: Re: [Qemu-devel] [RFC PATCH] tests/device-introspect: Test
> devices with all machines, not only with "none")
> 
> On Thu, Apr 26, 2018 at 01:54:43PM +0200, Markus Armbruster wrote:
> > Thomas Huth <thuth@redhat.com> writes:
> > > On 17.04.2018 14:12, Markus Armbruster wrote:
> > >> Thomas Huth <thuth@redhat.com> writes:
> > >> 
> > >>> Many device introspection crashes only happen if you are using a
> > >>> certain machine, e.g.:
> > >>>
> > >>> $ ppc-softmmu/qemu-system-ppc -S -M ref405ep,accel=qtest -qmp stdio
> > >>> {"QMP": {"version": {"qemu": {"micro": 50, "minor": 11, "major": 2},
> > >>>  "package": "build-all"}, "capabilities": []}}
> > >>> { 'execute': 'qmp_capabilities' }
> > >>> {"return": {}}
> > >>> { 'execute': 'device-list-properties',
> > >>>   'arguments': {'typename': 'macio-newworld'}}
> > >>> Unexpected error in qemu_chr_fe_init() at chardev/char-fe.c:222:
> > >>> Device 'serial0' is in use
> > >>> Aborted (core dumped)
> > >>>
> > >>> To be able to catch these problems, let's extend the device-introspect
> > >>> test to check the devices on all machine types. Since this is a rather
> > >>> slow operation, the test is only run in "SPEED=slow" mode.
> > >> 
> > >> If the device works with one machine type, it has a decent chance to
> > >> work with others, too.  Thus, testing each device with every machine
> > >> type is overkill.  I appreciate having overkill as an option :)
> > >> 
> > >> What I'd like to see for a quick "make check" is testing each device
> > >> once.  That should flush out most bugs.  
> > >
> > > That's already done with the "none" machine.
> > 
> > I was too terse.  We test each device with -machine none for every
> > target.  Fine if that's quick enough.  If not, we might want to reduce
> > redundancy there.
> > 
> > Actually, a worse offender in the "waste everybody's time via redunancy"
> > department could be qom-test.
> > 
> > > Anyway, do you think my patch here is useful and has a chance of getting
> > > included? I.e. shall I re-spin this as a non-RFC patch? Or shall we
> > > rather wait for Eduardo's python-based tests to get included into the
> > > repository?
> > 
> > I don't mind having make check SPEED=slow run more extensive tests.
> > Assuming we actually run them at least once in a while, which seems
> > doubtful.
> 
> We probably don't do that, but we really must be running a more
> extensive (and slower) test set at least once before every
> release.
> 
> Maybe some people are running SPEED=slow tests, or even more
> extensive test suites like avocado-vt once in a while, but we
> need to know who is running them, and when.
> 
> Today, the only test set I know people really run and would
> surely block a release is "make check [SPEED=quick]".
> 
> So, for anybody that runs automated QEMU tests once in a while,
> can we know:
> 
> * What test cases are you running?  Where can we get more
>   information about the tests you run?
> * When do you run them?  What triggers a new test run?

I manually run qemu-iotests on release candidates:

 $ (cd tests/qemu-iotests && ./check && ./check -qcow2)

The goal is to identify test failures that still need to be addressed
before the release is made.

I think Kevin Wolf and John Snow have also been running qemu-iotests to
eliminate regressions during the freeze.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Who is running QEMU automated tests, and when?

[John Snow]

On 05/08/2018 05:56 AM, Stefan Hajnoczi wrote:
> On Thu, Apr 26, 2018 at 10:57:55AM -0300, Eduardo Habkost wrote:
>> (Starting a new thread, for more visibility)
>>
>> (This was: Re: [Qemu-devel] [RFC PATCH] tests/device-introspect: Test
>> devices with all machines, not only with "none")
>>
>> On Thu, Apr 26, 2018 at 01:54:43PM +0200, Markus Armbruster wrote:
>>> Thomas Huth <thuth@redhat.com> writes:
>>>> On 17.04.2018 14:12, Markus Armbruster wrote:
>>>>> Thomas Huth <thuth@redhat.com> writes:
>>>>>
>>>>>> Many device introspection crashes only happen if you are using a
>>>>>> certain machine, e.g.:
>>>>>>
>>>>>> $ ppc-softmmu/qemu-system-ppc -S -M ref405ep,accel=qtest -qmp stdio
>>>>>> {"QMP": {"version": {"qemu": {"micro": 50, "minor": 11, "major": 2},
>>>>>>  "package": "build-all"}, "capabilities": []}}
>>>>>> { 'execute': 'qmp_capabilities' }
>>>>>> {"return": {}}
>>>>>> { 'execute': 'device-list-properties',
>>>>>>   'arguments': {'typename': 'macio-newworld'}}
>>>>>> Unexpected error in qemu_chr_fe_init() at chardev/char-fe.c:222:
>>>>>> Device 'serial0' is in use
>>>>>> Aborted (core dumped)
>>>>>>
>>>>>> To be able to catch these problems, let's extend the device-introspect
>>>>>> test to check the devices on all machine types. Since this is a rather
>>>>>> slow operation, the test is only run in "SPEED=slow" mode.
>>>>>
>>>>> If the device works with one machine type, it has a decent chance to
>>>>> work with others, too.  Thus, testing each device with every machine
>>>>> type is overkill.  I appreciate having overkill as an option :)
>>>>>
>>>>> What I'd like to see for a quick "make check" is testing each device
>>>>> once.  That should flush out most bugs.  
>>>>
>>>> That's already done with the "none" machine.
>>>
>>> I was too terse.  We test each device with -machine none for every
>>> target.  Fine if that's quick enough.  If not, we might want to reduce
>>> redundancy there.
>>>
>>> Actually, a worse offender in the "waste everybody's time via redunancy"
>>> department could be qom-test.
>>>
>>>> Anyway, do you think my patch here is useful and has a chance of getting
>>>> included? I.e. shall I re-spin this as a non-RFC patch? Or shall we
>>>> rather wait for Eduardo's python-based tests to get included into the
>>>> repository?
>>>
>>> I don't mind having make check SPEED=slow run more extensive tests.
>>> Assuming we actually run them at least once in a while, which seems
>>> doubtful.
>>
>> We probably don't do that, but we really must be running a more
>> extensive (and slower) test set at least once before every
>> release.
>>
>> Maybe some people are running SPEED=slow tests, or even more
>> extensive test suites like avocado-vt once in a while, but we
>> need to know who is running them, and when.
>>
>> Today, the only test set I know people really run and would
>> surely block a release is "make check [SPEED=quick]".
>>
>> So, for anybody that runs automated QEMU tests once in a while,
>> can we know:
>>
>> * What test cases are you running?  Where can we get more
>>   information about the tests you run?
>> * When do you run them?  What triggers a new test run?
> 
> I manually run qemu-iotests on release candidates:
> 
>  $ (cd tests/qemu-iotests && ./check && ./check -qcow2)
> 
> The goal is to identify test failures that still need to be addressed
> before the release is made.
> 
> I think Kevin Wolf and John Snow have also been running qemu-iotests to
> eliminate regressions during the freeze.
> 
> Stefan
> 

Yeah, it's always manual though.

I generally do an x86-for-x86 build with clang, make check, and do an
iotests across raw and qcow2 before sending a patch series.

When RC0 hits, I usually do a full build of all targets and do an
iotests check across all the file formats, but I don't bother with
protocols.

I generally don't use gcc because there's probably enough coverage out
there.

--js