[Qemu-devel] [RFC PATCH 0/2] Address OS X Travis failures

[Qemu-devel] [RFC PATCH 0/2] Address OS X Travis failures

[Cleber Rosa]

The OS X jobs on Travis fail for a number of reasons.  This is an
attempt to decide its faith, either as something that will be kept on
the configuration (and properly maintained), removed, or maybe active
but with failures waived (Travis allows for that).

The result of this can be seen in these jobs part of a recent build:

 - https://travis-ci.org/clebergnu/qemu/jobs/452517040
 - https://travis-ci.org/clebergnu/qemu/jobs/452517041
 - https://travis-ci.org/clebergnu/qemu/jobs/452517042
 - https://travis-ci.org/clebergnu/qemu/jobs/452517043
 - https://travis-ci.org/clebergnu/qemu/jobs/452517044
 - https://travis-ci.org/clebergnu/qemu/jobs/452517045
 - https://travis-ci.org/clebergnu/qemu/jobs/452517046

Cleber Rosa (2):
  tests/test-qht-par: test gets stuck intermittently on OSX
  Travis CI: break down OSX+clang jobs

 .travis.yml            | 29 ++++++++++++++++++++++++++++-
 tests/Makefile.include |  3 +++
 2 files changed, 31 insertions(+), 1 deletion(-)

-- 
2.19.1

[Qemu-devel] [RFC PATCH 1/2] tests/test-qht-par: test gets stuck intermittently on OSX

[Cleber Rosa]

To be fully honest, this may not be a OSX (alone) condition, but may
be a situation that only happens with OSX on Travis-CI, were resources
are quite limited.

I have personal experience with tests that exercise parallelism or
depend on timing to fail on Travis.  Because I'm not 100% certain that
this is a situation that only happens with OSX on Travis-CI, and
because I'm not certain that we should be skipping tests because
they're running on Travis-CI, let's disable them on OSX as a whole.

A small note: this type of change makes me believe that there should
be a list of testing related caveats or TODO list tracked on the
documentation.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/Makefile.include | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 074eece558..c821b01467 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -87,7 +87,10 @@ check-unit-y += tests/test-rcu-simpleq$(EXESUF)
 check-unit-y += tests/test-rcu-tailq$(EXESUF)
 check-unit-y += tests/test-qdist$(EXESUF)
 check-unit-y += tests/test-qht$(EXESUF)
+# test-qht-par gets stuck quite often on OSX
+ifneq ($(CONFIG_DARWIN),y)
 check-unit-y += tests/test-qht-par$(EXESUF)
+endif
 check-unit-y += tests/test-bitops$(EXESUF)
 check-unit-y += tests/test-bitcnt$(EXESUF)
 check-unit-y += tests/test-qdev-global-props$(EXESUF)
-- 
2.19.1

[Qemu-devel] [RFC PATCH 2/2] Travis CI: break down OSX+clang jobs

[Cleber Rosa]

The OSX jobs were timing out on Travis, due to the long time they need
to run tests with all targets.  Let's break them down to avoid hitting
the time limit, and at the same time, give faster results.

Additionally the qtest based tests were hanging intermittently.  The
first debugging attempt involved making their execution verbose.  That
alone seemed to produce a positive effect and no hangs were observed
after that.  Given that Travis checks for output to decide if a test
is hung or not, it makes sense that a verbose execution minimizes
false positives.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 .travis.yml | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)

diff --git a/.travis.yml b/.travis.yml
index aa49c7b114..505561aae6 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -107,7 +107,34 @@ matrix:
     - env: CONFIG="--disable-tcg"
            TEST_CMD=""
       compiler: gcc
-    - env: CONFIG=""
+    # osx+clang jobs are broken in a number of target specific sets to
+    # allow jobs to finish before timing out
+    - env: CONFIG="--target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,cris-softmmu"
+           TEST_CMD="make V=1 check"
+      os: osx
+      compiler: clang
+    - env: CONFIG="--target-list=i386-softmmu,lm32-softmmu,m68k-softmmu,moxie-softmmu"
+           TEST_CMD="make V=1 check"
+      os: osx
+      compiler: clang
+    - env: CONFIG="--target-list=mips-softmmu,mips64-softmmu,mips64el-softmmu,mipsel-softmmu"
+           TEST_CMD="make V=1 check"
+      os: osx
+      compiler: clang
+    - env: CONFIG="--target-list=nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,riscv32-softmmu"
+           TEST_CMD="make V=1 check"
+      os: osx
+      compiler: clang
+    - env: CONFIG="--target-list=s390x-softmmu,sh4-softmmu,sh4eb-softmmu,sparc-softmmu,sparc64-softmmu"
+           TEST_CMD="make V=1 check"
+      os: osx
+      compiler: clang
+    - env: CONFIG="--target-list=unicore32-softmmu,x86_64-softmmu,xtensa-softmmu,xtensaeb-softmmu"
+           TEST_CMD="make V=1 check"
+      os: osx
+      compiler: clang
+    - env: CONFIG="--target-list=hppa-softmmu,microblaze-softmmu,microblazeel-softmmu,tricore-softmmu,riscv64-softmmu"
+           TEST_CMD="make V=1 check"
       os: osx
       compiler: clang
     # Python builds
-- 
2.19.1

Re: [Qemu-devel] [RFC PATCH 1/2] tests/test-qht-par: test gets stuck intermittently on OSX

[Peter Maydell]

On 9 November 2018 at 15:30, Cleber Rosa <crosa@redhat.com> wrote:
> To be fully honest, this may not be a OSX (alone) condition, but may
> be a situation that only happens with OSX on Travis-CI, were resources
> are quite limited.
>
> I have personal experience with tests that exercise parallelism or
> depend on timing to fail on Travis.  Because I'm not 100% certain that
> this is a situation that only happens with OSX on Travis-CI, and
> because I'm not certain that we should be skipping tests because
> they're running on Travis-CI, let's disable them on OSX as a whole.

Does this test still get stuck in builds after a458774ad711bceabefb ?
That fixed an OSX-specific problem that caused hangs (notably
in test-bdrv-drain, but it could I guess also affect other tests).

thanks
-- PMM

Re: [Qemu-devel] [RFC PATCH 1/2] tests/test-qht-par: test gets stuck intermittently on OSX

[Emilio G. Cota]

On Fri, Nov 09, 2018 at 10:30:01 -0500, Cleber Rosa wrote:
> To be fully honest, this may not be a OSX (alone) condition, but may
> be a situation that only happens with OSX on Travis-CI, were resources
> are quite limited.
> 
> I have personal experience with tests that exercise parallelism or
> depend on timing to fail on Travis.  Because I'm not 100% certain that
> this is a situation that only happens with OSX on Travis-CI, and
> because I'm not certain that we should be skipping tests because
> they're running on Travis-CI, let's disable them on OSX as a whole.
> 
> A small note: this type of change makes me believe that there should
> be a list of testing related caveats or TODO list tracked on the
> documentation.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>

Is there a way to know where it hangs?

Also, are these the g_test_quick tests, or the other ones? It it's
just a matter of the program taking too long to run, we could just
reduce the length of the !quick tests (currently at 5s vs. 1s for
the quick ones).

Thanks,

		Emilio

Re: [Qemu-devel] [RFC PATCH 1/2] tests/test-qht-par: test gets stuck intermittently on OSX

[Cleber Rosa]

On 11/9/18 10:34 AM, Peter Maydell wrote:
> On 9 November 2018 at 15:30, Cleber Rosa <crosa@redhat.com> wrote:
>> To be fully honest, this may not be a OSX (alone) condition, but may
>> be a situation that only happens with OSX on Travis-CI, were resources
>> are quite limited.
>>
>> I have personal experience with tests that exercise parallelism or
>> depend on timing to fail on Travis.  Because I'm not 100% certain that
>> this is a situation that only happens with OSX on Travis-CI, and
>> because I'm not certain that we should be skipping tests because
>> they're running on Travis-CI, let's disable them on OSX as a whole.
> 
> Does this test still get stuck in builds after a458774ad711bceabefb ?
> That fixed an OSX-specific problem that caused hangs (notably
> in test-bdrv-drain, but it could I guess also affect other tests).
> 

It didn't get stuck here:

https://travis-ci.org/clebergnu/qemu/jobs/452663111

So I believe that was the common issue indeed.  Thanks for pointing that
out.  Also, the other patch in this series also seems to be unnecessary now.

- Cleber.

> thanks
> -- PMM
> 

Re: [Qemu-devel] [RFC PATCH 1/2] tests/test-qht-par: test gets stuck intermittently on OSX

[Cleber Rosa]

On 11/9/18 10:56 AM, Emilio G. Cota wrote:
> On Fri, Nov 09, 2018 at 10:30:01 -0500, Cleber Rosa wrote:
>> To be fully honest, this may not be a OSX (alone) condition, but may
>> be a situation that only happens with OSX on Travis-CI, were resources
>> are quite limited.
>>
>> I have personal experience with tests that exercise parallelism or
>> depend on timing to fail on Travis.  Because I'm not 100% certain that
>> this is a situation that only happens with OSX on Travis-CI, and
>> because I'm not certain that we should be skipping tests because
>> they're running on Travis-CI, let's disable them on OSX as a whole.
>>
>> A small note: this type of change makes me believe that there should
>> be a list of testing related caveats or TODO list tracked on the
>> documentation.
>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> 
> Is there a way to know where it hangs?
> 
> Also, are these the g_test_quick tests, or the other ones? It it's
> just a matter of the program taking too long to run, we could just
> reduce the length of the !quick tests (currently at 5s vs. 1s for
> the quick ones).
> 
> Thanks,
> 
> 		Emilio
> 

Looks like the common cause was fixed a458774ad711bceabefb, as pointed
out by Peter.

- Cleber.