All of lore.kernel.org
 help / color / mirror / Atom feed
From: Peter Maydell <peter.maydell@linaro.org>
To: Paolo Bonzini <pbonzini@redhat.com>
Cc: "Alex Bennée" <alex.bennee@linaro.org>,
	"QEMU Developers" <qemu-devel@nongnu.org>,
	"Stefan Hajnoczi" <stefanha@redhat.com>
Subject: Re: test-aio-multithread assertion
Date: Tue, 19 Nov 2019 17:55:05 +0000	[thread overview]
Message-ID: <CAFEAcA-aq-0_QuHyiqpjWMKcxSaadyyfgcvg5bKPUChpSbGmvA@mail.gmail.com> (raw)
In-Reply-To: <f4cfc080-2084-0da5-e3ad-2f65aaed41d5@redhat.com>

On Mon, 14 Oct 2019 at 11:50, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 14/10/19 12:26, Peter Maydell wrote:
> > On Thu, 5 Sep 2019 at 14:17, Peter Maydell <peter.maydell@linaro.org> wrote:
> >> Here's a different intermittent assertion on Netbsd from test-aio-multithread:
> >>
> >> MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}
> >> tests/test-aio-multithread -m=quick
> >> -k --tap < /dev/null | ./scripts/tap-driver.pl
> >> --test-name="test-aio-multithread"
> >> PASS 1 test-aio-multithread /aio/multi/lifecycle
> >> PASS 2 test-aio-multithread /aio/multi/schedule
> >> assertion "QSLIST_EMPTY(&ctx->scheduled_coroutines)" failed: file
> >> "util/async.c", line 279, function "
> >> aio_ctx_finalize"
> >> ERROR - too few tests run (expected 5, got 2)
> >
> > Just saw this QSLIST_EMPTY(&ctx->scheduled_coroutines) assertion
> > from FreeBSD as well, so it's still around as a definite
> > intermittent.
>
> This could be similar to what Stefan fixed in commit 2362a28ea1, only in
> the test infrastructure.

Here's a gdb backtrace from as assertion failure under NetBSD
when running ./tests/test-aio-multithread -p /aio/multi/mutex/contended
(which is pretty replicable for me); does it look like the same
thing as the commit 2362a28ea1 issue? (Apologies for all the
"optimized out" stuff in the backtraces; I'll try to rerun with
a --enable-debug build.)

Thread 7 (LWP 2):
#0  0x00007e4b44499c0a in ___lwp_park60 () from /usr/lib/libc.so.12
#1  0x00007e4b44809ee3 in pthread_cond_timedwait () from
/usr/lib/libpthread.so.1
#2  0x00000000004c0862 in qemu_futex_wait (val=4294967295, ev=0x760c00
<rcu_call_ready_event>)
    at /home/qemu/qemu-test.0WfUNX/src/util/qemu-thread-posix.c:364
#3  qemu_event_wait (ev=ev@entry=0x760c00 <rcu_call_ready_event>)
    at /home/qemu/qemu-test.0WfUNX/src/util/qemu-thread-posix.c:459
#4  0x00000000004ce825 in call_rcu_thread (opaque=opaque@entry=0x0)
    at /home/qemu/qemu-test.0WfUNX/src/util/rcu.c:260
#5  0x00000000004bfd17 in qemu_thread_start (args=0x7e4b47b22c00)
    at /home/qemu/qemu-test.0WfUNX/src/util/qemu-thread-posix.c:519
#6  0x00007e4b4480b8b5 in ?? () from /usr/lib/libpthread.so.1
#7  0x00007e4b44481e20 in ?? () from /usr/lib/libc.so.12
#8  0x00010102464c457f in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 6 (LWP 3):
#0  0x00007e4b4443e02a in write () from /usr/lib/libc.so.12
#1  0x00007e4b44807561 in write () from /usr/lib/libpthread.so.1
#2  0x00000000004beae0 in event_notifier_set (e=e@entry=0x7e4b46f01214)
    at /home/qemu/qemu-test.0WfUNX/src/util/event_notifier-posix.c:100
#3  0x00000000004bb66c in aio_notify (ctx=0x7e4b46f01120)
    at /home/qemu/qemu-test.0WfUNX/src/util/async.c:350
#4  0x00000000004bb936 in aio_co_schedule (ctx=0x7e4b46f01120,
co=0x7e4b46b01280)
    at /home/qemu/qemu-test.0WfUNX/src/util/async.c:473
#5  0x00000000004cf226 in qemu_co_mutex_wake (mutex=0x75fc00
<comutex>, co=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:198
#6  qemu_co_mutex_unlock (mutex=mutex@entry=0x75fc00 <comutex>)
    at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:301
#7  0x000000000040b33b in test_multi_co_mutex_entry (opaque=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:208
#8  0x00000000004cff6b in coroutine_trampoline (i0=<optimized out>,
i1=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/util/coroutine-ucontext.c:115
#9  0x00007e4b4445dba0 in ?? () from /usr/lib/libc.so.12
#10 0x00010102464c457f in ?? ()
#11 0x0000000000000000 in ?? ()

Thread 5 (LWP 4):
#0  0x000000000040b34a in test_multi_co_mutex_entry (opaque=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:205
#1  0x00000000004cff6b in coroutine_trampoline (i0=<optimized out>,
i1=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/util/coroutine-ucontext.c:115
#2  0x00007e4b4445dba0 in ?? () from /usr/lib/libc.so.12
#3  0x00010102464c457f in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 4 (LWP 5):
#0  0x00007e4b4443e02a in write () from /usr/lib/libc.so.12
#1  0x00007e4b44807561 in write () from /usr/lib/libpthread.so.1
#2  0x00000000004beae0 in event_notifier_set (e=e@entry=0x7e4b47701384)
    at /home/qemu/qemu-test.0WfUNX/src/util/event_notifier-posix.c:100
#3  0x00000000004bb66c in aio_notify (ctx=0x7e4b47701290)
    at /home/qemu/qemu-test.0WfUNX/src/util/async.c:350
#4  0x00000000004bb936 in aio_co_schedule (ctx=0x7e4b47701290,
co=0x7e4b46b01420)
    at /home/qemu/qemu-test.0WfUNX/src/util/async.c:473
#5  0x00000000004cf226 in qemu_co_mutex_wake (mutex=0x75fc00
<comutex>, co=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:198
#6  qemu_co_mutex_unlock (mutex=mutex@entry=0x75fc00 <comutex>)
    at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:301
#7  0x000000000040b33b in test_multi_co_mutex_entry (opaque=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:208
#8  0x00000000004cff6b in coroutine_trampoline (i0=<optimized out>,
i1=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/util/coroutine-ucontext.c:115
#9  0x00007e4b4445dba0 in ?? () from /usr/lib/libc.so.12
#10 0x00010102464c457f in ?? ()
#11 0x0000000000000000 in ?? ()

Thread 3 (LWP 7):
#0  0x00000000004cf200 in qemu_co_mutex_unlock
(mutex=mutex@entry=0x75fc00 <comutex>)
    at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:291
#1  0x000000000040b33b in test_multi_co_mutex_entry (opaque=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:208
#2  0x00000000004cff6b in coroutine_trampoline (i0=<optimized out>,
i1=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/util/coroutine-ucontext.c:115
#3  0x00007e4b4445dba0 in ?? () from /usr/lib/libc.so.12
#4  0x00010102464c457f in ?? ()
#5  0x0000000000000000 in ?? ()

Thread 2 (LWP 6):
#0  0x00007e4b444fe7ea in _lwp_kill () from /usr/lib/libc.so.12
#1  0x00007e4b444fe092 in abort () from /usr/lib/libc.so.12
#2  0x00007e4b4449fd57 in __assert13 () from /usr/lib/libc.so.12
#3  0x00000000004bb3e0 in aio_ctx_finalize (source=0x7e4b47701290)
    at /home/qemu/qemu-test.0WfUNX/src/util/async.c:279
#4  0x00007e4b46445596 in g_source_unref_internal () from
/usr/pkg/lib/libglib-2.0.so.0
#5  0x00000000004bb93e in aio_context_unref (ctx=0x7e4b47701290)
    at /home/qemu/qemu-test.0WfUNX/src/util/async.c:516
#6  aio_co_schedule (ctx=0x7e4b47701290, co=0x7e4b46b01420)
    at /home/qemu/qemu-test.0WfUNX/src/util/async.c:475
#7  0x00000000004cf226 in qemu_co_mutex_wake (mutex=0x75fc00
<comutex>, co=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:198
#8  qemu_co_mutex_unlock (mutex=mutex@entry=0x75fc00 <comutex>)
    at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:301
#9  0x000000000040b33b in test_multi_co_mutex_entry (opaque=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:208
#10 0x00000000004cff6b in coroutine_trampoline (i0=<optimized out>,
i1=<optimized out>)
    at /home/qemu/qemu-test.0WfUNX/src/util/coroutine-ucontext.c:115
#11 0x00007e4b4445dba0 in ?? () from /usr/lib/libc.so.12
#12 0x00010102464c457f in ?? ()
#13 0x0000000000000000 in ?? ()

Thread 1 (LWP 1):
#0  0x00007e4b4443e2ea in _sys___nanosleep50 () from /usr/lib/libc.so.12
#1  0x00007e4b44807287 in __nanosleep50 () from /usr/lib/libpthread.so.1
#2  0x00007e4b4646e511 in g_usleep () from /usr/pkg/lib/libglib-2.0.so.0
#3  0x000000000040b72e in test_multi_co_mutex (threads=<optimized
out>, seconds=1)
    at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:237
#4  0x00007e4b4646bd15 in g_test_run_suite_internal () from
/usr/pkg/lib/libglib-2.0.so.0
#5  0x00007e4b4646bc3e in g_test_run_suite_internal () from
/usr/pkg/lib/libglib-2.0.so.0
#6  0x00007e4b4646bc3e in g_test_run_suite_internal () from
/usr/pkg/lib/libglib-2.0.so.0
#7  0x00007e4b4646bc3e in g_test_run_suite_internal () from
/usr/pkg/lib/libglib-2.0.so.0
#8  0x00007e4b4646c1a0 in g_test_run_suite () from /usr/pkg/lib/libglib-2.0.so.0
#9  0x00007e4b4646c1e8 in g_test_run () from /usr/pkg/lib/libglib-2.0.so.0
#10 0x00000000004dbcdc in main (argc=1, argv=0x7f7fffd572f8)
    at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:459

thanks
-- PMM


  reply	other threads:[~2019-11-19 17:56 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-09-04 11:50 [Qemu-devel] test-aio-multithread assertion Peter Maydell
2019-09-05 13:17 ` Peter Maydell
2019-10-14 10:26   ` Peter Maydell
2019-10-14 10:50     ` Paolo Bonzini
2019-11-19 17:55       ` Peter Maydell [this message]
2019-11-19 18:01         ` Peter Maydell
2019-09-16 15:33 ` [Qemu-devel] " Stefan Hajnoczi

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAFEAcA-aq-0_QuHyiqpjWMKcxSaadyyfgcvg5bKPUChpSbGmvA@mail.gmail.com \
    --to=peter.maydell@linaro.org \
    --cc=alex.bennee@linaro.org \
    --cc=pbonzini@redhat.com \
    --cc=qemu-devel@nongnu.org \
    --cc=stefanha@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.