* [Qemu-devel] test-aio-multithread assertion
@ 2019-09-04 11:50 Peter Maydell
2019-09-05 13:17 ` Peter Maydell
2019-09-16 15:33 ` [Qemu-devel] " Stefan Hajnoczi
0 siblings, 2 replies; 7+ messages in thread
From: Peter Maydell @ 2019-09-04 11:50 UTC (permalink / raw)
To: QEMU Developers
Hi; I've been seeing intermittently on the BSDs this assertion
running test-aio-multithread as part of 'make check':
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}
tests/test-aio-multithread -m=quick -k --tap < /dev/null |
./scripts/tap-drive
r.pl --test-name="test-aio-multithread"
PASS 1 test-aio-multithread /aio/multi/lifecycle
PASS 2 test-aio-multithread /aio/multi/schedule
ERROR - too few tests run (expected 5, got 2)
Assertion failed: (!qemu_lockcnt_count(&ctx->list_lock)), function
aio_ctx_finalize, file /home/qemu/qemu-test.kaRA0o/src/util/async.c,
line 283.
Abort trap (core dumped)
gmake: *** [/home/qemu/qemu-test.kaRA0o/src/tests/Makefile.include:905:
check-unit] Error 1
Anybody got any ideas?
(this is with the tests/vm setups; I've seen this on FreeBSD
and NetBSD.)
thanks
-- PMM
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [Qemu-devel] test-aio-multithread assertion 2019-09-04 11:50 [Qemu-devel] test-aio-multithread assertion Peter Maydell @ 2019-09-05 13:17 ` Peter Maydell 2019-10-14 10:26 ` Peter Maydell 2019-09-16 15:33 ` [Qemu-devel] " Stefan Hajnoczi 1 sibling, 1 reply; 7+ messages in thread From: Peter Maydell @ 2019-09-05 13:17 UTC (permalink / raw) To: QEMU Developers On Wed, 4 Sep 2019 at 12:50, Peter Maydell <peter.maydell@linaro.org> wrote: > > Hi; I've been seeing intermittently on the BSDs this assertion > running test-aio-multithread as part of 'make check': > > MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} > tests/test-aio-multithread -m=quick -k --tap < /dev/null | > ./scripts/tap-drive > r.pl --test-name="test-aio-multithread" > PASS 1 test-aio-multithread /aio/multi/lifecycle > PASS 2 test-aio-multithread /aio/multi/schedule > ERROR - too few tests run (expected 5, got 2) > Assertion failed: (!qemu_lockcnt_count(&ctx->list_lock)), function > aio_ctx_finalize, file /home/qemu/qemu-test.kaRA0o/src/util/async.c, > line 283. > Abort trap (core dumped) > gmake: *** [/home/qemu/qemu-test.kaRA0o/src/tests/Makefile.include:905: > check-unit] Error 1 > > Anybody got any ideas? Here's a different intermittent assertion on Netbsd from test-aio-multithread: MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-aio-multithread -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-aio-multithread" PASS 1 test-aio-multithread /aio/multi/lifecycle PASS 2 test-aio-multithread /aio/multi/schedule assertion "QSLIST_EMPTY(&ctx->scheduled_coroutines)" failed: file "util/async.c", line 279, function " aio_ctx_finalize" ERROR - too few tests run (expected 5, got 2) thanks -- PMM ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: test-aio-multithread assertion 2019-09-05 13:17 ` Peter Maydell @ 2019-10-14 10:26 ` Peter Maydell 2019-10-14 10:50 ` Paolo Bonzini 0 siblings, 1 reply; 7+ messages in thread From: Peter Maydell @ 2019-10-14 10:26 UTC (permalink / raw) To: QEMU Developers, Stefan Hajnoczi, Paolo Bonzini On Thu, 5 Sep 2019 at 14:17, Peter Maydell <peter.maydell@linaro.org> wrote: > > On Wed, 4 Sep 2019 at 12:50, Peter Maydell <peter.maydell@linaro.org> wrote: > > > > Hi; I've been seeing intermittently on the BSDs this assertion > > running test-aio-multithread as part of 'make check': > > > > MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} > > tests/test-aio-multithread -m=quick -k --tap < /dev/null | > > ./scripts/tap-drive > > r.pl --test-name="test-aio-multithread" > > PASS 1 test-aio-multithread /aio/multi/lifecycle > > PASS 2 test-aio-multithread /aio/multi/schedule > > ERROR - too few tests run (expected 5, got 2) > > Assertion failed: (!qemu_lockcnt_count(&ctx->list_lock)), function > > aio_ctx_finalize, file /home/qemu/qemu-test.kaRA0o/src/util/async.c, > > line 283. > > Abort trap (core dumped) > > gmake: *** [/home/qemu/qemu-test.kaRA0o/src/tests/Makefile.include:905: > > check-unit] Error 1 > > > > Anybody got any ideas? > > Here's a different intermittent assertion on Netbsd from test-aio-multithread: > > MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} > tests/test-aio-multithread -m=quick > -k --tap < /dev/null | ./scripts/tap-driver.pl > --test-name="test-aio-multithread" > PASS 1 test-aio-multithread /aio/multi/lifecycle > PASS 2 test-aio-multithread /aio/multi/schedule > assertion "QSLIST_EMPTY(&ctx->scheduled_coroutines)" failed: file > "util/async.c", line 279, function " > aio_ctx_finalize" > ERROR - too few tests run (expected 5, got 2) Just saw this QSLIST_EMPTY(&ctx->scheduled_coroutines) assertion from FreeBSD as well, so it's still around as a definite intermittent. thanks -- PMM ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: test-aio-multithread assertion 2019-10-14 10:26 ` Peter Maydell @ 2019-10-14 10:50 ` Paolo Bonzini 2019-11-19 17:55 ` Peter Maydell 0 siblings, 1 reply; 7+ messages in thread From: Paolo Bonzini @ 2019-10-14 10:50 UTC (permalink / raw) To: Peter Maydell, QEMU Developers, Stefan Hajnoczi On 14/10/19 12:26, Peter Maydell wrote: > On Thu, 5 Sep 2019 at 14:17, Peter Maydell <peter.maydell@linaro.org> wrote: >> >> On Wed, 4 Sep 2019 at 12:50, Peter Maydell <peter.maydell@linaro.org> wrote: >>> >>> Hi; I've been seeing intermittently on the BSDs this assertion >>> running test-aio-multithread as part of 'make check': >>> >>> MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} >>> tests/test-aio-multithread -m=quick -k --tap < /dev/null | >>> ./scripts/tap-drive >>> r.pl --test-name="test-aio-multithread" >>> PASS 1 test-aio-multithread /aio/multi/lifecycle >>> PASS 2 test-aio-multithread /aio/multi/schedule >>> ERROR - too few tests run (expected 5, got 2) >>> Assertion failed: (!qemu_lockcnt_count(&ctx->list_lock)), function >>> aio_ctx_finalize, file /home/qemu/qemu-test.kaRA0o/src/util/async.c, >>> line 283. >>> Abort trap (core dumped) >>> gmake: *** [/home/qemu/qemu-test.kaRA0o/src/tests/Makefile.include:905: >>> check-unit] Error 1 >>> >>> Anybody got any ideas? >> >> Here's a different intermittent assertion on Netbsd from test-aio-multithread: >> >> MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} >> tests/test-aio-multithread -m=quick >> -k --tap < /dev/null | ./scripts/tap-driver.pl >> --test-name="test-aio-multithread" >> PASS 1 test-aio-multithread /aio/multi/lifecycle >> PASS 2 test-aio-multithread /aio/multi/schedule >> assertion "QSLIST_EMPTY(&ctx->scheduled_coroutines)" failed: file >> "util/async.c", line 279, function " >> aio_ctx_finalize" >> ERROR - too few tests run (expected 5, got 2) > > Just saw this QSLIST_EMPTY(&ctx->scheduled_coroutines) assertion > from FreeBSD as well, so it's still around as a definite > intermittent. This could be similar to what Stefan fixed in commit 2362a28ea1, only in the test infrastructure. Paolo ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: test-aio-multithread assertion 2019-10-14 10:50 ` Paolo Bonzini @ 2019-11-19 17:55 ` Peter Maydell 2019-11-19 18:01 ` Peter Maydell 0 siblings, 1 reply; 7+ messages in thread From: Peter Maydell @ 2019-11-19 17:55 UTC (permalink / raw) To: Paolo Bonzini; +Cc: Alex Bennée, QEMU Developers, Stefan Hajnoczi On Mon, 14 Oct 2019 at 11:50, Paolo Bonzini <pbonzini@redhat.com> wrote: > > On 14/10/19 12:26, Peter Maydell wrote: > > On Thu, 5 Sep 2019 at 14:17, Peter Maydell <peter.maydell@linaro.org> wrote: > >> Here's a different intermittent assertion on Netbsd from test-aio-multithread: > >> > >> MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} > >> tests/test-aio-multithread -m=quick > >> -k --tap < /dev/null | ./scripts/tap-driver.pl > >> --test-name="test-aio-multithread" > >> PASS 1 test-aio-multithread /aio/multi/lifecycle > >> PASS 2 test-aio-multithread /aio/multi/schedule > >> assertion "QSLIST_EMPTY(&ctx->scheduled_coroutines)" failed: file > >> "util/async.c", line 279, function " > >> aio_ctx_finalize" > >> ERROR - too few tests run (expected 5, got 2) > > > > Just saw this QSLIST_EMPTY(&ctx->scheduled_coroutines) assertion > > from FreeBSD as well, so it's still around as a definite > > intermittent. > > This could be similar to what Stefan fixed in commit 2362a28ea1, only in > the test infrastructure. Here's a gdb backtrace from as assertion failure under NetBSD when running ./tests/test-aio-multithread -p /aio/multi/mutex/contended (which is pretty replicable for me); does it look like the same thing as the commit 2362a28ea1 issue? (Apologies for all the "optimized out" stuff in the backtraces; I'll try to rerun with a --enable-debug build.) Thread 7 (LWP 2): #0 0x00007e4b44499c0a in ___lwp_park60 () from /usr/lib/libc.so.12 #1 0x00007e4b44809ee3 in pthread_cond_timedwait () from /usr/lib/libpthread.so.1 #2 0x00000000004c0862 in qemu_futex_wait (val=4294967295, ev=0x760c00 <rcu_call_ready_event>) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-thread-posix.c:364 #3 qemu_event_wait (ev=ev@entry=0x760c00 <rcu_call_ready_event>) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-thread-posix.c:459 #4 0x00000000004ce825 in call_rcu_thread (opaque=opaque@entry=0x0) at /home/qemu/qemu-test.0WfUNX/src/util/rcu.c:260 #5 0x00000000004bfd17 in qemu_thread_start (args=0x7e4b47b22c00) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-thread-posix.c:519 #6 0x00007e4b4480b8b5 in ?? () from /usr/lib/libpthread.so.1 #7 0x00007e4b44481e20 in ?? () from /usr/lib/libc.so.12 #8 0x00010102464c457f in ?? () #9 0x0000000000000000 in ?? () Thread 6 (LWP 3): #0 0x00007e4b4443e02a in write () from /usr/lib/libc.so.12 #1 0x00007e4b44807561 in write () from /usr/lib/libpthread.so.1 #2 0x00000000004beae0 in event_notifier_set (e=e@entry=0x7e4b46f01214) at /home/qemu/qemu-test.0WfUNX/src/util/event_notifier-posix.c:100 #3 0x00000000004bb66c in aio_notify (ctx=0x7e4b46f01120) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:350 #4 0x00000000004bb936 in aio_co_schedule (ctx=0x7e4b46f01120, co=0x7e4b46b01280) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:473 #5 0x00000000004cf226 in qemu_co_mutex_wake (mutex=0x75fc00 <comutex>, co=<optimized out>) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:198 #6 qemu_co_mutex_unlock (mutex=mutex@entry=0x75fc00 <comutex>) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:301 #7 0x000000000040b33b in test_multi_co_mutex_entry (opaque=<optimized out>) at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:208 #8 0x00000000004cff6b in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at /home/qemu/qemu-test.0WfUNX/src/util/coroutine-ucontext.c:115 #9 0x00007e4b4445dba0 in ?? () from /usr/lib/libc.so.12 #10 0x00010102464c457f in ?? () #11 0x0000000000000000 in ?? () Thread 5 (LWP 4): #0 0x000000000040b34a in test_multi_co_mutex_entry (opaque=<optimized out>) at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:205 #1 0x00000000004cff6b in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at /home/qemu/qemu-test.0WfUNX/src/util/coroutine-ucontext.c:115 #2 0x00007e4b4445dba0 in ?? () from /usr/lib/libc.so.12 #3 0x00010102464c457f in ?? () #4 0x0000000000000000 in ?? () Thread 4 (LWP 5): #0 0x00007e4b4443e02a in write () from /usr/lib/libc.so.12 #1 0x00007e4b44807561 in write () from /usr/lib/libpthread.so.1 #2 0x00000000004beae0 in event_notifier_set (e=e@entry=0x7e4b47701384) at /home/qemu/qemu-test.0WfUNX/src/util/event_notifier-posix.c:100 #3 0x00000000004bb66c in aio_notify (ctx=0x7e4b47701290) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:350 #4 0x00000000004bb936 in aio_co_schedule (ctx=0x7e4b47701290, co=0x7e4b46b01420) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:473 #5 0x00000000004cf226 in qemu_co_mutex_wake (mutex=0x75fc00 <comutex>, co=<optimized out>) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:198 #6 qemu_co_mutex_unlock (mutex=mutex@entry=0x75fc00 <comutex>) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:301 #7 0x000000000040b33b in test_multi_co_mutex_entry (opaque=<optimized out>) at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:208 #8 0x00000000004cff6b in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at /home/qemu/qemu-test.0WfUNX/src/util/coroutine-ucontext.c:115 #9 0x00007e4b4445dba0 in ?? () from /usr/lib/libc.so.12 #10 0x00010102464c457f in ?? () #11 0x0000000000000000 in ?? () Thread 3 (LWP 7): #0 0x00000000004cf200 in qemu_co_mutex_unlock (mutex=mutex@entry=0x75fc00 <comutex>) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:291 #1 0x000000000040b33b in test_multi_co_mutex_entry (opaque=<optimized out>) at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:208 #2 0x00000000004cff6b in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at /home/qemu/qemu-test.0WfUNX/src/util/coroutine-ucontext.c:115 #3 0x00007e4b4445dba0 in ?? () from /usr/lib/libc.so.12 #4 0x00010102464c457f in ?? () #5 0x0000000000000000 in ?? () Thread 2 (LWP 6): #0 0x00007e4b444fe7ea in _lwp_kill () from /usr/lib/libc.so.12 #1 0x00007e4b444fe092 in abort () from /usr/lib/libc.so.12 #2 0x00007e4b4449fd57 in __assert13 () from /usr/lib/libc.so.12 #3 0x00000000004bb3e0 in aio_ctx_finalize (source=0x7e4b47701290) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:279 #4 0x00007e4b46445596 in g_source_unref_internal () from /usr/pkg/lib/libglib-2.0.so.0 #5 0x00000000004bb93e in aio_context_unref (ctx=0x7e4b47701290) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:516 #6 aio_co_schedule (ctx=0x7e4b47701290, co=0x7e4b46b01420) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:475 #7 0x00000000004cf226 in qemu_co_mutex_wake (mutex=0x75fc00 <comutex>, co=<optimized out>) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:198 #8 qemu_co_mutex_unlock (mutex=mutex@entry=0x75fc00 <comutex>) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:301 #9 0x000000000040b33b in test_multi_co_mutex_entry (opaque=<optimized out>) at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:208 #10 0x00000000004cff6b in coroutine_trampoline (i0=<optimized out>, i1=<optimized out>) at /home/qemu/qemu-test.0WfUNX/src/util/coroutine-ucontext.c:115 #11 0x00007e4b4445dba0 in ?? () from /usr/lib/libc.so.12 #12 0x00010102464c457f in ?? () #13 0x0000000000000000 in ?? () Thread 1 (LWP 1): #0 0x00007e4b4443e2ea in _sys___nanosleep50 () from /usr/lib/libc.so.12 #1 0x00007e4b44807287 in __nanosleep50 () from /usr/lib/libpthread.so.1 #2 0x00007e4b4646e511 in g_usleep () from /usr/pkg/lib/libglib-2.0.so.0 #3 0x000000000040b72e in test_multi_co_mutex (threads=<optimized out>, seconds=1) at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:237 #4 0x00007e4b4646bd15 in g_test_run_suite_internal () from /usr/pkg/lib/libglib-2.0.so.0 #5 0x00007e4b4646bc3e in g_test_run_suite_internal () from /usr/pkg/lib/libglib-2.0.so.0 #6 0x00007e4b4646bc3e in g_test_run_suite_internal () from /usr/pkg/lib/libglib-2.0.so.0 #7 0x00007e4b4646bc3e in g_test_run_suite_internal () from /usr/pkg/lib/libglib-2.0.so.0 #8 0x00007e4b4646c1a0 in g_test_run_suite () from /usr/pkg/lib/libglib-2.0.so.0 #9 0x00007e4b4646c1e8 in g_test_run () from /usr/pkg/lib/libglib-2.0.so.0 #10 0x00000000004dbcdc in main (argc=1, argv=0x7f7fffd572f8) at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:459 thanks -- PMM ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: test-aio-multithread assertion 2019-11-19 17:55 ` Peter Maydell @ 2019-11-19 18:01 ` Peter Maydell 0 siblings, 0 replies; 7+ messages in thread From: Peter Maydell @ 2019-11-19 18:01 UTC (permalink / raw) To: Paolo Bonzini; +Cc: Alex Bennée, QEMU Developers, Stefan Hajnoczi On Tue, 19 Nov 2019 at 17:55, Peter Maydell <peter.maydell@linaro.org> wrote: > Here's a gdb backtrace from as assertion failure under NetBSD > when running ./tests/test-aio-multithread -p /aio/multi/mutex/contended > (which is pretty replicable for me); does it look like the same > thing as the commit 2362a28ea1 issue? (Apologies for all the > "optimized out" stuff in the backtraces; I'll try to rerun with > a --enable-debug build.) ...that was easier than I thought it might be. Here's the --enable-debug version: Thread 7 (LWP 2): #0 0x000078b302099c0a in ___lwp_park60 () from /usr/lib/libc.so.12 #1 0x000078b302409ee3 in pthread_cond_timedwait () from /usr/lib/libpthread.so.1 #2 0x0000000000519388 in qemu_futex_wait (ev=0x7d26e0 <rcu_call_ready_event>, val=4294967295) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-thread-posix.c:364 #3 0x000000000051959d in qemu_event_wait (ev=0x7d26e0 <rcu_call_ready_event>) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-thread-posix.c:459 #4 0x00000000005305db in call_rcu_thread (opaque=0x0) at /home/qemu/qemu-test.0WfUNX/src/util/rcu.c:260 #5 0x000000000051969f in qemu_thread_start (args=0x78b305722c00) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-thread-posix.c:519 #6 0x000078b30240b8b5 in ?? () from /usr/lib/libpthread.so.1 #7 0x000078b302081e20 in ?? () from /usr/lib/libc.so.12 #8 0x00010102464c457f in ?? () #9 0x0000000000000000 in ?? () Thread 6 (LWP 3): #0 0x000078b30203e02a in write () from /usr/lib/libc.so.12 #1 0x000078b302407561 in write () from /usr/lib/libpthread.so.1 #2 0x0000000000516e26 in event_notifier_set (e=0x78b304701284) at /home/qemu/qemu-test.0WfUNX/src/util/event_notifier-posix.c:100 #3 0x0000000000511ccb in aio_notify (ctx=0x78b304701180) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:350 #4 0x000000000051183d in qemu_bh_schedule (bh=0x78b304717050) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:167 #5 0x0000000000512100 in aio_co_schedule (ctx=0x78b304701180, co=0x78b303701280) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:473 #6 0x0000000000512180 in aio_co_enter (ctx=0x78b304701180, co=0x78b303701280) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:494 #7 0x000000000051214f in aio_co_wake (co=0x78b303701280) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:488 #8 0x0000000000531b3e in qemu_co_mutex_wake (mutex=0x7d1700 <comutex>, co=0x78b303701280) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:198 #9 0x0000000000531eb2 in qemu_co_mutex_unlock (mutex=0x7d1700 <comutex>) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:301 #10 0x000000000040b8b5 in test_multi_co_mutex_entry (opaque=0x0) at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:208 #11 0x0000000000532bc1 in coroutine_trampoline (i0=57676000, i1=30899) at /home/qemu/qemu-test.0WfUNX/src/util/coroutine-ucontext.c:115 #12 0x000078b30205dba0 in ?? () from /usr/lib/libc.so.12 #13 0x00010102464c457f in ?? () #14 0x0000000000000000 in ?? () Thread 5 (LWP 5): #0 0x000078b30203e02a in write () from /usr/lib/libc.so.12 #1 0x000078b302407561 in write () from /usr/lib/libpthread.so.1 #2 0x0000000000516e26 in event_notifier_set (e=0x78b305316284) at /home/qemu/qemu-test.0WfUNX/src/util/event_notifier-posix.c:100 #3 0x0000000000511ccb in aio_notify (ctx=0x78b305316180) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:350 #4 0x000000000051183d in qemu_bh_schedule (bh=0x78b305301080) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:167 #5 0x0000000000512100 in aio_co_schedule (ctx=0x78b305316180, co=0x78b303701350) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:473 #6 0x0000000000512180 in aio_co_enter (ctx=0x78b305316180, co=0x78b303701350) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:494 #7 0x000000000051214f in aio_co_wake (co=0x78b303701350) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:488 #8 0x0000000000531b3e in qemu_co_mutex_wake (mutex=0x7d1700 <comutex>, co=0x78b303701350) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:198 #9 0x0000000000531eb2 in qemu_co_mutex_unlock (mutex=0x7d1700 <comutex>) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:301 #10 0x000000000040b8b5 in test_multi_co_mutex_entry (opaque=0x0) at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:208 #11 0x0000000000532bc1 in coroutine_trampoline (i0=57676416, i1=30899) at /home/qemu/qemu-test.0WfUNX/src/util/coroutine-ucontext.c:115 #12 0x000078b30205dba0 in ?? () from /usr/lib/libc.so.12 #13 0x00010102464c457f in ?? () #14 0x0000000000000000 in ?? () Thread 4 (LWP 6): #0 0x000078b30203e28a in poll () from /usr/lib/libc.so.12 #1 0x000078b3024072d0 in poll () from /usr/lib/libpthread.so.1 #2 0x0000000000513bf9 in qemu_poll_ns (fds=0x78b3053050c0, nfds=1, timeout=-1) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-timer.c:351 #3 0x000000000051665a in aio_poll (ctx=0x78b305316180, blocking=true) at /home/qemu/qemu-test.0WfUNX/src/util/aio-posix.c:669 #4 0x00000000004ddf5c in iothread_run (opaque=0x78b30530b270) at /home/qemu/qemu-test.0WfUNX/src/tests/iothread.c:51 #5 0x000000000051969f in qemu_thread_start (args=0x78b305303080) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-thread-posix.c:519 #6 0x000078b30240b8b5 in ?? () from /usr/lib/libpthread.so.1 #7 0x000078b302081e20 in ?? () from /usr/lib/libc.so.12 #8 0x0000000000000000 in ?? () Thread 3 (LWP 7): #0 0x000078b30203e28a in poll () from /usr/lib/libc.so.12 #1 0x000078b3024072d0 in poll () from /usr/lib/libpthread.so.1 #2 0x0000000000513bf9 in qemu_poll_ns (fds=0x78b303f22080, nfds=1, timeout=-1) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-timer.c:351 #3 0x000000000051665a in aio_poll (ctx=0x78b303f01180, blocking=true) at /home/qemu/qemu-test.0WfUNX/src/util/aio-posix.c:669 #4 0x00000000004ddf5c in iothread_run (opaque=0x78b30530b300) at /home/qemu/qemu-test.0WfUNX/src/tests/iothread.c:51 #5 0x000000000051969f in qemu_thread_start (args=0x78b305303080) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-thread-posix.c:519 #6 0x000078b30240b8b5 in ?? () from /usr/lib/libpthread.so.1 #7 0x000078b302081e20 in ?? () from /usr/lib/libc.so.12 #8 0x0000000000000000 in ?? () Thread 2 (LWP 4): #0 0x000078b3020fe7ea in _lwp_kill () from /usr/lib/libc.so.12 #1 0x000078b3020fe092 in abort () from /usr/lib/libc.so.12 #2 0x000078b30209fd57 in __assert13 () from /usr/lib/libc.so.12 #3 0x0000000000511ae8 in aio_ctx_finalize (source=0x78b305316180) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:279 #4 0x000078b304045596 in g_source_unref_internal () from /usr/pkg/lib/libglib-2.0.so.0 #5 0x0000000000512247 in aio_context_unref (ctx=0x78b305316180) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:516 #6 0x000000000051210c in aio_co_schedule (ctx=0x78b305316180, co=0x78b303701350) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:475 #7 0x0000000000512180 in aio_co_enter (ctx=0x78b305316180, co=0x78b303701350) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:494 #8 0x000000000051214f in aio_co_wake (co=0x78b303701350) at /home/qemu/qemu-test.0WfUNX/src/util/async.c:488 #9 0x0000000000531b3e in qemu_co_mutex_wake (mutex=0x7d1700 <comutex>, co=0x78b303701350) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:198 #10 0x0000000000531eb2 in qemu_co_mutex_unlock (mutex=0x7d1700 <comutex>) at /home/qemu/qemu-test.0WfUNX/src/util/qemu-coroutine-lock.c:301 #11 0x000000000040b8b5 in test_multi_co_mutex_entry (opaque=0x0) at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:208 #12 0x0000000000532bc1 in coroutine_trampoline (i0=57676208, i1=30899) at /home/qemu/qemu-test.0WfUNX/src/util/coroutine-ucontext.c:115 #13 0x000078b30205dba0 in ?? () from /usr/lib/libc.so.12 #14 0x00010102464c457f in ?? () #15 0x0000000000000000 in ?? () Thread 1 (LWP 1): #0 0x000078b30203e2ea in _sys___nanosleep50 () from /usr/lib/libc.so.12 #1 0x000078b302407287 in __nanosleep50 () from /usr/lib/libpthread.so.1 #2 0x000078b30406e511 in g_usleep () from /usr/pkg/lib/libglib-2.0.so.0 #3 0x000000000040b9ba in test_multi_co_mutex (threads=5, seconds=1) at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:237 #4 0x000000000040ba75 in test_multi_co_mutex_1 () at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:255 #5 0x000078b30406bd15 in g_test_run_suite_internal () from /usr/pkg/lib/libglib-2.0.so.0 #6 0x000078b30406bc3e in g_test_run_suite_internal () from /usr/pkg/lib/libglib-2.0.so.0 #7 0x000078b30406bc3e in g_test_run_suite_internal () from /usr/pkg/lib/libglib-2.0.so.0 #8 0x000078b30406bc3e in g_test_run_suite_internal () from /usr/pkg/lib/libglib-2.0.so.0 #9 0x000078b30406c1a0 in g_test_run_suite () from /usr/pkg/lib/libglib-2.0.so.0 #10 0x000078b30406c1e8 in g_test_run () from /usr/pkg/lib/libglib-2.0.so.0 #11 0x000000000040bdc3 in main (argc=1, argv=0x7f7fffc34528) at /home/qemu/qemu-test.0WfUNX/src/tests/test-aio-multithread.c:459 thanks -- PMM ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Qemu-devel] test-aio-multithread assertion 2019-09-04 11:50 [Qemu-devel] test-aio-multithread assertion Peter Maydell 2019-09-05 13:17 ` Peter Maydell @ 2019-09-16 15:33 ` Stefan Hajnoczi 1 sibling, 0 replies; 7+ messages in thread From: Stefan Hajnoczi @ 2019-09-16 15:33 UTC (permalink / raw) To: Peter Maydell; +Cc: Paolo Bonzini, QEMU Developers [-- Attachment #1: Type: text/plain, Size: 1230 bytes --] On Wed, Sep 04, 2019 at 12:50:21PM +0100, Peter Maydell wrote: > Hi; I've been seeing intermittently on the BSDs this assertion > running test-aio-multithread as part of 'make check': > > MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} > tests/test-aio-multithread -m=quick -k --tap < /dev/null | > ./scripts/tap-drive > r.pl --test-name="test-aio-multithread" > PASS 1 test-aio-multithread /aio/multi/lifecycle > PASS 2 test-aio-multithread /aio/multi/schedule > ERROR - too few tests run (expected 5, got 2) > Assertion failed: (!qemu_lockcnt_count(&ctx->list_lock)), function > aio_ctx_finalize, file /home/qemu/qemu-test.kaRA0o/src/util/async.c, > line 283. > Abort trap (core dumped) > gmake: *** [/home/qemu/qemu-test.kaRA0o/src/tests/Makefile.include:905: > check-unit] Error 1 > > Anybody got any ideas? > > (this is with the tests/vm setups; I've seen this on FreeBSD > and NetBSD.) Last week I triaged a ppc64le test-aio-multithread test_multi_fair_mutex failure (race condition but reproduces after a few minutes) and sent it on to Paolo. I don't have time to investigate additional failures right now but please keep reminding us if you encounter failures. Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2019-11-19 18:04 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-09-04 11:50 [Qemu-devel] test-aio-multithread assertion Peter Maydell 2019-09-05 13:17 ` Peter Maydell 2019-10-14 10:26 ` Peter Maydell 2019-10-14 10:50 ` Paolo Bonzini 2019-11-19 17:55 ` Peter Maydell 2019-11-19 18:01 ` Peter Maydell 2019-09-16 15:33 ` [Qemu-devel] " Stefan Hajnoczi
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.