* intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host
@ 2021-07-10 13:30 Peter Maydell
2021-07-11 12:08 ` Coiby Xu
` (2 more replies)
0 siblings, 3 replies; 14+ messages in thread
From: Peter Maydell @ 2021-07-10 13:30 UTC (permalink / raw)
To: QEMU Developers; +Cc: Kevin Wolf, Coiby Xu, Raphael Norwitz
I've noticed recently that intermittently 'make check' will hang on
my aarch32 test system (really an aarch64 box with an aarch32 chroot).
I think from grep that this must be the vhost-user-blk test.
Here's the process tree:
pmaydell 13126 0.0 0.0 8988 6416 ? S Jul09 0:01 make
-C build/all-a32 check V=1 GCC_COLORS= -j9
pmaydell 19632 0.0 0.0 4432 2096 ? S Jul09 0:00 \_
bash -o pipefail -c echo 'MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((
${RANDOM:-0} % 255 + 1))} QTEST_QEMU_IMG=./qemu-img
G_TEST_DBUS_DAEMON=/home/peter.maydell/qemu/tests/dbus-vmstate-daemon.sh
QTEST_QEMU_BINARY=./qemu-system-i386
QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon
tests/qtest/qos-test --tap -k' &&
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}
QTEST_QEMU_IMG=./qemu-img
G_TEST_DBUS_DAEMON=/home/peter.maydell/qemu/tests/dbus-vmstate-daemon.sh
QTEST_QEMU_BINARY=./qemu-system-i386
QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon
tests/qtest/qos-test --tap -k -m quick < /dev/null |
./scripts/tap-driver.pl --test-name="qtest-i386/qos-test"
pmaydell 19634 0.0 0.0 13608 3076 ? Sl Jul09 0:02
\_ tests/qtest/qos-test --tap -k -m quick
pmaydell 20679 0.0 0.0 109076 16100 ? Sl Jul09 0:00
| \_ ./storage-daemon/qemu-storage-daemon --blockdev
driver=file,node-name=disk0,filename=qtest.X7RL2X --export
type=vhost-user-blk,id=disk0,addr.type=unix,addr.path=/tmp/qtest-19634-sock.9LJoHn,node-name=disk0,writable=on,num-queues=1
pmaydell 20681 0.0 0.2 447828 46544 ? Sl Jul09 0:00
| \_ ./qemu-system-i386 -qtest unix:/tmp/qtest-19634.sock -qtest-log
/dev/null -chardev socket,path=/tmp/qtest-19634.qmp,id=char0 -mon
chardev=char0,mode=control -display none -M pc -device
vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
-m 256M -chardev socket,id=char1,path=/tmp/qtest-19634-sock.9LJoHn
-accel qtest
pmaydell 19635 0.0 0.0 10256 7176 ? S Jul09 0:00
\_ perl ./scripts/tap-driver.pl --test-name=qtest-i386/qos-test
Backtrace from tests/qtest/qos-test (not as helpful as it could
be since this is an optimized build):
(gdb) thread apply all bt
Thread 2 (Thread 0xf76ff240 (LWP 19636)):
#0 syscall () at ../sysdeps/unix/sysv/linux/arm/syscall.S:37
#1 0x005206de in qemu_futex_wait (val=<optimised out>, f=<optimised
out>) at /home/peter.maydell/qemu/include/qemu/futex.h:29
#2 qemu_event_wait (ev=ev@entry=0x5816fc <rcu_call_ready_event>) at
../../util/qemu-thread-posix.c:480
#3 0x005469c2 in call_rcu_thread (opaque=<optimised out>) at
../../util/rcu.c:258
#4 0x0051fbc2 in qemu_thread_start (args=<optimised out>) at
../../util/qemu-thread-posix.c:541
#5 0xf785a614 in start_thread (arg=0xf6ce711c) at pthread_create.c:463
#6 0xf77f57ec in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Thread 1 (Thread 0xf7a04010 (LWP 19634)):
#0 __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
#1 0xf7861d8c in __libc_read (fd=12, buf=buf@entry=0xff9ce8e4,
nbytes=nbytes@entry=1024) at ../sysdeps/unix/sysv/linux/read.c:27
#2 0x004ebc5a in read (__nbytes=1024, __buf=0xff9ce8e4,
__fd=<optimised out>) at
/usr/include/arm-linux-gnueabihf/bits/unistd.h:44
#3 qtest_client_socket_recv_line (s=0x1a46cb8) at
../../tests/qtest/libqtest.c:494
#4 0x004ebd4e in qtest_rsp_args (s=s@entry=0x1a46cb8,
expected_args=expected_args@entry=1) at
../../tests/qtest/libqtest.c:521
#5 0x004ec1ee in qtest_query_target_endianness (s=0x1a46cb8) at
../../tests/qtest/libqtest.c:570
#6 0x004ec94a in qtest_init_without_qmp_handshake
(extra_args=<optimised out>) at ../../tests/qtest/libqtest.c:332
#7 0x004ecd7a in qtest_init (extra_args=<optimised out>) at
../../tests/qtest/libqtest.c:339
#8 0x004ded10 in qtest_start (
args=0x1a63710 "-M pc -device
vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
-m 256M -chardev socket,id=char1,path=/tmp/qtest-19634-so"...) at
../../tests/qtest/libqtest-single.h:29
#9 restart_qemu_or_continue (
path=0x1a63710 "-M pc -device
vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
-m 256M -chardev socket,id=char1,path=/tmp/qtest-19634-so"...) at
../../tests/qtest/qos-test.c:105
#10 run_one_test (arg=<optimised out>) at ../../tests/qtest/qos-test.c:178
#11 0xf794ee74 in ?? () from /usr/lib/arm-linux-gnueabihf/libglib-2.0.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Backtrace from qemu-system-i386:
(gdb) thread apply all bt
Thread 4 (Thread 0xdfd0cb90 (LWP 20734)):
#0 0xf6f85206 in __libc_do_syscall () at
../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47
#1 0xf6f93492 in __GI___sigtimedwait (set=<optimised out>,
set@entry=0xdfd0c3c4, info=info@entry=0xdfd0c324,
timeout=timeout@entry=0x0) at
../sysdeps/unix/sysv/linux/sigtimedwait.c:42
#2 0xf7073e6c in __sigwait (set=set@entry=0xdfd0c3c4,
sig=sig@entry=0xdfd0c3c0) at ../sysdeps/unix/sysv/linux/sigwait.c:28
#3 0x00c999a6 in dummy_cpu_thread_fn (arg=0x24d1168) at
../../accel/dummy-cpus.c:46
#4 0x00e18ba2 in qemu_thread_start (args=<optimised out>) at
../../util/qemu-thread-posix.c:541
#5 0xf706b614 in start_thread (arg=0xa8db8843) at pthread_create.c:463
#6 0xf70067ec in () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
Thread 3 (Thread 0xe07fcb90 (LWP 20710)):
#0 0xf6f85204 in __libc_do_syscall () at
../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
#1 0xf6fff948 in __GI___poll (fds=0x2409b90, nfds=3, timeout=-1) at
../sysdeps/unix/sysv/linux/poll.c:29
#2 0xf781692e in () at /usr/lib/arm-linux-gnueabihf/libglib-2.0.so.0
Thread 2 (Thread 0xf1201b90 (LWP 20703)):
#0 0xf7074f04 in __libc_do_syscall () at
../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
#1 0xf7072ad2 in __lll_lock_wait (futex=futex@entry=0x14f2e68
<qemu_global_mutex>, private=0) at lowlevellock.c:43
#2 0xf706d3e2 in __GI___pthread_mutex_lock
(mutex=mutex@entry=0x14f2e68 <qemu_global_mutex>) at
pthread_mutex_lock.c:78
#3 0x00e18c9c in qemu_mutex_lock_impl (mutex=0x14f2e68
<qemu_global_mutex>, file=0x1355bd4 "../../util/rcu.c", line=266)
at ../../util/qemu-thread-posix.c:79
#4 0x00c1403a in qemu_mutex_lock_iothread_impl (file=0x1355bd4
"../../util/rcu.c", line=line@entry=266) at ../../softmmu/cpus.c:493
#5 0x00e15ad4 in call_rcu_thread (opaque=<optimised out>) at
../../util/rcu.c:266
#6 0x00e18ba2 in qemu_thread_start (args=<optimised out>) at
../../util/qemu-thread-posix.c:541
#7 0xf706b614 in start_thread (arg=0xa8db8843) at pthread_create.c:463
#8 0xf70067ec in () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
Thread 1 (Thread 0xf12f9010 (LWP 20681)):
#0 0xf6f85204 in __libc_do_syscall () at
../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
#1 0xf6fff948 in __GI___poll (fds=0x3248040, nfds=2, timeout=-1) at
../sysdeps/unix/sysv/linux/poll.c:29
#2 0xf781692e in () at /usr/lib/arm-linux-gnueabihf/libglib-2.0.so.0
Backtrace of qemu-storage-daemon:
(gdb) thread apply all bt
Thread 2 (Thread 0xf21ffdd0 (LWP 20722)):
#0 syscall () at ../sysdeps/unix/sysv/linux/arm/syscall.S:37
#1 0x009a9986 in qemu_futex_wait (val=<optimised out>, f=<optimised
out>) at /home/peter.maydell/qemu/include/qemu/futex.h:29
#2 qemu_event_wait (ev=ev@entry=0xa64958 <rcu_call_ready_event>) at
../../util/qemu-thread-posix.c:480
#3 0x009a5d66 in call_rcu_thread (opaque=<optimised out>) at
../../util/rcu.c:258
#4 0x009a8e6a in qemu_thread_start (args=<optimised out>) at
../../util/qemu-thread-posix.c:541
#5 0xf79dd614 in start_thread (arg=0xa310c11f) at pthread_create.c:463
#6 0xf79787ec in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
Thread 1 (Thread 0xf22f7010 (LWP 20679)):
#0 __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
#1 0xf7971a04 in __GI_ppoll (fds=0x117c578, nfds=5,
timeout=<optimised out>, timeout@entry=0x0, sigmask=sigmask@entry=0x0)
at ../sysdeps/unix/sysv/linux/ppoll.c:39
#2 0x009a68fc in ppoll (__ss=0x0, __timeout=0x0, __nfds=<optimised
out>, __fds=<optimised out>)
at /usr/include/arm-linux-gnueabihf/bits/poll2.h:77
#3 qemu_poll_ns (fds=<optimised out>, nfds=<optimised out>,
timeout=<optimised out>) at ../../util/qemu-timer.c:336
#4 0x009c0222 in os_host_main_loop_wait (timeout=-1) at
../../util/main-loop.c:250
#5 main_loop_wait (nonblocking=<optimised out>) at ../../util/main-loop.c:531
#6 0x008ee92a in main (argc=5, argv=0xffdce5c4) at
../../storage-daemon/qemu-storage-daemon.c:345
Any ideas ?
thanks
-- PMM
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host
2021-07-10 13:30 intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host Peter Maydell
@ 2021-07-11 12:08 ` Coiby Xu
2021-07-11 12:16 ` Peter Maydell
2021-07-11 15:53 ` Peter Maydell
2021-07-22 8:45 ` Claudio Fontana
2 siblings, 1 reply; 14+ messages in thread
From: Coiby Xu @ 2021-07-11 12:08 UTC (permalink / raw)
To: Peter Maydell; +Cc: Kevin Wolf, QEMU Developers, Raphael Norwitz
Hi Peter,
On Sat, Jul 10, 2021 at 02:30:36PM +0100, Peter Maydell wrote:
>I've noticed recently that intermittently 'make check' will hang on
>my aarch32 test system (really an aarch64 box with an aarch32 chroot).
I have a newbie question. How do you do an aarch32 chroot on an aarch64
box? At least, this issue seems to be not reproducible on an aarch64 box
directly. I specifically ran the qos-test for 5 consecutive times and
each time the test could finish successfully,
$ MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} \
QTEST_QEMU_BINARY=build/i386-softmmu/qemu-system-i386 QTEST_QEMU_IMG=build/qemu-img \
QTEST_QEMU_STORAGE_DAEMON_BINARY=build/storage-daemon/qemu-storage-daemon \
build/tests/qtest/qos-test
>
>I think from grep that this must be the vhost-user-blk test.
>
>Here's the process tree:
>
>pmaydell 13126 0.0 0.0 8988 6416 ? S Jul09 0:01 make
>-C build/all-a32 check V=1 GCC_COLORS= -j9
>pmaydell 19632 0.0 0.0 4432 2096 ? S Jul09 0:00 \_
>bash -o pipefail -c echo 'MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((
>${RANDOM:-0} % 255 + 1))} QTEST_QEMU_IMG=./qemu-img
>G_TEST_DBUS_DAEMON=/home/peter.maydell/qemu/tests/dbus-vmstate-daemon.sh
>QTEST_QEMU_BINARY=./qemu-system-i386
>QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon
>tests/qtest/qos-test --tap -k' &&
>MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}
>QTEST_QEMU_IMG=./qemu-img
>G_TEST_DBUS_DAEMON=/home/peter.maydell/qemu/tests/dbus-vmstate-daemon.sh
>QTEST_QEMU_BINARY=./qemu-system-i386
>QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon
>tests/qtest/qos-test --tap -k -m quick < /dev/null |
>./scripts/tap-driver.pl --test-name="qtest-i386/qos-test"
>pmaydell 19634 0.0 0.0 13608 3076 ? Sl Jul09 0:02
>\_ tests/qtest/qos-test --tap -k -m quick
>pmaydell 20679 0.0 0.0 109076 16100 ? Sl Jul09 0:00
>| \_ ./storage-daemon/qemu-storage-daemon --blockdev
>driver=file,node-name=disk0,filename=qtest.X7RL2X --export
>type=vhost-user-blk,id=disk0,addr.type=unix,addr.path=/tmp/qtest-19634-sock.9LJoHn,node-name=disk0,writable=on,num-queues=1
>pmaydell 20681 0.0 0.2 447828 46544 ? Sl Jul09 0:00
>| \_ ./qemu-system-i386 -qtest unix:/tmp/qtest-19634.sock -qtest-log
>/dev/null -chardev socket,path=/tmp/qtest-19634.qmp,id=char0 -mon
>chardev=char0,mode=control -display none -M pc -device
>vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
>memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
>-m 256M -chardev socket,id=char1,path=/tmp/qtest-19634-sock.9LJoHn
>-accel qtest
>pmaydell 19635 0.0 0.0 10256 7176 ? S Jul09 0:00
>\_ perl ./scripts/tap-driver.pl --test-name=qtest-i386/qos-test
>
>
>Backtrace from tests/qtest/qos-test (not as helpful as it could
>be since this is an optimized build):
>
>(gdb) thread apply all bt
>
>Thread 2 (Thread 0xf76ff240 (LWP 19636)):
>#0 syscall () at ../sysdeps/unix/sysv/linux/arm/syscall.S:37
>#1 0x005206de in qemu_futex_wait (val=<optimised out>, f=<optimised
>out>) at /home/peter.maydell/qemu/include/qemu/futex.h:29
>#2 qemu_event_wait (ev=ev@entry=0x5816fc <rcu_call_ready_event>) at
>../../util/qemu-thread-posix.c:480
>#3 0x005469c2 in call_rcu_thread (opaque=<optimised out>) at
>../../util/rcu.c:258
>#4 0x0051fbc2 in qemu_thread_start (args=<optimised out>) at
>../../util/qemu-thread-posix.c:541
>#5 0xf785a614 in start_thread (arg=0xf6ce711c) at pthread_create.c:463
>#6 0xf77f57ec in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
>from /lib/arm-linux-gnueabihf/libc.so.6
>Backtrace stopped: previous frame identical to this frame (corrupt stack?)
>
>Thread 1 (Thread 0xf7a04010 (LWP 19634)):
>#0 __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
>#1 0xf7861d8c in __libc_read (fd=12, buf=buf@entry=0xff9ce8e4,
>nbytes=nbytes@entry=1024) at ../sysdeps/unix/sysv/linux/read.c:27
>#2 0x004ebc5a in read (__nbytes=1024, __buf=0xff9ce8e4,
>__fd=<optimised out>) at
>/usr/include/arm-linux-gnueabihf/bits/unistd.h:44
>#3 qtest_client_socket_recv_line (s=0x1a46cb8) at
>../../tests/qtest/libqtest.c:494
>#4 0x004ebd4e in qtest_rsp_args (s=s@entry=0x1a46cb8,
>expected_args=expected_args@entry=1) at
>../../tests/qtest/libqtest.c:521
>#5 0x004ec1ee in qtest_query_target_endianness (s=0x1a46cb8) at
>../../tests/qtest/libqtest.c:570
>#6 0x004ec94a in qtest_init_without_qmp_handshake
>(extra_args=<optimised out>) at ../../tests/qtest/libqtest.c:332
>#7 0x004ecd7a in qtest_init (extra_args=<optimised out>) at
>../../tests/qtest/libqtest.c:339
>#8 0x004ded10 in qtest_start (
> args=0x1a63710 "-M pc -device
>vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
>memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
>-m 256M -chardev socket,id=char1,path=/tmp/qtest-19634-so"...) at
>../../tests/qtest/libqtest-single.h:29
>#9 restart_qemu_or_continue (
> path=0x1a63710 "-M pc -device
>vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
>memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
>-m 256M -chardev socket,id=char1,path=/tmp/qtest-19634-so"...) at
>../../tests/qtest/qos-test.c:105
>#10 run_one_test (arg=<optimised out>) at ../../tests/qtest/qos-test.c:178
>#11 0xf794ee74 in ?? () from /usr/lib/arm-linux-gnueabihf/libglib-2.0.so.0
>Backtrace stopped: previous frame identical to this frame (corrupt stack?)
>
>
>Backtrace from qemu-system-i386:
>
>(gdb) thread apply all bt
>
>Thread 4 (Thread 0xdfd0cb90 (LWP 20734)):
>#0 0xf6f85206 in __libc_do_syscall () at
>../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47
>#1 0xf6f93492 in __GI___sigtimedwait (set=<optimised out>,
> set@entry=0xdfd0c3c4, info=info@entry=0xdfd0c324,
>timeout=timeout@entry=0x0) at
>../sysdeps/unix/sysv/linux/sigtimedwait.c:42
>#2 0xf7073e6c in __sigwait (set=set@entry=0xdfd0c3c4,
>sig=sig@entry=0xdfd0c3c0) at ../sysdeps/unix/sysv/linux/sigwait.c:28
>#3 0x00c999a6 in dummy_cpu_thread_fn (arg=0x24d1168) at
>../../accel/dummy-cpus.c:46
>#4 0x00e18ba2 in qemu_thread_start (args=<optimised out>) at
>../../util/qemu-thread-posix.c:541
>#5 0xf706b614 in start_thread (arg=0xa8db8843) at pthread_create.c:463
>#6 0xf70067ec in () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
>
>Thread 3 (Thread 0xe07fcb90 (LWP 20710)):
>#0 0xf6f85204 in __libc_do_syscall () at
>../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
>#1 0xf6fff948 in __GI___poll (fds=0x2409b90, nfds=3, timeout=-1) at
>../sysdeps/unix/sysv/linux/poll.c:29
>#2 0xf781692e in () at /usr/lib/arm-linux-gnueabihf/libglib-2.0.so.0
>
>Thread 2 (Thread 0xf1201b90 (LWP 20703)):
>#0 0xf7074f04 in __libc_do_syscall () at
>../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
>#1 0xf7072ad2 in __lll_lock_wait (futex=futex@entry=0x14f2e68
><qemu_global_mutex>, private=0) at lowlevellock.c:43
>#2 0xf706d3e2 in __GI___pthread_mutex_lock
>(mutex=mutex@entry=0x14f2e68 <qemu_global_mutex>) at
>pthread_mutex_lock.c:78
>#3 0x00e18c9c in qemu_mutex_lock_impl (mutex=0x14f2e68
><qemu_global_mutex>, file=0x1355bd4 "../../util/rcu.c", line=266)
> at ../../util/qemu-thread-posix.c:79
>#4 0x00c1403a in qemu_mutex_lock_iothread_impl (file=0x1355bd4
>"../../util/rcu.c", line=line@entry=266) at ../../softmmu/cpus.c:493
>#5 0x00e15ad4 in call_rcu_thread (opaque=<optimised out>) at
>../../util/rcu.c:266
>#6 0x00e18ba2 in qemu_thread_start (args=<optimised out>) at
>../../util/qemu-thread-posix.c:541
>#7 0xf706b614 in start_thread (arg=0xa8db8843) at pthread_create.c:463
>#8 0xf70067ec in () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
>
>Thread 1 (Thread 0xf12f9010 (LWP 20681)):
>#0 0xf6f85204 in __libc_do_syscall () at
>../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
>#1 0xf6fff948 in __GI___poll (fds=0x3248040, nfds=2, timeout=-1) at
>../sysdeps/unix/sysv/linux/poll.c:29
>#2 0xf781692e in () at /usr/lib/arm-linux-gnueabihf/libglib-2.0.so.0
>
>Backtrace of qemu-storage-daemon:
>
>(gdb) thread apply all bt
>
>Thread 2 (Thread 0xf21ffdd0 (LWP 20722)):
>#0 syscall () at ../sysdeps/unix/sysv/linux/arm/syscall.S:37
>#1 0x009a9986 in qemu_futex_wait (val=<optimised out>, f=<optimised
>out>) at /home/peter.maydell/qemu/include/qemu/futex.h:29
>#2 qemu_event_wait (ev=ev@entry=0xa64958 <rcu_call_ready_event>) at
>../../util/qemu-thread-posix.c:480
>#3 0x009a5d66 in call_rcu_thread (opaque=<optimised out>) at
>../../util/rcu.c:258
>#4 0x009a8e6a in qemu_thread_start (args=<optimised out>) at
>../../util/qemu-thread-posix.c:541
>#5 0xf79dd614 in start_thread (arg=0xa310c11f) at pthread_create.c:463
>#6 0xf79787ec in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
>from /lib/arm-linux-gnueabihf/libc.so.6
>Backtrace stopped: previous frame identical to this frame (corrupt stack?)
>
>Thread 1 (Thread 0xf22f7010 (LWP 20679)):
>#0 __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
>#1 0xf7971a04 in __GI_ppoll (fds=0x117c578, nfds=5,
>timeout=<optimised out>, timeout@entry=0x0, sigmask=sigmask@entry=0x0)
> at ../sysdeps/unix/sysv/linux/ppoll.c:39
>#2 0x009a68fc in ppoll (__ss=0x0, __timeout=0x0, __nfds=<optimised
>out>, __fds=<optimised out>)
> at /usr/include/arm-linux-gnueabihf/bits/poll2.h:77
>#3 qemu_poll_ns (fds=<optimised out>, nfds=<optimised out>,
>timeout=<optimised out>) at ../../util/qemu-timer.c:336
>#4 0x009c0222 in os_host_main_loop_wait (timeout=-1) at
>../../util/main-loop.c:250
>#5 main_loop_wait (nonblocking=<optimised out>) at ../../util/main-loop.c:531
>#6 0x008ee92a in main (argc=5, argv=0xffdce5c4) at
>../../storage-daemon/qemu-storage-daemon.c:345
>
>
>Any ideas ?
>
>thanks
>-- PMM
--
Best regards,
Coiby
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host
2021-07-11 12:08 ` Coiby Xu
@ 2021-07-11 12:16 ` Peter Maydell
2021-07-11 13:23 ` Richard Henderson
0 siblings, 1 reply; 14+ messages in thread
From: Peter Maydell @ 2021-07-11 12:16 UTC (permalink / raw)
To: Coiby Xu; +Cc: Kevin Wolf, QEMU Developers, Raphael Norwitz
On Sun, 11 Jul 2021 at 13:10, Coiby Xu <coiby.xu@gmail.com> wrote:
>
> Hi Peter,
>
> On Sat, Jul 10, 2021 at 02:30:36PM +0100, Peter Maydell wrote:
> >I've noticed recently that intermittently 'make check' will hang on
> >my aarch32 test system (really an aarch64 box with an aarch32 chroot).
>
> I have a newbie question. How do you do an aarch32 chroot on an aarch64
> box? At least, this issue seems to be not reproducible on an aarch64 box
> directly. I specifically ran the qos-test for 5 consecutive times and
> each time the test could finish successfully,
Your aarch64 host CPU needs to support aarch32 at EL0 (some
AArch64 CPUs are pure-64 bit these days). The host kernel needs
to implement the 32-bit compat layer. It probably also needs to be
built for 4K pages (which mostly means "not RedHat"). Then you can
set up the 32-bit chroot however you'd normally set up a chroot
(for Debian you can do this with debootstrap; other distros will vary;
schroot is also a bit nicer than raw chroot IMHO.)
-- PMM
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host
2021-07-11 12:16 ` Peter Maydell
@ 2021-07-11 13:23 ` Richard Henderson
2021-07-11 14:21 ` Peter Maydell
2021-07-11 22:19 ` Coiby Xu
0 siblings, 2 replies; 14+ messages in thread
From: Richard Henderson @ 2021-07-11 13:23 UTC (permalink / raw)
To: Peter Maydell, Coiby Xu; +Cc: Kevin Wolf, QEMU Developers, Raphael Norwitz
On 7/11/21 5:16 AM, Peter Maydell wrote:
> On Sun, 11 Jul 2021 at 13:10, Coiby Xu <coiby.xu@gmail.com> wrote:
>>
>> Hi Peter,
>>
>> On Sat, Jul 10, 2021 at 02:30:36PM +0100, Peter Maydell wrote:
>>> I've noticed recently that intermittently 'make check' will hang on
>>> my aarch32 test system (really an aarch64 box with an aarch32 chroot).
>>
>> I have a newbie question. How do you do an aarch32 chroot on an aarch64
>> box? At least, this issue seems to be not reproducible on an aarch64 box
>> directly. I specifically ran the qos-test for 5 consecutive times and
>> each time the test could finish successfully,
>
> Your aarch64 host CPU needs to support aarch32 at EL0 (some
> AArch64 CPUs are pure-64 bit these days). The host kernel needs
> to implement the 32-bit compat layer. It probably also needs to be
> built for 4K pages (which mostly means "not RedHat"). Then you can
> set up the 32-bit chroot however you'd normally set up a chroot
> (for Debian you can do this with debootstrap; other distros will vary;
> schroot is also a bit nicer than raw chroot IMHO.)
If you do have a kernel built with 64k pages ("RedHat"), but you do have a host cpu that
supports aarch32 at EL1 and EL0, then you can run aarch32 under KVM.
The command-line I use is
../run/bin/qemu-system-aarch64 -m 4096 -smp 8 -nographic \
-M virt -cpu host,aarch64=off --accel kvm \
-kernel vmlinuz-4.19.0-16-armmp-lpae \
-initrd initrd.img-4.19.0-16-armmp-lpae \
-append 'console=ttyAMA0 root=/dev/vda2' \
-drive if=none,file=hda.q,format=qcow2,id=hd,discard=on \
-device virtio-blk-device,drive=hd \
-netdev tap,id=tap0,br=virbr0,helper=/usr/libexec/qemu-bridge-helper \
-device virtio-net-device,netdev=tap0
I believe that I had to perform the install under tcg because I couldn't find the right
magic to boot off the debian cdrom with kvm.
r~
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host
2021-07-11 13:23 ` Richard Henderson
@ 2021-07-11 14:21 ` Peter Maydell
2021-07-11 14:27 ` Richard Henderson
2021-07-11 22:19 ` Coiby Xu
1 sibling, 1 reply; 14+ messages in thread
From: Peter Maydell @ 2021-07-11 14:21 UTC (permalink / raw)
To: Richard Henderson; +Cc: Kevin Wolf, QEMU Developers, Coiby Xu, Raphael Norwitz
On Sun, 11 Jul 2021 at 14:23, Richard Henderson
<richard.henderson@linaro.org> wrote:
> I believe that I had to perform the install under tcg because I couldn't find the right
> magic to boot off the debian cdrom with kvm.
Weird, it ought not in theory to care...
-- PMM
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host
2021-07-11 14:21 ` Peter Maydell
@ 2021-07-11 14:27 ` Richard Henderson
0 siblings, 0 replies; 14+ messages in thread
From: Richard Henderson @ 2021-07-11 14:27 UTC (permalink / raw)
To: Peter Maydell; +Cc: Kevin Wolf, QEMU Developers, Coiby Xu, Raphael Norwitz
On 7/11/21 7:21 AM, Peter Maydell wrote:
> On Sun, 11 Jul 2021 at 14:23, Richard Henderson
> <richard.henderson@linaro.org> wrote:
>> I believe that I had to perform the install under tcg because I couldn't find the right
>> magic to boot off the debian cdrom with kvm.
>
> Weird, it ought not in theory to care...
Looking back at the install script I used, I had u-boot boot off the cdrom, and I'm
booting the kernel directly for kvm. I guess there's something about the specific u-boot
image I had that didn't work with kvm. It has been long enough that I don't recall any
further details.
r~
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host
2021-07-10 13:30 intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host Peter Maydell
2021-07-11 12:08 ` Coiby Xu
@ 2021-07-11 15:53 ` Peter Maydell
2021-07-11 22:20 ` Coiby Xu
2021-07-16 14:31 ` Kevin Wolf
2021-07-22 8:45 ` Claudio Fontana
2 siblings, 2 replies; 14+ messages in thread
From: Peter Maydell @ 2021-07-11 15:53 UTC (permalink / raw)
To: QEMU Developers; +Cc: Kevin Wolf, Coiby Xu, Raphael Norwitz
On Sat, 10 Jul 2021 at 14:30, Peter Maydell <peter.maydell@linaro.org> wrote:
>
> I've noticed recently that intermittently 'make check' will hang on
> my aarch32 test system (really an aarch64 box with an aarch32 chroot).
>
> I think from grep that this must be the vhost-user-blk test.
I've also now seen this on qemu-system-i386 guest x86-64 Linux host:
Process tree:
petmay01 28992 0.0 0.0 123812 8612 ? Sl 14:46 0:01
\_ tests/qtest/qos-test --tap -k -m quick
petmay01 30068 0.0 0.0 379204 20580 ? Sl 14:46 0:00
| \_ ./storage-daemon/qemu-storage-daemon
--blockdev driver=file,node-name=disk0,filename=qtest.6kY6px --export
type=vhost-user-blk,id=disk0,addr.type=unix,addr.path=/tmp/qtest-28992-sock.4Kgtk1,node-name=disk0,writable=on,num-queues=1
petmay01 30070 0.0 0.1 1083248 63748 ? Sl 14:46 0:00
| \_ ./qemu-system-i386 -qtest
unix:/tmp/qtest-28992.sock -qtest-log /dev/null -chardev
socket,path=/tmp/qtest-28992.qmp,id=char0 -mon
chardev=char0,mode=control -display none -M pc -device
vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
-m 256M -chardev socket,id=char1,path=/tmp/qtest-28992-sock.4Kgtk1
-accel qtest
Backtrace, qos-test:
(gdb) thread apply all bt
Thread 2 (Thread 0x7fd086f1c700 (LWP 28995)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x000056448599484b in qemu_futex_wait (val=<optimised out>,
f=<optimised out>)
at /mnt/nvmedisk/linaro/qemu-for-merges/include/qemu/futex.h:29
#2 qemu_event_wait (ev=ev@entry=0x564485c322e8 <rcu_call_ready_event>)
at ../../util/qemu-thread-posix.c:480
#3 0x000056448599dc18 in call_rcu_thread (opaque=opaque@entry=0x0) at
../../util/rcu.c:258
#4 0x0000564485993966 in qemu_thread_start (args=<optimised out>)
at ../../util/qemu-thread-posix.c:541
#5 0x00007fd088b446db in start_thread (arg=0x7fd086f1c700) at
pthread_create.c:463
#6 0x00007fd08886d71f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 1 (Thread 0x7fd089d9a900 (LWP 28992)):
#0 0x00007fd088b4e474 in __libc_read (fd=6,
buf=buf@entry=0x7fff05f024f0, nbytes=nbytes@entry=1024)
at ../sysdeps/unix/sysv/linux/read.c:27
#1 0x0000564485947cb2 in read (__nbytes=1024, __buf=0x7fff05f024f0,
__fd=<optimised out>)
at /usr/include/x86_64-linux-gnu/bits/unistd.h:44
#2 qtest_client_socket_recv_line (s=0x5644866f38b0) at
../../tests/qtest/libqtest.c:494
#3 0x0000564485947e61 in qtest_rsp_args (s=s@entry=0x5644866f38b0,
expected_args=expected_args@entry=1) at ../../tests/qtest/libqtest.c:521
#4 0x000056448594846f in qtest_query_target_endianness (s=0x5644866f38b0)
at ../../tests/qtest/libqtest.c:570
#5 0x0000564485948ed2 in qtest_init_without_qmp_handshake
(extra_args=<optimised out>)
at ../../tests/qtest/libqtest.c:332
#6 0x0000564485949616 in qtest_init (extra_args=<optimised out>) at
../../tests/qtest/libqtest.c:339
#7 0x00005644859338cd in qtest_start (
args=0x5644866f6d00 "-M pc -device
vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
-m 256M -chardev socket,id=char1,path=/tmp/qtest-28992-so"...) at
../../tests/qtest/libqtest-single.h:29
#8 restart_qemu_or_continue (
path=0x5644866f6d00 "-M pc -device
vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
-m 256M -chardev socket,id=char1,path=/tmp/qtest-28992-so"...) at
../../tests/qtest/qos-test.c:105
#9 run_one_test (arg=<optimised out>) at ../../tests/qtest/qos-test.c:178
#10 0x00007fd08990c05a in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#11 0x00007fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#12 0x00007fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#13 0x00007fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#14 0x00007fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#15 0x00007fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#16 0x00007fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#17 0x00007fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#18 0x00007fd08990c232 in g_test_run_suite () from
/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#19 0x00007fd08990c251 in g_test_run () from
/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#20 0x0000564485932359 in main (argc=<optimised out>, argv=<optimised
out>, envp=<optimised out>)
at ../../tests/qtest/qos-test.c:338
Backtrace, qemu-system-i386:Thread 4 (Thread 0x7f965ac7f700 (LWP 30079)):
#0 0x00007f9674b6938c in __GI___sigtimedwait (set=<optimised out>,
set@entry=0x7f965ac7c090, info=info@entry=0x7f965ac7bfd0,
timeout=timeout@entry=0x0)
at ../sysdeps/unix/sysv/linux/sigtimedwait.c:42
#1 0x00007f9674f2c54c in __sigwait (set=set@entry=0x7f965ac7c090,
sig=sig@entry=0x7f965ac7c08c)
at ../sysdeps/unix/sysv/linux/sigwait.c:28
#2 0x000055c2a04af6b3 in dummy_cpu_thread_fn (arg=arg@entry=0x55c2a1727aa0)
at ../../accel/dummy-cpus.c:46
#3 0x000055c2a07664e6 in qemu_thread_start (args=<optimised out>)
at ../../util/qemu-thread-posix.c:541
#4 0x00007f9674f216db in start_thread (arg=0x7f965ac7f700) at
pthread_create.c:463
#5 0x00007f9674c4a71f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 3 (Thread 0x7f965b662700 (LWP 30078)):
#0 0x00007f9674c3dcb9 in __GI___poll (fds=0x7f963c003240, nfds=3, timeout=-1)
at ../sysdeps/unix/sysv/linux/poll.c:29
#1 0x00007f967969a6e9 in () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2 0x00007f967969aa82 in g_main_loop_run () at
/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3 0x000055c2a05ce261 in iothread_run
(opaque=opaque@entry=0x55c2a144af00) at ../../iothread.c:73
#4 0x000055c2a07664e6 in qemu_thread_start (args=<optimised out>)
at ../../util/qemu-thread-posix.c:541
#5 0x00007f9674f216db in start_thread (arg=0x7f965b662700) at
pthread_create.c:463
#6 0x00007f9674c4a71f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 2 (Thread 0x7f965bf64700 (LWP 30076)):
#0 0x00007f9674f2b1fd in __lll_lock_wait () at
../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1 0x00007f9674f24025 in __GI___pthread_mutex_lock
(mutex=mutex@entry=0x55c2a12cad80 <qemu_global_mutex>) at
../nptl/pthread_mutex_lock.c:80
#2 0x000055c2a0766649 in qemu_mutex_lock_impl (mutex=0x55c2a12cad80
<qemu_global_mutex>, file=0x55c2a0cec7c2 "../../util/rcu.c", line=266)
at ../../util/qemu-thread-posix.c:79
#3 0x000055c2a0522c3f in qemu_mutex_lock_iothread_impl
(file=file@entry=0x55c2a0cec7c2 "../../util/rcu.c",
line=line@entry=266) at ../../softmmu/cpus.c:493
#4 0x000055c2a0773a5d in call_rcu_thread (opaque=opaque@entry=0x0) at
../../util/rcu.c:266
#5 0x000055c2a07664e6 in qemu_thread_start (args=<optimised out>)
at ../../util/qemu-thread-posix.c:541
#6 0x00007f9674f216db in start_thread (arg=0x7f965bf64700) at
pthread_create.c:463
#7 0x00007f9674c4a71f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 1 (Thread 0x7f967cfc3440 (LWP 30070)):
#0 0x00007f9674c3dcb9 in __GI___poll (fds=0x55c2a165ea00, nfds=2, timeout=-1)
at ../sysdeps/unix/sysv/linux/poll.c:29
#1 0x00007f967969a6e9 in () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#2 0x00007f967969aa82 in g_main_loop_run () at
/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
#3 0x000055c2a04afe5e in vhost_user_read
(dev=dev@entry=0x55c2a24a9c80, msg=msg@entry=0x7ffc2dc57ca0)
at ../../hw/virtio/vhost-user.c:402
#4 0x000055c2a04b10f9 in vhost_user_get_u64
(dev=dev@entry=0x55c2a24a9c80, request=request@entry=1,
u64=u64@entry=0x7ffc2dc57f60) at ../../hw/virtio/vhost-user.c:1334
#5 0x000055c2a04b2b9e in vhost_user_get_features
(features=0x7ffc2dc57f60, dev=0x55c2a24a9c80)
at ../../hw/virtio/vhost-user.c:1356
#6 0x000055c2a04b2b9e in vhost_user_backend_init (dev=0x55c2a24a9c80,
opaque=0x55c2a24a9f00, errp=0x7ffc2dc580e0) at
../../hw/virtio/vhost-user.c:1877
#7 0x000055c2a04a0d06 in vhost_dev_init
(hdev=hdev@entry=0x55c2a24a9c80, opaque=opaque@entry=0x55c2a24a9f00,
backend_type=backend_type@entry=VHOST_BACKEND_TYPE_USER,
busyloop_timeout=busyloop_timeout@entry=0,
errp=errp@entry=0x7ffc2dc580e0) at ../../hw/virtio/vhost.c:1302
#8 0x000055c2a04bb09e in vhost_user_blk_connect
(dev=dev@entry=0x55c2a24a9a30, errp=errp@entry=0x7ffc2dc580e0) at
../../hw/block/vhost-user-blk.c:339
#9 0x000055c2a04bb59b in vhost_user_blk_realize_connect
(errp=0x7ffc2dc580e0, s=0x55c2a24a9a30)
at ../../hw/block/vhost-user-blk.c:440
#10 0x000055c2a04bb59b in vhost_user_blk_device_realize
(dev=0x55c2a24a9a30, errp=0x7ffc2dc580e0)
at ../../hw/block/vhost-user-blk.c:513
#11 0x000055c2a05544b8 in virtio_device_realize (dev=0x55c2a24a9a30,
errp=0x7ffc2dc58130)
at ../../hw/virtio/virtio.c:3657
#12 0x000055c2a05b9b76 in device_set_realized (obj=0x55c2a24a9a30,
value=true, errp=0x7ffc2dc581b0)
at ../../hw/core/qdev.c:761
#13 0x000055c2a05a5f86 in property_set_bool (obj=0x55c2a24a9a30,
v=<optimised out>, name=<optimised out>, opaque=0x55c2a14268e0,
errp=0x7ffc2dc581b0) at ../../qom/object.c:2257
#14 0x000055c2a05a83ab in object_property_set
(obj=obj@entry=0x55c2a24a9a30, name=name@entry=0x55c2a0902a99
"realized", v=v@entry=0x55c2a24f4eb0, errp=errp@entry=0x7ffc2dc58300)
at ../../qom/object.c:1402
#15 0x000055c2a05ab36f in object_property_set_qobject
(obj=obj@entry=0x55c2a24a9a30, name=name@entry=0x55c2a0902a99
"realized", value=value@entry=0x55c2a24f4df0,
errp=errp@entry=0x7ffc2dc58300)
at ../../qom/qom-qobject.c:28
#16 0x000055c2a05a85e4 in object_property_set_bool
(obj=0x55c2a24a9a30, name=0x55c2a0902a99 "realized", value=<optimised
out>, errp=0x7ffc2dc58300) at ../../qom/object.c:1472
#17 0x000055c2a036a47f in virtio_pci_realize (pci_dev=0x55c2a24a17b0,
errp=0x7ffc2dc58300)
at ../../hw/virtio/virtio-pci.c:1907
#18 0x000055c2a038f64f in pci_qdev_realize (qdev=0x55c2a24a17b0,
errp=<optimised out>)
at ../../hw/pci/pci.c:2116
#19 0x000055c2a05b9b76 in device_set_realized (obj=0x55c2a24a17b0,
value=true, errp=0x7ffc2dc58410)
at ../../hw/core/qdev.c:761
#20 0x000055c2a05a5f86 in property_set_bool (obj=0x55c2a24a17b0,
v=<optimised out>, name=<optimised out>, opaque=0x55c2a14268e0,
errp=0x7ffc2dc58410) at ../../qom/object.c:2257
#21 0x000055c2a05a83ab in object_property_set
(obj=obj@entry=0x55c2a24a17b0, name=name@entry=0x55c2a0902a99
"realized", v=v@entry=0x55c2a24ef1c0, errp=errp@entry=0x55c2a12ded88
<error_fatal>)
at ../../qom/object.c:1402
#22 0x000055c2a05ab36f in object_property_set_qobject
(obj=obj@entry=0x55c2a24a17b0, name=name@entry=0x55c2a0902a99
"realized", value=value@entry=0x55c2a24ef180,
errp=errp@entry=0x55c2a12ded88 <error_fatal>)
at ../../qom/qom-qobject.c:28
#23 0x000055c2a05a85e4 in object_property_set_bool
(obj=0x55c2a24a17b0, name=name@entry=0x55c2a0902a99 "realized",
value=value@entry=true, errp=errp@entry=0x55c2a12ded88 <error_fatal>)
at ../../qom/object.c:1472
#24 0x000055c2a05b8a3c in qdev_realize (dev=<optimised out>,
bus=bus@entry=0x55c2a189c160, errp=errp@entry=0x55c2a12ded88
<error_fatal>) at ../../hw/core/qdev.c:389
#25 0x000055c2a0312c8e in qdev_device_add (opts=0x55c2a1422400,
errp=errp@entry=0x55c2a12ded88 <error_fatal>) at
../../softmmu/qdev-monitor.c:673
#26 0x000055c2a04e5e7f in device_init_func (opaque=<optimised out>,
opts=<optimised out>, errp=0x55c2a12ded88 <error_fatal>) at
../../softmmu/vl.c:1183
#27 0x000055c2a076fbda in qemu_opts_foreach (list=<optimised out>,
func=func@entry=0x55c2a04e5e70 <device_init_func>,
opaque=opaque@entry=0x0, errp=errp@entry=0x55c2a12ded88 <error_fatal>)
at ../../util/qemu-option.c:1137
#28 0x000055c2a04e89dd in qemu_create_cli_devices () at ../../softmmu/vl.c:2617
#29 0x000055c2a04e89dd in qmp_x_exit_preconfig (errp=<optimised out>)
at ../../softmmu/vl.c:2665
#30 0x000055c2a04ebc0d in qemu_init (argc=<optimised out>,
argv=<optimised out>, envp=<optimised out>)
at ../../softmmu/vl.c:3697
#31 0x000055c2a01c68e9 in main (argc=<optimised out>, argv=<optimised
out>, envp=<optimised out>)
at ../../softmmu/main.c:49
Backtrace, qemu-storage-daemon:
Thread 2 (Thread 0x7fb0bc104700 (LWP 30077)):
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x000055a679ea5a4b in qemu_futex_wait (val=<optimised out>,
f=<optimised out>)
at /mnt/nvmedisk/linaro/qemu-for-merges/include/qemu/futex.h:29
#2 qemu_event_wait (ev=ev@entry=0x55a67a1c87c8 <rcu_call_ready_event>)
at ../../util/qemu-thread-posix.c:480
#3 0x000055a679eb1368 in call_rcu_thread (opaque=opaque@entry=0x0) at
../../util/rcu.c:258
#4 0x000055a679ea4b66 in qemu_thread_start (args=<optimised out>)
at ../../util/qemu-thread-posix.c:541
#5 0x00007fb0cbe7d6db in start_thread (arg=0x7fb0bc104700) at
pthread_create.c:463
#6 0x00007fb0cbba671f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 1 (Thread 0x7fb0ce6070c0 (LWP 30068)):
#0 0x00007fb0cbb99db6 in __GI_ppoll (fds=0x55a67aab9ec0, nfds=5,
timeout=<optimised out>,
timeout@entry=0x0, sigmask=sigmask@entry=0x0) at
../sysdeps/unix/sysv/linux/ppoll.c:39
#1 0x000055a679eb61b9 in ppoll (__ss=0x0, __timeout=0x0,
__nfds=<optimised out>,
__fds=<optimised out>) at /usr/include/x86_64-linux-gnu/bits/poll2.h:77
#2 qemu_poll_ns (fds=<optimised out>, nfds=<optimised out>,
timeout=timeout@entry=-1)
at ../../util/qemu-timer.c:336
#3 0x000055a679ebd325 in os_host_main_loop_wait (timeout=-1) at
../../util/main-loop.c:250
#4 main_loop_wait (nonblocking=<optimised out>) at ../../util/main-loop.c:531
#5 0x000055a679da650a in main (argc=5, argv=0x7fff3724cbf8)
at ../../storage-daemon/qemu-storage-daemon.c:345
thanks
-- PMM
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host
2021-07-11 13:23 ` Richard Henderson
2021-07-11 14:21 ` Peter Maydell
@ 2021-07-11 22:19 ` Coiby Xu
1 sibling, 0 replies; 14+ messages in thread
From: Coiby Xu @ 2021-07-11 22:19 UTC (permalink / raw)
To: Richard Henderson
Cc: Kevin Wolf, Peter Maydell, QEMU Developers, Raphael Norwitz
On Sun, Jul 11, 2021 at 06:23:41AM -0700, Richard Henderson wrote:
>On 7/11/21 5:16 AM, Peter Maydell wrote:
>>On Sun, 11 Jul 2021 at 13:10, Coiby Xu <coiby.xu@gmail.com> wrote:
>>>
>>>Hi Peter,
>>>
>>>On Sat, Jul 10, 2021 at 02:30:36PM +0100, Peter Maydell wrote:
>>>>I've noticed recently that intermittently 'make check' will hang on
>>>>my aarch32 test system (really an aarch64 box with an aarch32 chroot).
>>>
>>>I have a newbie question. How do you do an aarch32 chroot on an aarch64
>>>box? At least, this issue seems to be not reproducible on an aarch64 box
>>>directly. I specifically ran the qos-test for 5 consecutive times and
>>>each time the test could finish successfully,
>>
>>Your aarch64 host CPU needs to support aarch32 at EL0 (some
>>AArch64 CPUs are pure-64 bit these days). The host kernel needs
>>to implement the 32-bit compat layer. It probably also needs to be
>>built for 4K pages (which mostly means "not RedHat"). Then you can
>>set up the 32-bit chroot however you'd normally set up a chroot
>>(for Debian you can do this with debootstrap; other distros will vary;
>>schroot is also a bit nicer than raw chroot IMHO.)
>
>If you do have a kernel built with 64k pages ("RedHat"), but you do
>have a host cpu that supports aarch32 at EL1 and EL0, then you can run
>aarch32 under KVM.
>
>The command-line I use is
>
>../run/bin/qemu-system-aarch64 -m 4096 -smp 8 -nographic \
> -M virt -cpu host,aarch64=off --accel kvm \
> -kernel vmlinuz-4.19.0-16-armmp-lpae \
> -initrd initrd.img-4.19.0-16-armmp-lpae \
> -append 'console=ttyAMA0 root=/dev/vda2' \
> -drive if=none,file=hda.q,format=qcow2,id=hd,discard=on \
> -device virtio-blk-device,drive=hd \
> -netdev tap,id=tap0,br=virbr0,helper=/usr/libexec/qemu-bridge-helper \
> -device virtio-net-device,netdev=tap0
>
>I believe that I had to perform the install under tcg because I
>couldn't find the right magic to boot off the debian cdrom with kvm.
Thanks for the instructions! Since this issue is also reproducible on
qemu-system-i386 guest x86-64 Linux host according to Peter's new email,
I'll check it on i386 guest first.
>
>
>r~
--
Best regards,
Coiby
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host
2021-07-11 15:53 ` Peter Maydell
@ 2021-07-11 22:20 ` Coiby Xu
2021-07-11 22:50 ` Coiby Xu
2021-07-16 14:31 ` Kevin Wolf
1 sibling, 1 reply; 14+ messages in thread
From: Coiby Xu @ 2021-07-11 22:20 UTC (permalink / raw)
To: Peter Maydell; +Cc: Kevin Wolf, QEMU Developers, Raphael Norwitz
On Sun, Jul 11, 2021 at 04:53:51PM +0100, Peter Maydell wrote:
>On Sat, 10 Jul 2021 at 14:30, Peter Maydell <peter.maydell@linaro.org> wrote:
>>
>> I've noticed recently that intermittently 'make check' will hang on
>> my aarch32 test system (really an aarch64 box with an aarch32 chroot).
>>
>> I think from grep that this must be the vhost-user-blk test.
>
>I've also now seen this on qemu-system-i386 guest x86-64 Linux host:
Good to to know that! This makes it much easier for me to debug this
issue.
>
>Process tree:
>petmay01 28992 0.0 0.0 123812 8612 ? Sl 14:46 0:01
> \_ tests/qtest/qos-test --tap -k -m quick
>petmay01 30068 0.0 0.0 379204 20580 ? Sl 14:46 0:00
> | \_ ./storage-daemon/qemu-storage-daemon
>--blockdev driver=file,node-name=disk0,filename=qtest.6kY6px --export
>type=vhost-user-blk,id=disk0,addr.type=unix,addr.path=/tmp/qtest-28992-sock.4Kgtk1,node-name=disk0,writable=on,num-queues=1
>petmay01 30070 0.0 0.1 1083248 63748 ? Sl 14:46 0:00
> | \_ ./qemu-system-i386 -qtest
>unix:/tmp/qtest-28992.sock -qtest-log /dev/null -chardev
>socket,path=/tmp/qtest-28992.qmp,id=char0 -mon
>chardev=char0,mode=control -display none -M pc -device
>vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
>memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
>-m 256M -chardev socket,id=char1,path=/tmp/qtest-28992-sock.4Kgtk1
>-accel qtest
>
>
>Backtrace, qos-test:
>(gdb) thread apply all bt
>
>Thread 2 (Thread 0x7fd086f1c700 (LWP 28995)):
>#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
>#1 0x000056448599484b in qemu_futex_wait (val=<optimised out>,
>f=<optimised out>)
> at /mnt/nvmedisk/linaro/qemu-for-merges/include/qemu/futex.h:29
>#2 qemu_event_wait (ev=ev@entry=0x564485c322e8 <rcu_call_ready_event>)
> at ../../util/qemu-thread-posix.c:480
>#3 0x000056448599dc18 in call_rcu_thread (opaque=opaque@entry=0x0) at
>../../util/rcu.c:258
>#4 0x0000564485993966 in qemu_thread_start (args=<optimised out>)
> at ../../util/qemu-thread-posix.c:541
>#5 0x00007fd088b446db in start_thread (arg=0x7fd086f1c700) at
>pthread_create.c:463
>#6 0x00007fd08886d71f in clone () at
>../sysdeps/unix/sysv/linux/x86_64/clone.S:95
>
>Thread 1 (Thread 0x7fd089d9a900 (LWP 28992)):
>#0 0x00007fd088b4e474 in __libc_read (fd=6,
>buf=buf@entry=0x7fff05f024f0, nbytes=nbytes@entry=1024)
> at ../sysdeps/unix/sysv/linux/read.c:27
>#1 0x0000564485947cb2 in read (__nbytes=1024, __buf=0x7fff05f024f0,
>__fd=<optimised out>)
> at /usr/include/x86_64-linux-gnu/bits/unistd.h:44
>#2 qtest_client_socket_recv_line (s=0x5644866f38b0) at
>../../tests/qtest/libqtest.c:494
>#3 0x0000564485947e61 in qtest_rsp_args (s=s@entry=0x5644866f38b0,
> expected_args=expected_args@entry=1) at ../../tests/qtest/libqtest.c:521
>#4 0x000056448594846f in qtest_query_target_endianness (s=0x5644866f38b0)
> at ../../tests/qtest/libqtest.c:570
>#5 0x0000564485948ed2 in qtest_init_without_qmp_handshake
>(extra_args=<optimised out>)
> at ../../tests/qtest/libqtest.c:332
>#6 0x0000564485949616 in qtest_init (extra_args=<optimised out>) at
>../../tests/qtest/libqtest.c:339
>#7 0x00005644859338cd in qtest_start (
> args=0x5644866f6d00 "-M pc -device
>vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
>memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
>-m 256M -chardev socket,id=char1,path=/tmp/qtest-28992-so"...) at
>../../tests/qtest/libqtest-single.h:29
>#8 restart_qemu_or_continue (
> path=0x5644866f6d00 "-M pc -device
>vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
>memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
>-m 256M -chardev socket,id=char1,path=/tmp/qtest-28992-so"...) at
>../../tests/qtest/qos-test.c:105
>#9 run_one_test (arg=<optimised out>) at ../../tests/qtest/qos-test.c:178
>#10 0x00007fd08990c05a in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
>#11 0x00007fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
>#12 0x00007fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
>#13 0x00007fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
>#14 0x00007fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
>#15 0x00007fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
>#16 0x00007fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
>#17 0x00007fd08990bf8b in ?? () from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
>#18 0x00007fd08990c232 in g_test_run_suite () from
>/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
>#19 0x00007fd08990c251 in g_test_run () from
>/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
>#20 0x0000564485932359 in main (argc=<optimised out>, argv=<optimised
>out>, envp=<optimised out>)
> at ../../tests/qtest/qos-test.c:338
>
>Backtrace, qemu-system-i386:Thread 4 (Thread 0x7f965ac7f700 (LWP 30079)):
>#0 0x00007f9674b6938c in __GI___sigtimedwait (set=<optimised out>,
> set@entry=0x7f965ac7c090, info=info@entry=0x7f965ac7bfd0,
>timeout=timeout@entry=0x0)
> at ../sysdeps/unix/sysv/linux/sigtimedwait.c:42
>#1 0x00007f9674f2c54c in __sigwait (set=set@entry=0x7f965ac7c090,
>sig=sig@entry=0x7f965ac7c08c)
> at ../sysdeps/unix/sysv/linux/sigwait.c:28
>#2 0x000055c2a04af6b3 in dummy_cpu_thread_fn (arg=arg@entry=0x55c2a1727aa0)
> at ../../accel/dummy-cpus.c:46
>#3 0x000055c2a07664e6 in qemu_thread_start (args=<optimised out>)
> at ../../util/qemu-thread-posix.c:541
>#4 0x00007f9674f216db in start_thread (arg=0x7f965ac7f700) at
>pthread_create.c:463
>#5 0x00007f9674c4a71f in clone () at
>../sysdeps/unix/sysv/linux/x86_64/clone.S:95
>
>Thread 3 (Thread 0x7f965b662700 (LWP 30078)):
>#0 0x00007f9674c3dcb9 in __GI___poll (fds=0x7f963c003240, nfds=3, timeout=-1)
> at ../sysdeps/unix/sysv/linux/poll.c:29
>#1 0x00007f967969a6e9 in () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
>#2 0x00007f967969aa82 in g_main_loop_run () at
>/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
>#3 0x000055c2a05ce261 in iothread_run
>(opaque=opaque@entry=0x55c2a144af00) at ../../iothread.c:73
>#4 0x000055c2a07664e6 in qemu_thread_start (args=<optimised out>)
> at ../../util/qemu-thread-posix.c:541
>#5 0x00007f9674f216db in start_thread (arg=0x7f965b662700) at
>pthread_create.c:463
>#6 0x00007f9674c4a71f in clone () at
>../sysdeps/unix/sysv/linux/x86_64/clone.S:95
>
>Thread 2 (Thread 0x7f965bf64700 (LWP 30076)):
>#0 0x00007f9674f2b1fd in __lll_lock_wait () at
>../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
>#1 0x00007f9674f24025 in __GI___pthread_mutex_lock
>(mutex=mutex@entry=0x55c2a12cad80 <qemu_global_mutex>) at
>../nptl/pthread_mutex_lock.c:80
>#2 0x000055c2a0766649 in qemu_mutex_lock_impl (mutex=0x55c2a12cad80
><qemu_global_mutex>, file=0x55c2a0cec7c2 "../../util/rcu.c", line=266)
>at ../../util/qemu-thread-posix.c:79
>#3 0x000055c2a0522c3f in qemu_mutex_lock_iothread_impl
>(file=file@entry=0x55c2a0cec7c2 "../../util/rcu.c",
>line=line@entry=266) at ../../softmmu/cpus.c:493
>#4 0x000055c2a0773a5d in call_rcu_thread (opaque=opaque@entry=0x0) at
>../../util/rcu.c:266
>#5 0x000055c2a07664e6 in qemu_thread_start (args=<optimised out>)
> at ../../util/qemu-thread-posix.c:541
>#6 0x00007f9674f216db in start_thread (arg=0x7f965bf64700) at
>pthread_create.c:463
>#7 0x00007f9674c4a71f in clone () at
>../sysdeps/unix/sysv/linux/x86_64/clone.S:95
>
>Thread 1 (Thread 0x7f967cfc3440 (LWP 30070)):
>#0 0x00007f9674c3dcb9 in __GI___poll (fds=0x55c2a165ea00, nfds=2, timeout=-1)
> at ../sysdeps/unix/sysv/linux/poll.c:29
>#1 0x00007f967969a6e9 in () at /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
>#2 0x00007f967969aa82 in g_main_loop_run () at
>/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0
>#3 0x000055c2a04afe5e in vhost_user_read
>(dev=dev@entry=0x55c2a24a9c80, msg=msg@entry=0x7ffc2dc57ca0)
> at ../../hw/virtio/vhost-user.c:402
>#4 0x000055c2a04b10f9 in vhost_user_get_u64
>(dev=dev@entry=0x55c2a24a9c80, request=request@entry=1,
>u64=u64@entry=0x7ffc2dc57f60) at ../../hw/virtio/vhost-user.c:1334
>#5 0x000055c2a04b2b9e in vhost_user_get_features
>(features=0x7ffc2dc57f60, dev=0x55c2a24a9c80)
> at ../../hw/virtio/vhost-user.c:1356
>#6 0x000055c2a04b2b9e in vhost_user_backend_init (dev=0x55c2a24a9c80,
>opaque=0x55c2a24a9f00, errp=0x7ffc2dc580e0) at
>../../hw/virtio/vhost-user.c:1877
>#7 0x000055c2a04a0d06 in vhost_dev_init
>(hdev=hdev@entry=0x55c2a24a9c80, opaque=opaque@entry=0x55c2a24a9f00,
>backend_type=backend_type@entry=VHOST_BACKEND_TYPE_USER,
>busyloop_timeout=busyloop_timeout@entry=0,
>errp=errp@entry=0x7ffc2dc580e0) at ../../hw/virtio/vhost.c:1302
>#8 0x000055c2a04bb09e in vhost_user_blk_connect
>(dev=dev@entry=0x55c2a24a9a30, errp=errp@entry=0x7ffc2dc580e0) at
>../../hw/block/vhost-user-blk.c:339
>#9 0x000055c2a04bb59b in vhost_user_blk_realize_connect
>(errp=0x7ffc2dc580e0, s=0x55c2a24a9a30)
> at ../../hw/block/vhost-user-blk.c:440
>#10 0x000055c2a04bb59b in vhost_user_blk_device_realize
>(dev=0x55c2a24a9a30, errp=0x7ffc2dc580e0)
> at ../../hw/block/vhost-user-blk.c:513
>#11 0x000055c2a05544b8 in virtio_device_realize (dev=0x55c2a24a9a30,
>errp=0x7ffc2dc58130)
> at ../../hw/virtio/virtio.c:3657
>#12 0x000055c2a05b9b76 in device_set_realized (obj=0x55c2a24a9a30,
>value=true, errp=0x7ffc2dc581b0)
> at ../../hw/core/qdev.c:761
>#13 0x000055c2a05a5f86 in property_set_bool (obj=0x55c2a24a9a30,
>v=<optimised out>, name=<optimised out>, opaque=0x55c2a14268e0,
>errp=0x7ffc2dc581b0) at ../../qom/object.c:2257
>#14 0x000055c2a05a83ab in object_property_set
>(obj=obj@entry=0x55c2a24a9a30, name=name@entry=0x55c2a0902a99
>"realized", v=v@entry=0x55c2a24f4eb0, errp=errp@entry=0x7ffc2dc58300)
>at ../../qom/object.c:1402
>#15 0x000055c2a05ab36f in object_property_set_qobject
>(obj=obj@entry=0x55c2a24a9a30, name=name@entry=0x55c2a0902a99
>"realized", value=value@entry=0x55c2a24f4df0,
>errp=errp@entry=0x7ffc2dc58300)
> at ../../qom/qom-qobject.c:28
>#16 0x000055c2a05a85e4 in object_property_set_bool
>(obj=0x55c2a24a9a30, name=0x55c2a0902a99 "realized", value=<optimised
>out>, errp=0x7ffc2dc58300) at ../../qom/object.c:1472
>#17 0x000055c2a036a47f in virtio_pci_realize (pci_dev=0x55c2a24a17b0,
>errp=0x7ffc2dc58300)
> at ../../hw/virtio/virtio-pci.c:1907
>#18 0x000055c2a038f64f in pci_qdev_realize (qdev=0x55c2a24a17b0,
>errp=<optimised out>)
> at ../../hw/pci/pci.c:2116
>#19 0x000055c2a05b9b76 in device_set_realized (obj=0x55c2a24a17b0,
>value=true, errp=0x7ffc2dc58410)
> at ../../hw/core/qdev.c:761
>#20 0x000055c2a05a5f86 in property_set_bool (obj=0x55c2a24a17b0,
>v=<optimised out>, name=<optimised out>, opaque=0x55c2a14268e0,
>errp=0x7ffc2dc58410) at ../../qom/object.c:2257
>#21 0x000055c2a05a83ab in object_property_set
>(obj=obj@entry=0x55c2a24a17b0, name=name@entry=0x55c2a0902a99
>"realized", v=v@entry=0x55c2a24ef1c0, errp=errp@entry=0x55c2a12ded88
><error_fatal>)
> at ../../qom/object.c:1402
>#22 0x000055c2a05ab36f in object_property_set_qobject
>(obj=obj@entry=0x55c2a24a17b0, name=name@entry=0x55c2a0902a99
>"realized", value=value@entry=0x55c2a24ef180,
>errp=errp@entry=0x55c2a12ded88 <error_fatal>)
> at ../../qom/qom-qobject.c:28
>#23 0x000055c2a05a85e4 in object_property_set_bool
>(obj=0x55c2a24a17b0, name=name@entry=0x55c2a0902a99 "realized",
>value=value@entry=true, errp=errp@entry=0x55c2a12ded88 <error_fatal>)
> at ../../qom/object.c:1472
>#24 0x000055c2a05b8a3c in qdev_realize (dev=<optimised out>,
>bus=bus@entry=0x55c2a189c160, errp=errp@entry=0x55c2a12ded88
><error_fatal>) at ../../hw/core/qdev.c:389
>#25 0x000055c2a0312c8e in qdev_device_add (opts=0x55c2a1422400,
>errp=errp@entry=0x55c2a12ded88 <error_fatal>) at
>../../softmmu/qdev-monitor.c:673
>#26 0x000055c2a04e5e7f in device_init_func (opaque=<optimised out>,
>opts=<optimised out>, errp=0x55c2a12ded88 <error_fatal>) at
>../../softmmu/vl.c:1183
>#27 0x000055c2a076fbda in qemu_opts_foreach (list=<optimised out>,
>func=func@entry=0x55c2a04e5e70 <device_init_func>,
>opaque=opaque@entry=0x0, errp=errp@entry=0x55c2a12ded88 <error_fatal>)
> at ../../util/qemu-option.c:1137
>#28 0x000055c2a04e89dd in qemu_create_cli_devices () at ../../softmmu/vl.c:2617
>#29 0x000055c2a04e89dd in qmp_x_exit_preconfig (errp=<optimised out>)
>at ../../softmmu/vl.c:2665
>#30 0x000055c2a04ebc0d in qemu_init (argc=<optimised out>,
>argv=<optimised out>, envp=<optimised out>)
> at ../../softmmu/vl.c:3697
>#31 0x000055c2a01c68e9 in main (argc=<optimised out>, argv=<optimised
>out>, envp=<optimised out>)
> at ../../softmmu/main.c:49
>
>
>Backtrace, qemu-storage-daemon:
>Thread 2 (Thread 0x7fb0bc104700 (LWP 30077)):
>#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
>#1 0x000055a679ea5a4b in qemu_futex_wait (val=<optimised out>,
>f=<optimised out>)
> at /mnt/nvmedisk/linaro/qemu-for-merges/include/qemu/futex.h:29
>#2 qemu_event_wait (ev=ev@entry=0x55a67a1c87c8 <rcu_call_ready_event>)
> at ../../util/qemu-thread-posix.c:480
>#3 0x000055a679eb1368 in call_rcu_thread (opaque=opaque@entry=0x0) at
>../../util/rcu.c:258
>#4 0x000055a679ea4b66 in qemu_thread_start (args=<optimised out>)
> at ../../util/qemu-thread-posix.c:541
>#5 0x00007fb0cbe7d6db in start_thread (arg=0x7fb0bc104700) at
>pthread_create.c:463
>#6 0x00007fb0cbba671f in clone () at
>../sysdeps/unix/sysv/linux/x86_64/clone.S:95
>
>Thread 1 (Thread 0x7fb0ce6070c0 (LWP 30068)):
>#0 0x00007fb0cbb99db6 in __GI_ppoll (fds=0x55a67aab9ec0, nfds=5,
>timeout=<optimised out>,
> timeout@entry=0x0, sigmask=sigmask@entry=0x0) at
>../sysdeps/unix/sysv/linux/ppoll.c:39
>#1 0x000055a679eb61b9 in ppoll (__ss=0x0, __timeout=0x0,
>__nfds=<optimised out>,
> __fds=<optimised out>) at /usr/include/x86_64-linux-gnu/bits/poll2.h:77
>#2 qemu_poll_ns (fds=<optimised out>, nfds=<optimised out>,
>timeout=timeout@entry=-1)
> at ../../util/qemu-timer.c:336
>#3 0x000055a679ebd325 in os_host_main_loop_wait (timeout=-1) at
>../../util/main-loop.c:250
>#4 main_loop_wait (nonblocking=<optimised out>) at ../../util/main-loop.c:531
>#5 0x000055a679da650a in main (argc=5, argv=0x7fff3724cbf8)
> at ../../storage-daemon/qemu-storage-daemon.c:345
>
>thanks
>-- PMM
--
Best regards,
Coiby
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host
2021-07-11 22:20 ` Coiby Xu
@ 2021-07-11 22:50 ` Coiby Xu
2021-07-12 9:39 ` Peter Maydell
0 siblings, 1 reply; 14+ messages in thread
From: Coiby Xu @ 2021-07-11 22:50 UTC (permalink / raw)
To: Peter Maydell; +Cc: Kevin Wolf, QEMU Developers, Raphael Norwitz
On Mon, Jul 12, 2021 at 06:20:33AM +0800, Coiby Xu wrote:
>On Sun, Jul 11, 2021 at 04:53:51PM +0100, Peter Maydell wrote:
>>On Sat, 10 Jul 2021 at 14:30, Peter Maydell <peter.maydell@linaro.org> wrote:
>>>
>>>I've noticed recently that intermittently 'make check' will hang on
>>>my aarch32 test system (really an aarch64 box with an aarch32 chroot).
>>>
>>>I think from grep that this must be the vhost-user-blk test.
>>
>>I've also now seen this on qemu-system-i386 guest x86-64 Linux host:
>
>Good to to know that! This makes it much easier for me to debug this
>issue.
Which i386 image do you use for the guest? Could you share the download
link? I can't find a suitable i386 qcow2 image. For example, [1] is
outdated.
[1] http://people.debian.org/~aurel32/qemu
--
Best regards,
Coiby
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host
2021-07-11 22:50 ` Coiby Xu
@ 2021-07-12 9:39 ` Peter Maydell
2021-07-16 12:30 ` Coiby Xu
0 siblings, 1 reply; 14+ messages in thread
From: Peter Maydell @ 2021-07-12 9:39 UTC (permalink / raw)
To: Coiby Xu; +Cc: Kevin Wolf, QEMU Developers, Raphael Norwitz
On Sun, 11 Jul 2021 at 23:55, Coiby Xu <coiby.xu@gmail.com> wrote:
>
> On Mon, Jul 12, 2021 at 06:20:33AM +0800, Coiby Xu wrote:
> >On Sun, Jul 11, 2021 at 04:53:51PM +0100, Peter Maydell wrote:
> >>On Sat, 10 Jul 2021 at 14:30, Peter Maydell <peter.maydell@linaro.org> wrote:
> >>>
> >>>I've noticed recently that intermittently 'make check' will hang on
> >>>my aarch32 test system (really an aarch64 box with an aarch32 chroot).
> >>>
> >>>I think from grep that this must be the vhost-user-blk test.
> >>
> >>I've also now seen this on qemu-system-i386 guest x86-64 Linux host:
> >
> >Good to to know that! This makes it much easier for me to debug this
> >issue.
>
> Which i386 image do you use for the guest? Could you share the download
> link? I can't find a suitable i386 qcow2 image. For example, [1] is
> outdated.
I'm just running "make check" on the x86-64 host, which runs tests
on qemu-system-i386 (assuming you built i386 targets).
-- PMM
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host
2021-07-12 9:39 ` Peter Maydell
@ 2021-07-16 12:30 ` Coiby Xu
0 siblings, 0 replies; 14+ messages in thread
From: Coiby Xu @ 2021-07-16 12:30 UTC (permalink / raw)
To: Peter Maydell; +Cc: Kevin Wolf, QEMU Developers, Raphael Norwitz
On Mon, Jul 12, 2021 at 10:39:50AM +0100, Peter Maydell wrote:
>On Sun, 11 Jul 2021 at 23:55, Coiby Xu <coiby.xu@gmail.com> wrote:
>>
>> On Mon, Jul 12, 2021 at 06:20:33AM +0800, Coiby Xu wrote:
>> >On Sun, Jul 11, 2021 at 04:53:51PM +0100, Peter Maydell wrote:
>> >>On Sat, 10 Jul 2021 at 14:30, Peter Maydell <peter.maydell@linaro.org> wrote:
>> >>>
>> >>>I've noticed recently that intermittently 'make check' will hang on
>> >>>my aarch32 test system (really an aarch64 box with an aarch32 chroot).
>> >>>
>> >>>I think from grep that this must be the vhost-user-blk test.
>> >>
>> >>I've also now seen this on qemu-system-i386 guest x86-64 Linux host:
>> >
>> >Good to to know that! This makes it much easier for me to debug this
>> >issue.
>>
>> Which i386 image do you use for the guest? Could you share the download
>> link? I can't find a suitable i386 qcow2 image. For example, [1] is
>> outdated.
>
>I'm just running "make check" on the x86-64 host, which runs tests
>on qemu-system-i386 (assuming you built i386 targets).
How often can this issue happen? Unfortunately, I can't reproduce it in
the past four days on my own laptop and two Openstack machines every now
and then,
git clone git://git.qemu.org/qemu.g
mkdir build && cd build
../configure --target-list=i386-softmmu
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} \
QTEST_QEMU_BINARY=build/i386-softmmu/qemu-system-i386 QTEST_QEMU_IMG=build/qemu-img \
QTEST_QEMU_STORAGE_DAEMON_BINARY=build/storage-daemon/qemu-storage-daemon \
build/tests/qtest/qos-test
I've also tried running the whole tests using "make check" or "make
check-qtest" but the results were the same.
>
>-- PMM
--
Best regards,
Coiby
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host
2021-07-11 15:53 ` Peter Maydell
2021-07-11 22:20 ` Coiby Xu
@ 2021-07-16 14:31 ` Kevin Wolf
1 sibling, 0 replies; 14+ messages in thread
From: Kevin Wolf @ 2021-07-16 14:31 UTC (permalink / raw)
To: Peter Maydell; +Cc: QEMU Developers, Coiby Xu, Raphael Norwitz
Am 11.07.2021 um 17:53 hat Peter Maydell geschrieben:
> On Sat, 10 Jul 2021 at 14:30, Peter Maydell <peter.maydell@linaro.org> wrote:
> >
> > I've noticed recently that intermittently 'make check' will hang on
> > my aarch32 test system (really an aarch64 box with an aarch32 chroot).
> >
> > I think from grep that this must be the vhost-user-blk test.
>
> I've also now seen this on qemu-system-i386 guest x86-64 Linux host:
Your two stack traces look very different to me, the common thing is
just that one process requests something and the other seems to have
ignored it and is just idle.
In the first stack trace, it was the qtest sending the very first
command ('endianness') over the qtest socket and QEMU seemed to ignore
it. In the second stack trace, it is the vhost-user-blk realize() code
in QEMU sending a request to the export in qemu-storage-daemon and never
getting an answer.
If this is the same bug, it looks to me as if it's something with the
event notification in the main loop? Can we check if there would
actually be an event pending in the apparently idle process if ppoll()
returned?
Kevin
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host
2021-07-10 13:30 intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host Peter Maydell
2021-07-11 12:08 ` Coiby Xu
2021-07-11 15:53 ` Peter Maydell
@ 2021-07-22 8:45 ` Claudio Fontana
2 siblings, 0 replies; 14+ messages in thread
From: Claudio Fontana @ 2021-07-22 8:45 UTC (permalink / raw)
To: Peter Maydell, QEMU Developers
Cc: Kevin Wolf, Philippe Mathieu-Daudé, Coiby Xu, Raphael Norwitz
On 7/10/21 3:30 PM, Peter Maydell wrote:
> I've noticed recently that intermittently 'make check' will hang on
> my aarch32 test system (really an aarch64 box with an aarch32 chroot).
>
> I think from grep that this must be the vhost-user-blk test.
>
> Here's the process tree:
>
> pmaydell 13126 0.0 0.0 8988 6416 ? S Jul09 0:01 make
> -C build/all-a32 check V=1 GCC_COLORS= -j9
> pmaydell 19632 0.0 0.0 4432 2096 ? S Jul09 0:00 \_
> bash -o pipefail -c echo 'MALLOC_PERTURB_=${MALLOC_PERTURB_:-$((
> ${RANDOM:-0} % 255 + 1))} QTEST_QEMU_IMG=./qemu-img
> G_TEST_DBUS_DAEMON=/home/peter.maydell/qemu/tests/dbus-vmstate-daemon.sh
> QTEST_QEMU_BINARY=./qemu-system-i386
> QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon
> tests/qtest/qos-test --tap -k' &&
> MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}
> QTEST_QEMU_IMG=./qemu-img
> G_TEST_DBUS_DAEMON=/home/peter.maydell/qemu/tests/dbus-vmstate-daemon.sh
> QTEST_QEMU_BINARY=./qemu-system-i386
> QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon
> tests/qtest/qos-test --tap -k -m quick < /dev/null |
> ./scripts/tap-driver.pl --test-name="qtest-i386/qos-test"
> pmaydell 19634 0.0 0.0 13608 3076 ? Sl Jul09 0:02
> \_ tests/qtest/qos-test --tap -k -m quick
> pmaydell 20679 0.0 0.0 109076 16100 ? Sl Jul09 0:00
> | \_ ./storage-daemon/qemu-storage-daemon --blockdev
> driver=file,node-name=disk0,filename=qtest.X7RL2X --export
> type=vhost-user-blk,id=disk0,addr.type=unix,addr.path=/tmp/qtest-19634-sock.9LJoHn,node-name=disk0,writable=on,num-queues=1
> pmaydell 20681 0.0 0.2 447828 46544 ? Sl Jul09 0:00
> | \_ ./qemu-system-i386 -qtest unix:/tmp/qtest-19634.sock -qtest-log
> /dev/null -chardev socket,path=/tmp/qtest-19634.qmp,id=char0 -mon
> chardev=char0,mode=control -display none -M pc -device
> vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
> memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
> -m 256M -chardev socket,id=char1,path=/tmp/qtest-19634-sock.9LJoHn
> -accel qtest
> pmaydell 19635 0.0 0.0 10256 7176 ? S Jul09 0:00
> \_ perl ./scripts/tap-driver.pl --test-name=qtest-i386/qos-test
>
>
> Backtrace from tests/qtest/qos-test (not as helpful as it could
> be since this is an optimized build):
>
> (gdb) thread apply all bt
>
> Thread 2 (Thread 0xf76ff240 (LWP 19636)):
> #0 syscall () at ../sysdeps/unix/sysv/linux/arm/syscall.S:37
> #1 0x005206de in qemu_futex_wait (val=<optimised out>, f=<optimised
> out>) at /home/peter.maydell/qemu/include/qemu/futex.h:29
> #2 qemu_event_wait (ev=ev@entry=0x5816fc <rcu_call_ready_event>) at
> ../../util/qemu-thread-posix.c:480
> #3 0x005469c2 in call_rcu_thread (opaque=<optimised out>) at
> ../../util/rcu.c:258
> #4 0x0051fbc2 in qemu_thread_start (args=<optimised out>) at
> ../../util/qemu-thread-posix.c:541
> #5 0xf785a614 in start_thread (arg=0xf6ce711c) at pthread_create.c:463
> #6 0xf77f57ec in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
> from /lib/arm-linux-gnueabihf/libc.so.6
> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
>
> Thread 1 (Thread 0xf7a04010 (LWP 19634)):
> #0 __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
> #1 0xf7861d8c in __libc_read (fd=12, buf=buf@entry=0xff9ce8e4,
> nbytes=nbytes@entry=1024) at ../sysdeps/unix/sysv/linux/read.c:27
> #2 0x004ebc5a in read (__nbytes=1024, __buf=0xff9ce8e4,
> __fd=<optimised out>) at
> /usr/include/arm-linux-gnueabihf/bits/unistd.h:44
> #3 qtest_client_socket_recv_line (s=0x1a46cb8) at
> ../../tests/qtest/libqtest.c:494
> #4 0x004ebd4e in qtest_rsp_args (s=s@entry=0x1a46cb8,
> expected_args=expected_args@entry=1) at
> ../../tests/qtest/libqtest.c:521
> #5 0x004ec1ee in qtest_query_target_endianness (s=0x1a46cb8) at
> ../../tests/qtest/libqtest.c:570
> #6 0x004ec94a in qtest_init_without_qmp_handshake
> (extra_args=<optimised out>) at ../../tests/qtest/libqtest.c:332
> #7 0x004ecd7a in qtest_init (extra_args=<optimised out>) at
> ../../tests/qtest/libqtest.c:339
> #8 0x004ded10 in qtest_start (
> args=0x1a63710 "-M pc -device
> vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
> memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
> -m 256M -chardev socket,id=char1,path=/tmp/qtest-19634-so"...) at
> ../../tests/qtest/libqtest-single.h:29
> #9 restart_qemu_or_continue (
> path=0x1a63710 "-M pc -device
> vhost-user-blk-pci,id=drv0,chardev=char1,addr=4.0 -object
> memory-backend-memfd,id=mem,size=256M,share=on -M memory-backend=mem
> -m 256M -chardev socket,id=char1,path=/tmp/qtest-19634-so"...) at
> ../../tests/qtest/qos-test.c:105
> #10 run_one_test (arg=<optimised out>) at ../../tests/qtest/qos-test.c:178
> #11 0xf794ee74 in ?? () from /usr/lib/arm-linux-gnueabihf/libglib-2.0.so.0
> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
>
>
> Backtrace from qemu-system-i386:
>
> (gdb) thread apply all bt
>
> Thread 4 (Thread 0xdfd0cb90 (LWP 20734)):
> #0 0xf6f85206 in __libc_do_syscall () at
> ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47
> #1 0xf6f93492 in __GI___sigtimedwait (set=<optimised out>,
> set@entry=0xdfd0c3c4, info=info@entry=0xdfd0c324,
> timeout=timeout@entry=0x0) at
> ../sysdeps/unix/sysv/linux/sigtimedwait.c:42
> #2 0xf7073e6c in __sigwait (set=set@entry=0xdfd0c3c4,
> sig=sig@entry=0xdfd0c3c0) at ../sysdeps/unix/sysv/linux/sigwait.c:28
> #3 0x00c999a6 in dummy_cpu_thread_fn (arg=0x24d1168) at
> ../../accel/dummy-cpus.c:46
> #4 0x00e18ba2 in qemu_thread_start (args=<optimised out>) at
> ../../util/qemu-thread-posix.c:541
> #5 0xf706b614 in start_thread (arg=0xa8db8843) at pthread_create.c:463
> #6 0xf70067ec in () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
>
> Thread 3 (Thread 0xe07fcb90 (LWP 20710)):
> #0 0xf6f85204 in __libc_do_syscall () at
> ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
> #1 0xf6fff948 in __GI___poll (fds=0x2409b90, nfds=3, timeout=-1) at
> ../sysdeps/unix/sysv/linux/poll.c:29
> #2 0xf781692e in () at /usr/lib/arm-linux-gnueabihf/libglib-2.0.so.0
>
> Thread 2 (Thread 0xf1201b90 (LWP 20703)):
> #0 0xf7074f04 in __libc_do_syscall () at
> ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
> #1 0xf7072ad2 in __lll_lock_wait (futex=futex@entry=0x14f2e68
> <qemu_global_mutex>, private=0) at lowlevellock.c:43
> #2 0xf706d3e2 in __GI___pthread_mutex_lock
> (mutex=mutex@entry=0x14f2e68 <qemu_global_mutex>) at
> pthread_mutex_lock.c:78
> #3 0x00e18c9c in qemu_mutex_lock_impl (mutex=0x14f2e68
> <qemu_global_mutex>, file=0x1355bd4 "../../util/rcu.c", line=266)
> at ../../util/qemu-thread-posix.c:79
> #4 0x00c1403a in qemu_mutex_lock_iothread_impl (file=0x1355bd4
> "../../util/rcu.c", line=line@entry=266) at ../../softmmu/cpus.c:493
> #5 0x00e15ad4 in call_rcu_thread (opaque=<optimised out>) at
> ../../util/rcu.c:266
> #6 0x00e18ba2 in qemu_thread_start (args=<optimised out>) at
> ../../util/qemu-thread-posix.c:541
> #7 0xf706b614 in start_thread (arg=0xa8db8843) at pthread_create.c:463
> #8 0xf70067ec in () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
>
> Thread 1 (Thread 0xf12f9010 (LWP 20681)):
> #0 0xf6f85204 in __libc_do_syscall () at
> ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
> #1 0xf6fff948 in __GI___poll (fds=0x3248040, nfds=2, timeout=-1) at
> ../sysdeps/unix/sysv/linux/poll.c:29
> #2 0xf781692e in () at /usr/lib/arm-linux-gnueabihf/libglib-2.0.so.0
>
> Backtrace of qemu-storage-daemon:
>
> (gdb) thread apply all bt
>
> Thread 2 (Thread 0xf21ffdd0 (LWP 20722)):
> #0 syscall () at ../sysdeps/unix/sysv/linux/arm/syscall.S:37
> #1 0x009a9986 in qemu_futex_wait (val=<optimised out>, f=<optimised
> out>) at /home/peter.maydell/qemu/include/qemu/futex.h:29
> #2 qemu_event_wait (ev=ev@entry=0xa64958 <rcu_call_ready_event>) at
> ../../util/qemu-thread-posix.c:480
> #3 0x009a5d66 in call_rcu_thread (opaque=<optimised out>) at
> ../../util/rcu.c:258
> #4 0x009a8e6a in qemu_thread_start (args=<optimised out>) at
> ../../util/qemu-thread-posix.c:541
> #5 0xf79dd614 in start_thread (arg=0xa310c11f) at pthread_create.c:463
> #6 0xf79787ec in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73
> from /lib/arm-linux-gnueabihf/libc.so.6
> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
>
> Thread 1 (Thread 0xf22f7010 (LWP 20679)):
> #0 __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:46
> #1 0xf7971a04 in __GI_ppoll (fds=0x117c578, nfds=5,
> timeout=<optimised out>, timeout@entry=0x0, sigmask=sigmask@entry=0x0)
> at ../sysdeps/unix/sysv/linux/ppoll.c:39
> #2 0x009a68fc in ppoll (__ss=0x0, __timeout=0x0, __nfds=<optimised
> out>, __fds=<optimised out>)
> at /usr/include/arm-linux-gnueabihf/bits/poll2.h:77
> #3 qemu_poll_ns (fds=<optimised out>, nfds=<optimised out>,
> timeout=<optimised out>) at ../../util/qemu-timer.c:336
> #4 0x009c0222 in os_host_main_loop_wait (timeout=-1) at
> ../../util/main-loop.c:250
> #5 main_loop_wait (nonblocking=<optimised out>) at ../../util/main-loop.c:531
> #6 0x008ee92a in main (argc=5, argv=0xffdce5c4) at
> ../../storage-daemon/qemu-storage-daemon.c:345
>
>
> Any ideas ?
>
> thanks
> -- PMM
>
I see something very similar on X86:
https://lists.gnu.org/archive/html/qemu-devel/2021-07/msg05814.html
It is easy to reproduce in my case (seems to happen all the time to me).
Ciao,
Claudio
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2021-07-22 8:46 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-07-10 13:30 intermittent hang in qos-test for qemu-system-i386 on 32-bit arm host Peter Maydell
2021-07-11 12:08 ` Coiby Xu
2021-07-11 12:16 ` Peter Maydell
2021-07-11 13:23 ` Richard Henderson
2021-07-11 14:21 ` Peter Maydell
2021-07-11 14:27 ` Richard Henderson
2021-07-11 22:19 ` Coiby Xu
2021-07-11 15:53 ` Peter Maydell
2021-07-11 22:20 ` Coiby Xu
2021-07-11 22:50 ` Coiby Xu
2021-07-12 9:39 ` Peter Maydell
2021-07-16 12:30 ` Coiby Xu
2021-07-16 14:31 ` Kevin Wolf
2021-07-22 8:45 ` Claudio Fontana
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.