Re: [PULL v2 31/82] vhost: Change the sequence of device start

[Bernhard Beschow <shentey@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 6942 bytes --]

On Wed, Nov 2, 2022 at 5:24 PM Michael S. Tsirkin <mst@redhat.com> wrote:

> From: Yajun Wu <yajunw@nvidia.com>
>
> This patch is part of adding vhost-user vhost_dev_start support. The
> motivation is to improve backend configuration speed and reduce live
> migration VM downtime.
>
> Moving the device start routines after finishing all the necessary device
> and VQ configuration, further aligning to the virtio specification for
> "device initialization sequence".
>
> Following patch will add vhost-user vhost_dev_start support.
>
> Signed-off-by: Yajun Wu <yajunw@nvidia.com>
> Acked-by: Parav Pandit <parav@nvidia.com>
>
> Message-Id: <20221017064452.1226514-2-yajunw@nvidia.com>
> Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> ---
>  hw/block/vhost-user-blk.c | 18 +++++++++++-------
>  hw/net/vhost_net.c        | 12 ++++++------
>  2 files changed, 17 insertions(+), 13 deletions(-)
>

A git bisect tells me that this is the first bad commit for failing
qos-tests which only fail when parallel jobs are enabled, e.g. `make
check-qtest -j8`:

Summary of Failures:

 76/541 qemu:qtest+qtest-aarch64 / qtest-aarch64/qos-test
      ERROR          18.68s   killed by signal 6 SIGABRT
 77/541 qemu:qtest+qtest-arm / qtest-arm/qos-test
      ERROR          17.60s   killed by signal 6 SIGABRT
 93/541 qemu:qtest+qtest-i386 / qtest-i386/qos-test
      ERROR          18.98s   killed by signal 6 SIGABRT
108/541 qemu:qtest+qtest-ppc64 / qtest-ppc64/qos-test
      ERROR          16.40s   killed by signal 6 SIGABRT
112/541 qemu:qtest+qtest-i386 / qtest-i386/bios-tables-test
      ERROR          145.94s   killed by signal 6 SIGABRT
130/541 qemu:qtest+qtest-x86_64 / qtest-x86_64/qos-test
      ERROR          17.32s   killed by signal 6 SIGABRT
243/541 qemu:qtest+qtest-x86_64 / qtest-x86_64/bios-tables-test
      ERROR          127.70s   killed by signal 6 SIGABRT

Ok:                 500
Expected Fail:      0
Fail:               7
Unexpected Pass:    0
Skipped:            34
Timeout:            0

Can anyone else reproduce this?

Here is a log of one failed case:

# child process
(/aarch64/virt/generic-pcihost/pci-bus-generic/pci-bus/virtio-net-pci/virtio-net/virtio-net-tests/vhost-user/flags-mismatch/subprocess
[384604]) killed by signal 6 (Aborted), core dumped
# child process
(/aarch64/virt/generic-pcihost/pci-bus-generic/pci-bus/virtio-net-pci/virtio-net/virtio-net-tests/vhost-user/flags-mismatch/subprocess
[384604]) stdout: ""
# child process
(/aarch64/virt/generic-pcihost/pci-bus-generic/pci-bus/virtio-net-pci/virtio-net/virtio-net-tests/vhost-user/flags-mismatch/subprocess
[384604]) stderr: "qemu-system-aarch64: -chardev
socket,id=chr-flags-mismatch,path=/tmp/vhost-test-BBEAV1/flags-mismatch.sock,server=on:
info: QEMU waiting for connection on:
disconnected:unix:/tmp/vhost-test-BBEAV1/flags-mismatch.sock,server=on\nqemu-system-aarch64:
Failed to write msg. Wrote -1 instead of 52.\nqemu-system-aarch64:
vhost_set_mem_table failed: Invalid argument (22)\nqemu-system-aarch64:
Failed to set msg fds.\nqemu-system-aarch64: vhost VQ 0 ring restore
failed: -22: Invalid argument (22)\nBroken
pipe\n../src/tests/qtest/libqtest.c:188: kill_qemu() detected QEMU death
from signal 11 (Segmentation fault) (core dumped)\n"
Bail out! ERROR:../src/tests/qtest/qos-test.c:191:subprocess_run_one_test:
child process
(/aarch64/virt/generic-pcihost/pci-bus-generic/pci-bus/virtio-net-pci/virtio-net/virtio-net-tests/vhost-user/flags-mismatch/subprocess
[384604]) failed unexpectedly
----------------------------------- stderr
-----------------------------------
qemu-system-aarch64: Failed to write msg. Wrote -1 instead of 20.
qemu-system-aarch64: vhost VQ 0 ring restore failed: -22: Invalid argument
(22)
qemu-system-aarch64: Failed to set msg fds.
qemu-system-aarch64: vhost VQ 1 ring restore failed: -22: Invalid argument
(22)
qemu-system-aarch64: -chardev
socket,id=chr-reconnect,path=/tmp/vhost-test-FTJ0U1/reconnect.sock,server=on:
info: QEMU waiting for connection on:
disconnected:unix:/tmp/vhost-test-FTJ0U1/reconnect.sock,server=on
qemu-system-aarch64: Failed to write msg. Wrote -1 instead of 20.
qemu-system-aarch64: vhost VQ 0 ring restore failed: -22: Invalid argument
(22)
qemu-system-aarch64: Failed to set msg fds.
qemu-system-aarch64: vhost VQ 1 ring restore failed: -22: Invalid argument
(22)
qemu-system-aarch64: -chardev
socket,id=chr-connect-fail,path=/tmp/vhost-test-4W4DV1/connect-fail.sock,server=on:
info: QEMU waiting for connection on:
disconnected:unix:/tmp/vhost-test-4W4DV1/connect-fail.sock,server=on
qemu-system-aarch64: -netdev
vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: Failed to read
msg header. Read 0 instead of 12. Original request 1.
qemu-system-aarch64: -netdev
vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on:
vhost_backend_init failed: Protocol error
qemu-system-aarch64: -netdev
vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: failed to init
vhost_net for queue 0
qemu-system-aarch64: -netdev
vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: info: QEMU
waiting for connection on:
disconnected:unix:/tmp/vhost-test-4W4DV1/connect-fail.sock,server=on
qemu-system-aarch64: Failed to set msg fds.
qemu-system-aarch64: vhost VQ 0 ring restore failed: -22: Invalid argument
(22)
qemu-system-aarch64: Failed to set msg fds.
qemu-system-aarch64: vhost VQ 1 ring restore failed: -22: Invalid argument
(22)
qemu-system-aarch64: -chardev
socket,id=chr-flags-mismatch,path=/tmp/vhost-test-BBEAV1/flags-mismatch.sock,server=on:
info: QEMU waiting for connection on:
disconnected:unix:/tmp/vhost-test-BBEAV1/flags-mismatch.sock,server=on
qemu-system-aarch64: Failed to write msg. Wrote -1 instead of 52.
qemu-system-aarch64: vhost_set_mem_table failed: Invalid argument (22)
qemu-system-aarch64: Failed to set msg fds.
qemu-system-aarch64: vhost VQ 0 ring restore failed: -22: Invalid argument
(22)
Broken pipe
../src/tests/qtest/libqtest.c:188: kill_qemu() detected QEMU death from
signal 11 (Segmentation fault) (core dumped)
**
ERROR:../src/tests/qtest/qos-test.c:191:subprocess_run_one_test: child
process
(/aarch64/virt/generic-pcihost/pci-bus-generic/pci-bus/virtio-net-pci/virtio-net/virtio-net-tests/vhost-user/flags-mismatch/subprocess
[384604]) failed unexpectedly

(test program exited with status code -6)
==============================================================================

=================================== 77/541
===================================
test:         qemu:qtest+qtest-arm / qtest-arm/qos-test
start time:   16:04:10
duration:     17.60s
result:       killed by signal 6 SIGABRT
command:      MALLOC_PERTURB_=98 QTEST_QEMU_IMG=./qemu-img
QTEST_QEMU_STORAGE_DAEMON_BINARY=./storage-daemon/qemu-storage-daemon
QTEST_QEMU_BINARY=./qemu-system-arm
G_TEST_DBUS_DAEMON=../src/tests/dbus-vmstate-daemon.sh
./tests/qtest/qos-test --tap -k

Best regards,
Bernhard

[-- Attachment #2: Type: text/html, Size: 8404 bytes --]