[RFC PATCH] virtio: re-order vm_running and use_started checks

[RFC PATCH] virtio: re-order vm_running and use_started checks

[Alex Bennée]

During migration the virtio device state can be restored before we
restart the VM. As no devices can be running while the VM is paused it
makes sense to bail out early in that case.

This returns the order introduced in:

 9f6bcfd99f (hw/virtio: move vm_running check to virtio_device_started)

to what virtio-sock was doing longhand.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
---
 include/hw/virtio/virtio.h | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index f41b4a7e64..ebb58feaac 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -385,14 +385,14 @@ static inline bool virtio_is_big_endian(VirtIODevice *vdev)
 
 static inline bool virtio_device_started(VirtIODevice *vdev, uint8_t status)
 {
-    if (vdev->use_started) {
-        return vdev->started;
-    }
-
     if (!vdev->vm_running) {
         return false;
     }
 
+    if (vdev->use_started) {
+        return vdev->started;
+    }
+
     return status & VIRTIO_CONFIG_S_DRIVER_OK;
 }
 
-- 
2.34.1

Re: [RFC PATCH] virtio: re-order vm_running and use_started checks

[Christian Borntraeger]

Am 14.10.22 um 15:21 schrieb Alex Bennée:
> During migration the virtio device state can be restored before we
> restart the VM. As no devices can be running while the VM is paused it
> makes sense to bail out early in that case.
> 
> This returns the order introduced in:
> 
>   9f6bcfd99f (hw/virtio: move vm_running check to virtio_device_started)
> 
> to what virtio-sock was doing longhand.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
Tested-by: Christian Borntraeger <borntraeger@linux.ibm.com>

> ---
>   include/hw/virtio/virtio.h | 8 ++++----
>   1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
> index f41b4a7e64..ebb58feaac 100644
> --- a/include/hw/virtio/virtio.h
> +++ b/include/hw/virtio/virtio.h
> @@ -385,14 +385,14 @@ static inline bool virtio_is_big_endian(VirtIODevice *vdev)
>   
>   static inline bool virtio_device_started(VirtIODevice *vdev, uint8_t status)
>   {
> -    if (vdev->use_started) {
> -        return vdev->started;
> -    }
> -
>       if (!vdev->vm_running) {
>           return false;
>       }
>   
> +    if (vdev->use_started) {
> +        return vdev->started;
> +    }
> +
>       return status & VIRTIO_CONFIG_S_DRIVER_OK;
>   }
>   

Re: [RFC PATCH] virtio: re-order vm_running and use_started checks

[Philippe Mathieu-Daudé]

On 14/10/22 15:21, Alex Bennée wrote:
> During migration the virtio device state can be restored before we
> restart the VM. As no devices can be running while the VM is paused it
> makes sense to bail out early in that case.
> 
> This returns the order introduced in:
> 
>   9f6bcfd99f (hw/virtio: move vm_running check to virtio_device_started)
> 
> to what virtio-sock was doing longhand.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
> ---
>   include/hw/virtio/virtio.h | 8 ++++----
>   1 file changed, 4 insertions(+), 4 deletions(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

Re: [RFC PATCH] virtio: re-order vm_running and use_started checks

[Michael S. Tsirkin]

On Fri, Oct 14, 2022 at 02:21:08PM +0100, Alex Bennée wrote:
> During migration the virtio device state can be restored before we
> restart the VM. As no devices can be running while the VM is paused it
> makes sense to bail out early in that case.
> 
> This returns the order introduced in:
> 
>  9f6bcfd99f (hw/virtio: move vm_running check to virtio_device_started)
> 
> to what virtio-sock was doing longhand.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Cc: Christian Borntraeger <borntraeger@linux.ibm.com>

Queued but Alex, generally pls post non-RFC if you expect patch
to me merged.

> ---
>  include/hw/virtio/virtio.h | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
> index f41b4a7e64..ebb58feaac 100644
> --- a/include/hw/virtio/virtio.h
> +++ b/include/hw/virtio/virtio.h
> @@ -385,14 +385,14 @@ static inline bool virtio_is_big_endian(VirtIODevice *vdev)
>  
>  static inline bool virtio_device_started(VirtIODevice *vdev, uint8_t status)
>  {
> -    if (vdev->use_started) {
> -        return vdev->started;
> -    }
> -
>      if (!vdev->vm_running) {
>          return false;
>      }
>  
> +    if (vdev->use_started) {
> +        return vdev->started;
> +    }
> +
>      return status & VIRTIO_CONFIG_S_DRIVER_OK;
>  }
>  
> -- 
> 2.34.1

Re: [RFC PATCH] virtio: re-order vm_running and use_started checks

[Michael S. Tsirkin]

On Fri, Oct 14, 2022 at 02:21:08PM +0100, Alex Bennée wrote:
> During migration the virtio device state can be restored before we
> restart the VM. As no devices can be running while the VM is paused it
> makes sense to bail out early in that case.
> 
> This returns the order introduced in:
> 
>  9f6bcfd99f (hw/virtio: move vm_running check to virtio_device_started)
> 
> to what virtio-sock was doing longhand.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Cc: Christian Borntraeger <borntraeger@linux.ibm.com>

Still checking but this seems to cause make check to fail on Fedora 36 under ubsan.

> ---
>  include/hw/virtio/virtio.h | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
> index f41b4a7e64..ebb58feaac 100644
> --- a/include/hw/virtio/virtio.h
> +++ b/include/hw/virtio/virtio.h
> @@ -385,14 +385,14 @@ static inline bool virtio_is_big_endian(VirtIODevice *vdev)
>  
>  static inline bool virtio_device_started(VirtIODevice *vdev, uint8_t status)
>  {
> -    if (vdev->use_started) {
> -        return vdev->started;
> -    }
> -
>      if (!vdev->vm_running) {
>          return false;
>      }
>  
> +    if (vdev->use_started) {
> +        return vdev->started;
> +    }
> +
>      return status & VIRTIO_CONFIG_S_DRIVER_OK;
>  }
>  
> -- 
> 2.34.1

Re: [RFC PATCH] virtio: re-order vm_running and use_started checks

[Michael S. Tsirkin]

On Fri, Oct 14, 2022 at 02:21:08PM +0100, Alex Bennée wrote:
> During migration the virtio device state can be restored before we
> restart the VM. As no devices can be running while the VM is paused it
> makes sense to bail out early in that case.
> 
> This returns the order introduced in:
> 
>  9f6bcfd99f (hw/virtio: move vm_running check to virtio_device_started)
> 
> to what virtio-sock was doing longhand.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Cc: Christian Borntraeger <borntraeger@linux.ibm.com>


What happens now:

with this applied I get:

https://gitlab.com/mitsirkin/qemu/-/pipelines/685829158/failures

――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
stderr:
qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: -chardev socket,id=chr-reconnect,path=/tmp/vhost-test-QLKXU1/reconnect.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-QLKXU1/reconnect.sock,server=on
qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: -chardev socket,id=chr-connect-fail,path=/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on
qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: Failed to read msg header. Read 0 instead of 12. Original request 1.
qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: vhost_backend_init failed: Protocol error
qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: failed to init vhost_net for queue 0
qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on
qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: -chardev socket,id=chr-flags-mismatch,path=/tmp/vhost-test-3MO5U1/flags-mismatch.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-3MO5U1/flags-mismatch.sock,server=on
qemu-system-arm: Failed to write msg. Wrote -1 instead of 52.
qemu-system-arm: vhost_set_mem_table failed: Invalid argument (22)
qemu-system-arm: unable to start vhost net: 22: falling back on userspace virtio
vhost lacks feature mask 0x40000000 for backend
qemu-system-arm: failed to init vhost_net for queue 0
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 2 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 3 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
qemu-system-arm: Failed to set msg fds.
qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
qemu-system-arm: vhost VQ 0 ring restore failed: -5: Input/output error (5)
qemu-system-arm: ../hw/virtio/virtio-bus.c:211: void virtio_bus_release_ioeventfd(VirtioBusState *): Assertion `bus->ioeventfd_grabbed != 0' failed.
../tests/qtest/libqtest.c:188: kill_qemu() detected QEMU death from signal 6 (Aborted) (core dumped)
**
ERROR:../tests/qtest/qos-test.c:191:subprocess_run_one_test: child process (/arm/virt/virtio-mmio/virtio-bus/vhost-user-gpio-device/vhost-user-gpio/vhost-user-gpio-tests/read-guest-mem/memfile/subprocess [8735]) failed unexpectedly
(test program exited with status code -6)
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――




without this it passes:

https://gitlab.com/mitsirkin/qemu/-/jobs/3275949777


this only triggers under github, clang-system job.
trying to decide what to do now. revert just this?


> ---
>  include/hw/virtio/virtio.h | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
> index f41b4a7e64..ebb58feaac 100644
> --- a/include/hw/virtio/virtio.h
> +++ b/include/hw/virtio/virtio.h
> @@ -385,14 +385,14 @@ static inline bool virtio_is_big_endian(VirtIODevice *vdev)
>  
>  static inline bool virtio_device_started(VirtIODevice *vdev, uint8_t status)
>  {
> -    if (vdev->use_started) {
> -        return vdev->started;
> -    }
> -
>      if (!vdev->vm_running) {
>          return false;
>      }
>  
> +    if (vdev->use_started) {
> +        return vdev->started;
> +    }
> +
>      return status & VIRTIO_CONFIG_S_DRIVER_OK;
>  }
>  
> -- 
> 2.34.1

Re: [RFC PATCH] virtio: re-order vm_running and use_started checks

[Christian Borntraeger]

Am 04.11.22 um 16:56 schrieb Michael S. Tsirkin:
> On Fri, Oct 14, 2022 at 02:21:08PM +0100, Alex Bennée wrote:
>> During migration the virtio device state can be restored before we
>> restart the VM. As no devices can be running while the VM is paused it
>> makes sense to bail out early in that case.
>>
>> This returns the order introduced in:
>>
>>   9f6bcfd99f (hw/virtio: move vm_running check to virtio_device_started)
>>
>> to what virtio-sock was doing longhand.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
> 
> 
> What happens now:
> 
> with this applied I get:
> 
> https://gitlab.com/mitsirkin/qemu/-/pipelines/685829158/failures
> 
> ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> stderr:
> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: -chardev socket,id=chr-reconnect,path=/tmp/vhost-test-QLKXU1/reconnect.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-QLKXU1/reconnect.sock,server=on
> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: -chardev socket,id=chr-connect-fail,path=/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on
> qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: Failed to read msg header. Read 0 instead of 12. Original request 1.
> qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: vhost_backend_init failed: Protocol error
> qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: failed to init vhost_net for queue 0
> qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on
> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: -chardev socket,id=chr-flags-mismatch,path=/tmp/vhost-test-3MO5U1/flags-mismatch.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-3MO5U1/flags-mismatch.sock,server=on
> qemu-system-arm: Failed to write msg. Wrote -1 instead of 52.
> qemu-system-arm: vhost_set_mem_table failed: Invalid argument (22)
> qemu-system-arm: unable to start vhost net: 22: falling back on userspace virtio
> vhost lacks feature mask 0x40000000 for backend
> qemu-system-arm: failed to init vhost_net for queue 0
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 2 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 3 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> qemu-system-arm: vhost VQ 0 ring restore failed: -5: Input/output error (5)
> qemu-system-arm: ../hw/virtio/virtio-bus.c:211: void virtio_bus_release_ioeventfd(VirtioBusState *): Assertion `bus->ioeventfd_grabbed != 0' failed.
> ../tests/qtest/libqtest.c:188: kill_qemu() detected QEMU death from signal 6 (Aborted) (core dumped)
> **
> ERROR:../tests/qtest/qos-test.c:191:subprocess_run_one_test: child process (/arm/virt/virtio-mmio/virtio-bus/vhost-user-gpio-device/vhost-user-gpio/vhost-user-gpio-tests/read-guest-mem/memfile/subprocess [8735]) failed unexpectedly
> (test program exited with status code -6)
> ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
> 
> 
> 
> 
> without this it passes:
> 
> https://gitlab.com/mitsirkin/qemu/-/jobs/3275949777
> 
> 
> this only triggers under github, clang-system job.
> trying to decide what to do now. revert just this?
When we revert this save/restore for vsock is broken. Not sure, maybe we must use a more fine-grained fix as outlined in my initial mail?

Re: [RFC PATCH] virtio: re-order vm_running and use_started checks

[Michael S. Tsirkin]

On Fri, Nov 04, 2022 at 04:59:35PM +0100, Christian Borntraeger wrote:
> 
> 
> Am 04.11.22 um 16:56 schrieb Michael S. Tsirkin:
> > On Fri, Oct 14, 2022 at 02:21:08PM +0100, Alex Bennée wrote:
> > > During migration the virtio device state can be restored before we
> > > restart the VM. As no devices can be running while the VM is paused it
> > > makes sense to bail out early in that case.
> > > 
> > > This returns the order introduced in:
> > > 
> > >   9f6bcfd99f (hw/virtio: move vm_running check to virtio_device_started)
> > > 
> > > to what virtio-sock was doing longhand.
> > > 
> > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> > > Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
> > 
> > 
> > What happens now:
> > 
> > with this applied I get:
> > 
> > https://gitlab.com/mitsirkin/qemu/-/pipelines/685829158/failures
> > 
> > ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> > stderr:
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: -chardev socket,id=chr-reconnect,path=/tmp/vhost-test-QLKXU1/reconnect.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-QLKXU1/reconnect.sock,server=on
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: -chardev socket,id=chr-connect-fail,path=/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on
> > qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: Failed to read msg header. Read 0 instead of 12. Original request 1.
> > qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: vhost_backend_init failed: Protocol error
> > qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: failed to init vhost_net for queue 0
> > qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: -chardev socket,id=chr-flags-mismatch,path=/tmp/vhost-test-3MO5U1/flags-mismatch.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-3MO5U1/flags-mismatch.sock,server=on
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 52.
> > qemu-system-arm: vhost_set_mem_table failed: Invalid argument (22)
> > qemu-system-arm: unable to start vhost net: 22: falling back on userspace virtio
> > vhost lacks feature mask 0x40000000 for backend
> > qemu-system-arm: failed to init vhost_net for queue 0
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 2 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 3 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -5: Input/output error (5)
> > qemu-system-arm: ../hw/virtio/virtio-bus.c:211: void virtio_bus_release_ioeventfd(VirtioBusState *): Assertion `bus->ioeventfd_grabbed != 0' failed.
> > ../tests/qtest/libqtest.c:188: kill_qemu() detected QEMU death from signal 6 (Aborted) (core dumped)
> > **
> > ERROR:../tests/qtest/qos-test.c:191:subprocess_run_one_test: child process (/arm/virt/virtio-mmio/virtio-bus/vhost-user-gpio-device/vhost-user-gpio/vhost-user-gpio-tests/read-guest-mem/memfile/subprocess [8735]) failed unexpectedly
> > (test program exited with status code -6)
> > ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
> > 
> > 
> > 
> > 
> > without this it passes:
> > 
> > https://gitlab.com/mitsirkin/qemu/-/jobs/3275949777
> > 
> > 
> > this only triggers under github, clang-system job.
> > trying to decide what to do now. revert just this?
> When we revert this save/restore for vsock is broken. Not sure, maybe we must use a more fine-grained fix as outlined in my initial mail?

I realize this, we should also revert 9f6bcfd99f.

-- 
MST

Re: [RFC PATCH] virtio: re-order vm_running and use_started checks

[Michael S. Tsirkin]

On Fri, Nov 04, 2022 at 04:59:35PM +0100, Christian Borntraeger wrote:
> 
> 
> Am 04.11.22 um 16:56 schrieb Michael S. Tsirkin:
> > On Fri, Oct 14, 2022 at 02:21:08PM +0100, Alex Bennée wrote:
> > > During migration the virtio device state can be restored before we
> > > restart the VM. As no devices can be running while the VM is paused it
> > > makes sense to bail out early in that case.
> > > 
> > > This returns the order introduced in:
> > > 
> > >   9f6bcfd99f (hw/virtio: move vm_running check to virtio_device_started)
> > > 
> > > to what virtio-sock was doing longhand.
> > > 
> > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> > > Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
> > 
> > 
> > What happens now:
> > 
> > with this applied I get:
> > 
> > https://gitlab.com/mitsirkin/qemu/-/pipelines/685829158/failures
> > 
> > ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> > stderr:
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: -chardev socket,id=chr-reconnect,path=/tmp/vhost-test-QLKXU1/reconnect.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-QLKXU1/reconnect.sock,server=on
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: -chardev socket,id=chr-connect-fail,path=/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on
> > qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: Failed to read msg header. Read 0 instead of 12. Original request 1.
> > qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: vhost_backend_init failed: Protocol error
> > qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: failed to init vhost_net for queue 0
> > qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: -chardev socket,id=chr-flags-mismatch,path=/tmp/vhost-test-3MO5U1/flags-mismatch.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-3MO5U1/flags-mismatch.sock,server=on
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 52.
> > qemu-system-arm: vhost_set_mem_table failed: Invalid argument (22)
> > qemu-system-arm: unable to start vhost net: 22: falling back on userspace virtio
> > vhost lacks feature mask 0x40000000 for backend
> > qemu-system-arm: failed to init vhost_net for queue 0
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 2 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 3 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -5: Input/output error (5)
> > qemu-system-arm: ../hw/virtio/virtio-bus.c:211: void virtio_bus_release_ioeventfd(VirtioBusState *): Assertion `bus->ioeventfd_grabbed != 0' failed.
> > ../tests/qtest/libqtest.c:188: kill_qemu() detected QEMU death from signal 6 (Aborted) (core dumped)
> > **
> > ERROR:../tests/qtest/qos-test.c:191:subprocess_run_one_test: child process (/arm/virt/virtio-mmio/virtio-bus/vhost-user-gpio-device/vhost-user-gpio/vhost-user-gpio-tests/read-guest-mem/memfile/subprocess [8735]) failed unexpectedly
> > (test program exited with status code -6)
> > ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
> > 
> > 
> > 
> > 
> > without this it passes:
> > 
> > https://gitlab.com/mitsirkin/qemu/-/jobs/3275949777
> > 
> > 
> > this only triggers under github, clang-system job.
> > trying to decide what to do now. revert just this?
> When we revert this save/restore for vsock is broken. Not sure, maybe we must use a more fine-grained fix as outlined in my initial mail?



Could you take a look here pls?

https://gitlab.com/mitsirkin/qemu/-/tree/testrevert1



-- 
MST

Re: [RFC PATCH] virtio: re-order vm_running and use_started checks

[Alex Bennée]

"Michael S. Tsirkin" <mst@redhat.com> writes:

> On Fri, Oct 14, 2022 at 02:21:08PM +0100, Alex Bennée wrote:
>> During migration the virtio device state can be restored before we
>> restart the VM. As no devices can be running while the VM is paused it
>> makes sense to bail out early in that case.
>> 
>> This returns the order introduced in:
>> 
>>  9f6bcfd99f (hw/virtio: move vm_running check to virtio_device_started)
>> 
>> to what virtio-sock was doing longhand.
>> 
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
>
>
> What happens now:
>
> with this applied I get:
>
> https://gitlab.com/mitsirkin/qemu/-/pipelines/685829158/failures
>
> ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> stderr:
> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: -chardev
> socket,id=chr-reconnect,path=/tmp/vhost-test-QLKXU1/reconnect.sock,server=on:
> info: QEMU waiting for connection on:
> disconnected:unix:/tmp/vhost-test-QLKXU1/reconnect.sock,server=on
> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: -chardev
> socket,id=chr-connect-fail,path=/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on:
> info: QEMU waiting for connection on:
> disconnected:unix:/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on
> qemu-system-arm: -netdev
> vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: Failed to
> read msg header. Read 0 instead of 12. Original request 1.
> qemu-system-arm: -netdev
> vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on:
> vhost_backend_init failed: Protocol error
> qemu-system-arm: -netdev
> vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: failed to
> init vhost_net for queue 0
> qemu-system-arm: -netdev
> vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: info: QEMU
> waiting for connection on:
> disconnected:unix:/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on
> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: -chardev
> socket,id=chr-flags-mismatch,path=/tmp/vhost-test-3MO5U1/flags-mismatch.sock,server=on:
> info: QEMU waiting for connection on:
> disconnected:unix:/tmp/vhost-test-3MO5U1/flags-mismatch.sock,server=on
> qemu-system-arm: Failed to write msg. Wrote -1 instead of 52.
> qemu-system-arm: vhost_set_mem_table failed: Invalid argument (22)
> qemu-system-arm: unable to start vhost net: 22: falling back on userspace virtio
> vhost lacks feature mask 0x40000000 for backend
> qemu-system-arm: failed to init vhost_net for queue 0
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 2 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 3 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
> qemu-system-arm: Failed to set msg fds.
> qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> qemu-system-arm: vhost VQ 0 ring restore failed: -5: Input/output error (5)
> qemu-system-arm: ../hw/virtio/virtio-bus.c:211: void
> virtio_bus_release_ioeventfd(VirtioBusState *): Assertion
> `bus->ioeventfd_grabbed != 0' failed.
> ../tests/qtest/libqtest.c:188: kill_qemu() detected QEMU death from signal 6 (Aborted) (core dumped)
> **
> ERROR:../tests/qtest/qos-test.c:191:subprocess_run_one_test: child
> process
> (/arm/virt/virtio-mmio/virtio-bus/vhost-user-gpio-device/vhost-user-gpio/vhost-user-gpio-tests/read-guest-mem/memfile/subprocess
> [8735]) failed unexpectedly
> (test program exited with status code -6)
> ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
>
>
>
>
> without this it passes:
>
> https://gitlab.com/mitsirkin/qemu/-/jobs/3275949777
>
>
> this only triggers under github, clang-system job.
> trying to decide what to do now. revert just this?

I must admit I didn't run that directly. My build box is currently out
of commission but can we get more detail about the abort?

It looks like the vhost negotiation is totally broken and can't even
find the VQs

>
>
>> ---
>>  include/hw/virtio/virtio.h | 8 ++++----
>>  1 file changed, 4 insertions(+), 4 deletions(-)
>> 
>> diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
>> index f41b4a7e64..ebb58feaac 100644
>> --- a/include/hw/virtio/virtio.h
>> +++ b/include/hw/virtio/virtio.h
>> @@ -385,14 +385,14 @@ static inline bool virtio_is_big_endian(VirtIODevice *vdev)
>>  
>>  static inline bool virtio_device_started(VirtIODevice *vdev, uint8_t status)
>>  {
>> -    if (vdev->use_started) {
>> -        return vdev->started;
>> -    }
>> -
>>      if (!vdev->vm_running) {
>>          return false;
>>      }
>>  
>> +    if (vdev->use_started) {
>> +        return vdev->started;
>> +    }
>> +
>>      return status & VIRTIO_CONFIG_S_DRIVER_OK;
>>  }
>>  
>> -- 
>> 2.34.1


-- 
Alex Bennée

Re: [RFC PATCH] virtio: re-order vm_running and use_started checks

[Christian Borntraeger]

Am 04.11.22 um 17:14 schrieb Michael S. Tsirkin:
> On Fri, Nov 04, 2022 at 04:59:35PM +0100, Christian Borntraeger wrote:
>>
>>
>> Am 04.11.22 um 16:56 schrieb Michael S. Tsirkin:
>>> On Fri, Oct 14, 2022 at 02:21:08PM +0100, Alex Bennée wrote:
>>>> During migration the virtio device state can be restored before we
>>>> restart the VM. As no devices can be running while the VM is paused it
>>>> makes sense to bail out early in that case.
>>>>
>>>> This returns the order introduced in:
>>>>
>>>>    9f6bcfd99f (hw/virtio: move vm_running check to virtio_device_started)
>>>>
>>>> to what virtio-sock was doing longhand.
>>>>
>>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>>> Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
>>>
>>>
>>> What happens now:
>>>
>>> with this applied I get:
>>>
>>> https://gitlab.com/mitsirkin/qemu/-/pipelines/685829158/failures
>>>
>>> ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
>>> stderr:
>>> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
>>> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
>>> qemu-system-arm: Failed to set msg fds.
>>> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
>>> qemu-system-arm: -chardev socket,id=chr-reconnect,path=/tmp/vhost-test-QLKXU1/reconnect.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-QLKXU1/reconnect.sock,server=on
>>> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
>>> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
>>> qemu-system-arm: Failed to set msg fds.
>>> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
>>> qemu-system-arm: -chardev socket,id=chr-connect-fail,path=/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on
>>> qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: Failed to read msg header. Read 0 instead of 12. Original request 1.
>>> qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: vhost_backend_init failed: Protocol error
>>> qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: failed to init vhost_net for queue 0
>>> qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on
>>> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
>>> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
>>> qemu-system-arm: Failed to set msg fds.
>>> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
>>> qemu-system-arm: -chardev socket,id=chr-flags-mismatch,path=/tmp/vhost-test-3MO5U1/flags-mismatch.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-3MO5U1/flags-mismatch.sock,server=on
>>> qemu-system-arm: Failed to write msg. Wrote -1 instead of 52.
>>> qemu-system-arm: vhost_set_mem_table failed: Invalid argument (22)
>>> qemu-system-arm: unable to start vhost net: 22: falling back on userspace virtio
>>> vhost lacks feature mask 0x40000000 for backend
>>> qemu-system-arm: failed to init vhost_net for queue 0
>>> qemu-system-arm: Failed to set msg fds.
>>> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
>>> qemu-system-arm: Failed to set msg fds.
>>> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
>>> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
>>> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
>>> qemu-system-arm: Failed to set msg fds.
>>> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
>>> qemu-system-arm: Failed to set msg fds.
>>> qemu-system-arm: vhost VQ 2 ring restore failed: -22: Invalid argument (22)
>>> qemu-system-arm: Failed to set msg fds.
>>> qemu-system-arm: vhost VQ 3 ring restore failed: -22: Invalid argument (22)
>>> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
>>> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
>>> qemu-system-arm: Failed to set msg fds.
>>> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
>>> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
>>> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
>>> qemu-system-arm: Failed to set msg fds.
>>> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
>>> qemu-system-arm: Failed to set msg fds.
>>> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
>>> qemu-system-arm: Failed to set msg fds.
>>> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
>>> qemu-system-arm: Failed to set msg fds.
>>> qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
>>> qemu-system-arm: Failed to set msg fds.
>>> qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
>>> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
>>> qemu-system-arm: vhost VQ 0 ring restore failed: -5: Input/output error (5)
>>> qemu-system-arm: ../hw/virtio/virtio-bus.c:211: void virtio_bus_release_ioeventfd(VirtioBusState *): Assertion `bus->ioeventfd_grabbed != 0' failed.
>>> ../tests/qtest/libqtest.c:188: kill_qemu() detected QEMU death from signal 6 (Aborted) (core dumped)
>>> **
>>> ERROR:../tests/qtest/qos-test.c:191:subprocess_run_one_test: child process (/arm/virt/virtio-mmio/virtio-bus/vhost-user-gpio-device/vhost-user-gpio/vhost-user-gpio-tests/read-guest-mem/memfile/subprocess [8735]) failed unexpectedly
>>> (test program exited with status code -6)
>>> ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
>>>
>>>
>>>
>>>
>>> without this it passes:
>>>
>>> https://gitlab.com/mitsirkin/qemu/-/jobs/3275949777
>>>
>>>
>>> this only triggers under github, clang-system job.
>>> trying to decide what to do now. revert just this?
>> When we revert this save/restore for vsock is broken. Not sure, maybe we must use a more fine-grained fix as outlined in my initial mail?
> 
> 
> 
> Could you take a look here pls?
> 
> https://gitlab.com/mitsirkin/qemu/-/tree/testrevert1

I kicked of our regression test suite for s390.

Re: [RFC PATCH] virtio: re-order vm_running and use_started checks

[Christian Borntraeger]

Am 04.11.22 um 17:51 schrieb Christian Borntraeger:
> 
> 
> Am 04.11.22 um 17:14 schrieb Michael S. Tsirkin:
>> On Fri, Nov 04, 2022 at 04:59:35PM +0100, Christian Borntraeger wrote:
>>>
>>>
>>> Am 04.11.22 um 16:56 schrieb Michael S. Tsirkin:
>>>> On Fri, Oct 14, 2022 at 02:21:08PM +0100, Alex Bennée wrote:
>>>>> During migration the virtio device state can be restored before we
>>>>> restart the VM. As no devices can be running while the VM is paused it
>>>>> makes sense to bail out early in that case.
>>>>>
>>>>> This returns the order introduced in:
>>>>>
>>>>>    9f6bcfd99f (hw/virtio: move vm_running check to virtio_device_started)
>>>>>
>>>>> to what virtio-sock was doing longhand.
>>>>>
>>>>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>>>>> Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
>>>>
>>>>
>>>> What happens now:
>>>>
>>>> with this applied I get:
>>>>
>>>> https://gitlab.com/mitsirkin/qemu/-/pipelines/685829158/failures
>>>>
>>>> ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
>>>> stderr:
>>>> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
>>>> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
>>>> qemu-system-arm: Failed to set msg fds.
>>>> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
>>>> qemu-system-arm: -chardev socket,id=chr-reconnect,path=/tmp/vhost-test-QLKXU1/reconnect.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-QLKXU1/reconnect.sock,server=on
>>>> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
>>>> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
>>>> qemu-system-arm: Failed to set msg fds.
>>>> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
>>>> qemu-system-arm: -chardev socket,id=chr-connect-fail,path=/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on
>>>> qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: Failed to read msg header. Read 0 instead of 12. Original request 1.
>>>> qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: vhost_backend_init failed: Protocol error
>>>> qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: failed to init vhost_net for queue 0
>>>> qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on
>>>> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
>>>> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
>>>> qemu-system-arm: Failed to set msg fds.
>>>> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
>>>> qemu-system-arm: -chardev socket,id=chr-flags-mismatch,path=/tmp/vhost-test-3MO5U1/flags-mismatch.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-3MO5U1/flags-mismatch.sock,server=on
>>>> qemu-system-arm: Failed to write msg. Wrote -1 instead of 52.
>>>> qemu-system-arm: vhost_set_mem_table failed: Invalid argument (22)
>>>> qemu-system-arm: unable to start vhost net: 22: falling back on userspace virtio
>>>> vhost lacks feature mask 0x40000000 for backend
>>>> qemu-system-arm: failed to init vhost_net for queue 0
>>>> qemu-system-arm: Failed to set msg fds.
>>>> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
>>>> qemu-system-arm: Failed to set msg fds.
>>>> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
>>>> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
>>>> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
>>>> qemu-system-arm: Failed to set msg fds.
>>>> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
>>>> qemu-system-arm: Failed to set msg fds.
>>>> qemu-system-arm: vhost VQ 2 ring restore failed: -22: Invalid argument (22)
>>>> qemu-system-arm: Failed to set msg fds.
>>>> qemu-system-arm: vhost VQ 3 ring restore failed: -22: Invalid argument (22)
>>>> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
>>>> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
>>>> qemu-system-arm: Failed to set msg fds.
>>>> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
>>>> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
>>>> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
>>>> qemu-system-arm: Failed to set msg fds.
>>>> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
>>>> qemu-system-arm: Failed to set msg fds.
>>>> qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
>>>> qemu-system-arm: Failed to set msg fds.
>>>> qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
>>>> qemu-system-arm: Failed to set msg fds.
>>>> qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
>>>> qemu-system-arm: Failed to set msg fds.
>>>> qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
>>>> qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
>>>> qemu-system-arm: vhost VQ 0 ring restore failed: -5: Input/output error (5)
>>>> qemu-system-arm: ../hw/virtio/virtio-bus.c:211: void virtio_bus_release_ioeventfd(VirtioBusState *): Assertion `bus->ioeventfd_grabbed != 0' failed.
>>>> ../tests/qtest/libqtest.c:188: kill_qemu() detected QEMU death from signal 6 (Aborted) (core dumped)
>>>> **
>>>> ERROR:../tests/qtest/qos-test.c:191:subprocess_run_one_test: child process (/arm/virt/virtio-mmio/virtio-bus/vhost-user-gpio-device/vhost-user-gpio/vhost-user-gpio-tests/read-guest-mem/memfile/subprocess [8735]) failed unexpectedly
>>>> (test program exited with status code -6)
>>>> ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
>>>>
>>>>
>>>>
>>>>
>>>> without this it passes:
>>>>
>>>> https://gitlab.com/mitsirkin/qemu/-/jobs/3275949777
>>>>
>>>>
>>>> this only triggers under github, clang-system job.
>>>> trying to decide what to do now. revert just this?
>>> When we revert this save/restore for vsock is broken. Not sure, maybe we must use a more fine-grained fix as outlined in my initial mail?
>>
>>
>>
>> Could you take a look here pls?
>>
>> https://gitlab.com/mitsirkin/qemu/-/tree/testrevert1
> 
> I kicked of our regression test suite for s390.

This branch also seems to fix my original problem.
Regression is still running but looks good so far.

Re: [RFC PATCH] virtio: re-order vm_running and use_started checks

[Michael S. Tsirkin]

On Fri, Nov 04, 2022 at 04:31:08PM +0000, Alex Bennée wrote:
> 
> "Michael S. Tsirkin" <mst@redhat.com> writes:
> 
> > On Fri, Oct 14, 2022 at 02:21:08PM +0100, Alex Bennée wrote:
> >> During migration the virtio device state can be restored before we
> >> restart the VM. As no devices can be running while the VM is paused it
> >> makes sense to bail out early in that case.
> >> 
> >> This returns the order introduced in:
> >> 
> >>  9f6bcfd99f (hw/virtio: move vm_running check to virtio_device_started)
> >> 
> >> to what virtio-sock was doing longhand.
> >> 
> >> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> >> Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
> >
> >
> > What happens now:
> >
> > with this applied I get:
> >
> > https://gitlab.com/mitsirkin/qemu/-/pipelines/685829158/failures
> >
> > ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> > stderr:
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: -chardev
> > socket,id=chr-reconnect,path=/tmp/vhost-test-QLKXU1/reconnect.sock,server=on:
> > info: QEMU waiting for connection on:
> > disconnected:unix:/tmp/vhost-test-QLKXU1/reconnect.sock,server=on
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: -chardev
> > socket,id=chr-connect-fail,path=/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on:
> > info: QEMU waiting for connection on:
> > disconnected:unix:/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on
> > qemu-system-arm: -netdev
> > vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: Failed to
> > read msg header. Read 0 instead of 12. Original request 1.
> > qemu-system-arm: -netdev
> > vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on:
> > vhost_backend_init failed: Protocol error
> > qemu-system-arm: -netdev
> > vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: failed to
> > init vhost_net for queue 0
> > qemu-system-arm: -netdev
> > vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: info: QEMU
> > waiting for connection on:
> > disconnected:unix:/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: -chardev
> > socket,id=chr-flags-mismatch,path=/tmp/vhost-test-3MO5U1/flags-mismatch.sock,server=on:
> > info: QEMU waiting for connection on:
> > disconnected:unix:/tmp/vhost-test-3MO5U1/flags-mismatch.sock,server=on
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 52.
> > qemu-system-arm: vhost_set_mem_table failed: Invalid argument (22)
> > qemu-system-arm: unable to start vhost net: 22: falling back on userspace virtio
> > vhost lacks feature mask 0x40000000 for backend
> > qemu-system-arm: failed to init vhost_net for queue 0
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 2 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 3 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
> > qemu-system-arm: Failed to set msg fds.
> > qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
> > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > qemu-system-arm: vhost VQ 0 ring restore failed: -5: Input/output error (5)
> > qemu-system-arm: ../hw/virtio/virtio-bus.c:211: void
> > virtio_bus_release_ioeventfd(VirtioBusState *): Assertion
> > `bus->ioeventfd_grabbed != 0' failed.
> > ../tests/qtest/libqtest.c:188: kill_qemu() detected QEMU death from signal 6 (Aborted) (core dumped)
> > **
> > ERROR:../tests/qtest/qos-test.c:191:subprocess_run_one_test: child
> > process
> > (/arm/virt/virtio-mmio/virtio-bus/vhost-user-gpio-device/vhost-user-gpio/vhost-user-gpio-tests/read-guest-mem/memfile/subprocess
> > [8735]) failed unexpectedly
> > (test program exited with status code -6)
> > ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
> >
> >
> >
> >
> > without this it passes:
> >
> > https://gitlab.com/mitsirkin/qemu/-/jobs/3275949777
> >
> >
> > this only triggers under github, clang-system job.
> > trying to decide what to do now. revert just this?
> 
> I must admit I didn't run that directly. My build box is currently out
> of commission but can we get more detail about the abort?

Just run it under gitlab. The only trick is to set QEMU_CI to 1
when running the pipeline.


> It looks like the vhost negotiation is totally broken and can't even
> find the VQs

Donnu really. I'm trying to figure out what kind of revert gets
a working tree.


> >
> >
> >> ---
> >>  include/hw/virtio/virtio.h | 8 ++++----
> >>  1 file changed, 4 insertions(+), 4 deletions(-)
> >> 
> >> diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
> >> index f41b4a7e64..ebb58feaac 100644
> >> --- a/include/hw/virtio/virtio.h
> >> +++ b/include/hw/virtio/virtio.h
> >> @@ -385,14 +385,14 @@ static inline bool virtio_is_big_endian(VirtIODevice *vdev)
> >>  
> >>  static inline bool virtio_device_started(VirtIODevice *vdev, uint8_t status)
> >>  {
> >> -    if (vdev->use_started) {
> >> -        return vdev->started;
> >> -    }
> >> -
> >>      if (!vdev->vm_running) {
> >>          return false;
> >>      }
> >>  
> >> +    if (vdev->use_started) {
> >> +        return vdev->started;
> >> +    }
> >> +
> >>      return status & VIRTIO_CONFIG_S_DRIVER_OK;
> >>  }
> >>  
> >> -- 
> >> 2.34.1
> 
> 
> -- 
> Alex Bennée

Re: [RFC PATCH] virtio: re-order vm_running and use_started checks

[Michael S. Tsirkin]

On Fri, Nov 04, 2022 at 05:58:17PM +0100, Christian Borntraeger wrote:
> Am 04.11.22 um 17:51 schrieb Christian Borntraeger:
> > 
> > 
> > Am 04.11.22 um 17:14 schrieb Michael S. Tsirkin:
> > > On Fri, Nov 04, 2022 at 04:59:35PM +0100, Christian Borntraeger wrote:
> > > > 
> > > > 
> > > > Am 04.11.22 um 16:56 schrieb Michael S. Tsirkin:
> > > > > On Fri, Oct 14, 2022 at 02:21:08PM +0100, Alex Bennée wrote:
> > > > > > During migration the virtio device state can be restored before we
> > > > > > restart the VM. As no devices can be running while the VM is paused it
> > > > > > makes sense to bail out early in that case.
> > > > > > 
> > > > > > This returns the order introduced in:
> > > > > > 
> > > > > >    9f6bcfd99f (hw/virtio: move vm_running check to virtio_device_started)
> > > > > > 
> > > > > > to what virtio-sock was doing longhand.
> > > > > > 
> > > > > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> > > > > > Cc: Christian Borntraeger <borntraeger@linux.ibm.com>
> > > > > 
> > > > > 
> > > > > What happens now:
> > > > > 
> > > > > with this applied I get:
> > > > > 
> > > > > https://gitlab.com/mitsirkin/qemu/-/pipelines/685829158/failures
> > > > > 
> > > > > ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> > > > > stderr:
> > > > > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > > > > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > > > > qemu-system-arm: -chardev socket,id=chr-reconnect,path=/tmp/vhost-test-QLKXU1/reconnect.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-QLKXU1/reconnect.sock,server=on
> > > > > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > > > > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > > > > qemu-system-arm: -chardev socket,id=chr-connect-fail,path=/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on
> > > > > qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: Failed to read msg header. Read 0 instead of 12. Original request 1.
> > > > > qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: vhost_backend_init failed: Protocol error
> > > > > qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: failed to init vhost_net for queue 0
> > > > > qemu-system-arm: -netdev vhost-user,id=hs0,chardev=chr-connect-fail,vhostforce=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-L9Q6U1/connect-fail.sock,server=on
> > > > > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > > > > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > > > > qemu-system-arm: -chardev socket,id=chr-flags-mismatch,path=/tmp/vhost-test-3MO5U1/flags-mismatch.sock,server=on: info: QEMU waiting for connection on: disconnected:unix:/tmp/vhost-test-3MO5U1/flags-mismatch.sock,server=on
> > > > > qemu-system-arm: Failed to write msg. Wrote -1 instead of 52.
> > > > > qemu-system-arm: vhost_set_mem_table failed: Invalid argument (22)
> > > > > qemu-system-arm: unable to start vhost net: 22: falling back on userspace virtio
> > > > > vhost lacks feature mask 0x40000000 for backend
> > > > > qemu-system-arm: failed to init vhost_net for queue 0
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > > > > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > > > > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 2 ring restore failed: -22: Invalid argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 3 ring restore failed: -22: Invalid argument (22)
> > > > > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > > > > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > > > > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > > > > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
> > > > > qemu-system-arm: Failed to set msg fds.
> > > > > qemu-system-arm: vhost_set_vring_call failed: Invalid argument (22)
> > > > > qemu-system-arm: Failed to write msg. Wrote -1 instead of 20.
> > > > > qemu-system-arm: vhost VQ 0 ring restore failed: -5: Input/output error (5)
> > > > > qemu-system-arm: ../hw/virtio/virtio-bus.c:211: void virtio_bus_release_ioeventfd(VirtioBusState *): Assertion `bus->ioeventfd_grabbed != 0' failed.
> > > > > ../tests/qtest/libqtest.c:188: kill_qemu() detected QEMU death from signal 6 (Aborted) (core dumped)
> > > > > **
> > > > > ERROR:../tests/qtest/qos-test.c:191:subprocess_run_one_test: child process (/arm/virt/virtio-mmio/virtio-bus/vhost-user-gpio-device/vhost-user-gpio/vhost-user-gpio-tests/read-guest-mem/memfile/subprocess [8735]) failed unexpectedly
> > > > > (test program exited with status code -6)
> > > > > ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > without this it passes:
> > > > > 
> > > > > https://gitlab.com/mitsirkin/qemu/-/jobs/3275949777
> > > > > 
> > > > > 
> > > > > this only triggers under github, clang-system job.
> > > > > trying to decide what to do now. revert just this?
> > > > When we revert this save/restore for vsock is broken. Not sure, maybe we must use a more fine-grained fix as outlined in my initial mail?
> > > 
> > > 
> > > 
> > > Could you take a look here pls?
> > > 
> > > https://gitlab.com/mitsirkin/qemu/-/tree/testrevert1
> > 
> > I kicked of our regression test suite for s390.
> 
> This branch also seems to fix my original problem.
> Regression is still running but looks good so far.

Good to know. And I think that's a better fix in fact it's just a
couple of lines. However, the original gitlab failure is still
with us there :( Trying to figure out what is going on.