From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=E4a+h8zhsQFy46ieIh6Kvn/jOEzS7Fv/bh2/6yR4ys4=; b=Zw5iMHq1ZDShANnKXLNPyvu//hbmyCu8HWmjidkJmbNkRMNOLCO6dQzXS2eNkgASRfxNvlk2gT94S/+QVg17uMfSTaRSZeBw46d+lToTcEalewTyHdxPq8EGVtedrc3Xac4O5JNB5bpDU0DZxe4HOYMkJfyQgtBb6jdrMcr1DDAL/fr9Nn2a57jQi9T3fMoAqe+zY+PpHcEbsM0b4LyrmuSVT0qCeNYqV1MSOHi4PpNVHOoRdfWcUC25utZA/j+C8YttW79VBK9UPpCdI7ROwG1WssORxg+iv55njnjNypPYBrShQ6sbf4hfsDKWQ8eAPwwCf3IFwlMraFdkVDo9+g== From: Prashant Dewan Date: Tue, 24 May 2022 14:29:12 +0000 Message-ID: MIME-Version: 1.0 Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_BY5PR14MB38937584E1C443A7BB3B0BCF86D79BY5PR14MB3893namp_" Subject: [Virtio-fs] vhost_set_vring_kick failed List-Id: Development discussions about virtio-fs List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "virtio-fs@redhat.com" --_000_BY5PR14MB38937584E1C443A7BB3B0BCF86D79BY5PR14MB3893namp_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable I am trying to setup virtfs using KVM and QEMU. 1. Inside the VM, I do * mount -t virtiofs myfs /dev/myfile --succeeds * cd /dev/myfile -hangs 1. The host logs - qemu-system: Failed to write msg. Wrote -1 instead of 20. May 24 14:01:08 device-myhost[644]: qemu-system: vhost_set_vring_kick faile= d: Invalid argument (22) May 24 14:01:08 device-myhost[644]: qemu-system: Failed to set msg fds. May 24 14:01:08 device-myhost[644]: qemu-system: vhost VQ 0 ring restore fa= iled: -1: Invalid argument (22) May 24 14:01:08 device-myhost[644]: qemu-system: Error starting vhost: 22 May 24 14:01:08 device-myhost[644]: qemu-system: Failed to set msg fds. May 24 14:01:08 device-myhost[644]: qemu-system: vhost_set_vring_call faile= d: Resource temporarily unavailable (11) May 24 14:01:08 device-myhost[644]: qemu-system: Failed to set msg fds. May 24 14:01:08 device-myhost[644]: qemu-system: vhost_set_vring_call faile= d: Resource temporarily unavailable (11) May 24 14:01:08 device-myhost[644]: qemu-system: Failed to read from slave. 1. Qemu parameters - "-device", "vhost-user-fs-device,queue-size=3D1024,chardev=3Dchar0,tag=3Dmyfs", "-object", "memory-backend-memfd,id=3Dmem,size=3D4G,share=3Don" Any input is appreciated... Thanks --_000_BY5PR14MB38937584E1C443A7BB3B0BCF86D79BY5PR14MB3893namp_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
I am trying to setup virtfs using KVM and QEMU.

  1. Inside the VM, I do 
    1. mount -t virtiofs myfs /dev/myfile  --succeeds
    2. c= d /dev/myfile  -hangs

  1. The host logs - 

 qemu-system: Failed to write msg. Wrote -1 instead of 20.
May 24 14:01:08 device-myhost[644]: qemu-system: vhost_set_vring_kick = failed: Invalid argument (22)
May 24 14:01:08 device-myhost[644]: qemu-system: Failed to s= et msg fds.
May 24 14:01:08 device-myhost[644]: qemu-system: vhost VQ 0 = ring restore failed: -1: Invalid argument (22)
May 24 14:01:08 device-myhost[644]: qemu-system: Error start= ing vhost: 22
May 24 14:01:08 device-myhost[644]: qemu-system: Failed to s= et msg fds.
May 24 14:01:08 device-myhost[644]: qemu-system: vhost_set_v= ring_call failed: Resource temporarily unavailable (11)
May 24 14:01:08 device-myhost[644]: qemu-system: Failed to s= et msg fds.
May 24 14:01:08 device-myhost[644]: qemu-system: vhost_set_v= ring_call failed: Resource temporarily unavailable (11)
May 24 14:01:08 device-myhost[644]: qemu-system: Failed to read f= rom slave.

  1.  Qemu parameters -

 "-device",
    "vhost-user-fs-device,queue-size=3D1024,chardev=3Dc= har0,tag=3Dmyfs",
    "-object",
    "memory-backend-memfd,id=3Dmem,size=3D4G,share=3Don"= ;

Any input is appreciated...

Thanks

--_000_BY5PR14MB38937584E1C443A7BB3B0BCF86D79BY5PR14MB3893namp_-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Wed, 25 May 2022 17:02:24 +0100 From: Stefan Hajnoczi Message-ID: References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="AOWdCJO3rdfoV8S7" Content-Disposition: inline In-Reply-To: Subject: Re: [Virtio-fs] vhost_set_vring_kick failed List-Id: Development discussions about virtio-fs List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Prashant Dewan Cc: "virtio-fs@redhat.com" --AOWdCJO3rdfoV8S7 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, May 24, 2022 at 02:29:12PM +0000, Prashant Dewan wrote: > I am trying to setup virtfs using KVM and QEMU. >=20 >=20 > 1. Inside the VM, I do > * mount -t virtiofs myfs /dev/myfile --succeeds > * cd /dev/myfile -hangs >=20 >=20 > 1. The host logs - >=20 > qemu-system: Failed to write msg. Wrote -1 instead of 20. Sending a vhost-user protocol message failed. It is likely that the virtiofsd process terminated and the UNIX domain socket closed on QEMU. Is there any output from virtiofsd? Stefan --AOWdCJO3rdfoV8S7 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEhpWov9P5fNqsNXdanKSrs4Grc8gFAmKOUxAACgkQnKSrs4Gr c8gsMAgAtCVE+Nve97olEp+e2B948lRAk/w3rdRJVCc8fcIrOEYpO0mJCw2Toj8y atmOWV5gjC23VxAkgBsoHK34PN7i68pdp84vUy89PBycKpuXDgrVo3Z143qvtXqQ FvaVZvhLmK7tKOXR6ZTBZb4ts2n6Gn91Hjlw6W0CokCBYfOMOQUI4FzRuuETJaCn +zrc71mLTVVd9kPU4fAPcO/Jc8cF6pUe/yMLQBeiGSoz1W3Tn2zF3Hgx9VO3SVxO ILWlsT3ZR6XEI8iB3RgyLQJw1omqShaV2cAnr2CvGTzSSTS0NJIh+nWlGRxtIViU XER3OOYQ7YpLjh8r0yAcswYkD0w0Fw== =oAu9 -----END PGP SIGNATURE----- --AOWdCJO3rdfoV8S7-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rXnZZ5I7MI7wE6hETy432AHLU4rYu+Yodvl2mTkJ+1s=; b=Guon4+YBUWKG1dOlQh8WPDZNeZutnF3e/zll8Zhfn0LYtSLeBVWivzl/2zLmBWymGni+LBTnYz+HCKv0rernfQN5sh7H6ExPT+SHTs5+IsqL45Sd5cJhnF9TFful2JEKqQBEKHULv/PKruWbRm2OkB+rCTfMtGHHky8ApcZWawxrX9BbnZisjAtiZV+SYnUi21wzKfhIRgg2B2gf9d3Z9/ex6NpYpw2ZhKbVJtyOaspegCNyGRqHNpNnoPlULrxs0nTSA/UHIcRFnK2evIsCl75SmLEZH9v3R0EMPDch+r5QXu2hmOLsQBKOlJDji23TIynmtz3XNYAGY2AmnC8TkA== From: Prashant Dewan Date: Thu, 26 May 2022 05:34:42 +0000 Message-ID: MIME-Version: 1.0 Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_BY5PR14MB3893A6AA17C5FEBE843B640286D99BY5PR14MB3893namp_" Subject: Re: [Virtio-fs] vhost_set_vring_kick failed List-Id: Development discussions about virtio-fs List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Stefan Hajnoczi Cc: "virtio-fs@redhat.com" --_000_BY5PR14MB3893A6AA17C5FEBE843B640286D99BY5PR14MB3893namp_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hello Stefan, Here is the output from virtofsd.. ------------------------------------ - Journal begins at Thu 2022-05-26 04:23:42 UTC, ends at Thu 2022-05-26 04:= 41:57 UTC. -- May 26 04:24:13 localhost systemd[1]: Started virtio-fs vhost-user device d= aemon.<>. May 26 04:24:13 localhost virtiofsd[505]: [2022-05-26T04:24:13Z INFO virti= ofsd] Waiting for vhost-user socket connection... May 26 04:27:11 device-3e4b479e2ee9c647 virtiofsd[505]: [2022-05-26T04:27:1= 1Z INFO virtiofsd] Client connected, servicing requests May 26 04:27:33 device-3e4b479e2ee9c647 virtiofsd[505]: [2022-05-26T04:27:3= 3Z ERROR virtiofsd] Waiting for daemon failed: HandleRequest(InvalidParam) May 26 04:27:33 device-3e4b479e2ee9c647 systemd[1]: virtiofsd.service: Deac= tivated successfully. May 26 04:27:33 device-3e4b479e2ee9c647 systemd[1]: virtiofsd.service: Sche= duled restart job, restart counter is at 1. ________________________________ From: Stefan Hajnoczi Sent: Wednesday, May 25, 2022 4:02 PM To: Prashant Dewan Cc: virtio-fs@redhat.com Subject: Re: [Virtio-fs] vhost_set_vring_kick failed On Tue, May 24, 2022 at 02:29:12PM +0000, Pra Dew wrote: > I am trying to setup virtfs using KVM and QEMU. > > > 1. Inside the VM, I do > * mount -t virtiofs myfs /dev/myfile --succeeds > * cd /dev/myfile -hangs > > > 1. The host logs - > > qemu-system: Failed to write msg. Wrote -1 instead of 20. Sending a vhost-user protocol message failed. It is likely that the virtiofsd process terminated and the UNIX domain socket closed on QEMU. Is there any output from virtiofsd? Stefan --_000_BY5PR14MB3893A6AA17C5FEBE843B640286D99BY5PR14MB3893namp_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Hello Stefan, 

Here is the output from virtofsd..

------------------------------------
- Journal begins at Thu 2022-05-26 04:23:42 U= TC, ends at Thu 2022-05-26 04:41:57 UTC. --
May 26 04:24:13 localhost systemd[1]: Started virtio-fs vhost-user dev= ice daemon.<<Placeholder: run this daemon as non-privileged user>&= gt;.
May 26 04:24:13 localhost virtiofsd[505]: [2022-05-26T04:24:13Z INFO &= nbsp;virtiofsd] Waiting for vhost-user socket connection...
May 26 04:27:11 device-3e4b479e2ee9c647 virtiofsd[505]: [2022-05-26T04= :27:11Z INFO  virtiofsd] Client connected, servicing requests
May 26 04:27:33 device-3e4b479e2ee9c647 virtiofsd[505]: [2022-05-26T04= :27:33Z ERROR virtiofsd] Waiting for daemon failed: HandleRequest(Invali= dParam)
May 26 04:27:33 device-3e4b479e2ee9c647 systemd[1]: virtiofsd.service:= Deactivated successfully.
May 26 04:27:33 device-3e4b479e2ee9c647 systemd[1]: virtiofsd.service: Sche= duled restart job, restart counter is at 1.

From: Stefan Hajnoczi
Sent: Wednesday, May 25, 2022 4:02 PM
To: Prashant Dewan
Cc: virtio-fs@redhat.com
Subject: Re: [Virtio-fs] vhost_set_vring_kick failed

On Tue, May 24, 2022 at 02:29:12PM = +0000, Pra Dew wrote:
> I am trying to setup virtfs using KVM and QEMU.
>
>
>   1.  Inside the VM, I do
>      *   mount -t virtiofs myfs /de= v/myfile  --succeeds
>      *   cd /dev/myfile  -hang= s
>
>
>   1.  The host logs -
>
>  qemu-system: Failed to write msg. Wrote -1 instead of 20.

Sending a vhost-user protocol message failed. It is likely that the
virtiofsd process terminated and the UNIX domain socket closed on QEMU.

Is there any output from virtiofsd?

Stefan
--_000_BY5PR14MB3893A6AA17C5FEBE843B640286D99BY5PR14MB3893namp_-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Thu, 26 May 2022 08:49:34 +0100 From: Stefan Hajnoczi Message-ID: References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="rlDir5U3QoVKFdNx" Content-Disposition: inline In-Reply-To: Subject: Re: [Virtio-fs] vhost_set_vring_kick failed List-Id: Development discussions about virtio-fs List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Prashant Dewan Cc: "virtio-fs@redhat.com" , slp@redhat.com, gmaglione@redhat.com --rlDir5U3QoVKFdNx Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, May 26, 2022 at 05:34:42AM +0000, Prashant Dewan wrote: > Hello Stefan, >=20 > Here is the output from virtofsd.. >=20 > ------------------------------------ > - Journal begins at Thu 2022-05-26 04:23:42 UTC, ends at Thu 2022-05-26 0= 4:41:57 UTC. -- > May 26 04:24:13 localhost systemd[1]: Started virtio-fs vhost-user device= daemon.<>. > May 26 04:24:13 localhost virtiofsd[505]: [2022-05-26T04:24:13Z INFO vir= tiofsd] Waiting for vhost-user socket connection... > May 26 04:27:11 device-3e4b479e2ee9c647 virtiofsd[505]: [2022-05-26T04:27= :11Z INFO virtiofsd] Client connected, servicing requests > May 26 04:27:33 device-3e4b479e2ee9c647 virtiofsd[505]: [2022-05-26T04:27= :33Z ERROR virtiofsd] Waiting for daemon failed: HandleRequest(InvalidParam) virtiofsd terminated. I have CCed Sergio and German, who may have suggestions about what to check next. Stefan > May 26 04:27:33 device-3e4b479e2ee9c647 systemd[1]: virtiofsd.service: De= activated successfully. > May 26 04:27:33 device-3e4b479e2ee9c647 systemd[1]: virtiofsd.service: Sc= heduled restart job, restart counter is at 1. >=20 > ________________________________ > From: Stefan Hajnoczi > Sent: Wednesday, May 25, 2022 4:02 PM > To: Prashant Dewan > Cc: virtio-fs@redhat.com > Subject: Re: [Virtio-fs] vhost_set_vring_kick failed >=20 > On Tue, May 24, 2022 at 02:29:12PM +0000, Pra Dew wrote: > > I am trying to setup virtfs using KVM and QEMU. > > > > > > 1. Inside the VM, I do > > * mount -t virtiofs myfs /dev/myfile --succeeds > > * cd /dev/myfile -hangs > > > > > > 1. The host logs - > > > > qemu-system: Failed to write msg. Wrote -1 instead of 20. >=20 > Sending a vhost-user protocol message failed. It is likely that the > virtiofsd process terminated and the UNIX domain socket closed on QEMU. >=20 > Is there any output from virtiofsd? >=20 > Stefan --rlDir5U3QoVKFdNx Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEhpWov9P5fNqsNXdanKSrs4Grc8gFAmKPMQ4ACgkQnKSrs4Gr c8jg/gf9GEyzok9tVmhJzZu3hEJuC1UiHLt8l8uXnde55S/Iw+KF5muW8zQgzkl7 K6u2S1Oku2e1QvyKvgvu8/7Ntb73kXTa1UIwXB7zjaLSNQ559QjqHpbeTL37iNzD CB867aFuht2R7DBHb+5CBXfKNOiRvoFJyGIHnBZzv4rPOxlHLcE7BipYeKAFVJqr MXrWRcPLg1z5lNvNUjGIje2xdgvmmB240X/MfP2HlKJF3ZWBZMavmQDnXomtjelr pAXOp63Rx1tVqvoyDj62do/Ak/F25Gy7v+jLp48jlaZmbY2gosEtn6dwkfKfdi6B U/ZuEufMY/jm/fq1NQvYlkDVdvIvAA== =s2mN -----END PGP SIGNATURE----- --rlDir5U3QoVKFdNx-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Thu, 26 May 2022 10:06:46 +0200 From: Sergio Lopez Message-ID: <20220526080646.vsfmqgxyiglhe64r@mhamilton> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="feyfazohf67u3abn" Content-Disposition: inline In-Reply-To: Subject: Re: [Virtio-fs] vhost_set_vring_kick failed List-Id: Development discussions about virtio-fs List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Stefan Hajnoczi Cc: Prashant Dewan , "virtio-fs@redhat.com" , gmaglione@redhat.com --feyfazohf67u3abn Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, May 26, 2022 at 08:49:34AM +0100, Stefan Hajnoczi wrote: > On Thu, May 26, 2022 at 05:34:42AM +0000, Prashant Dewan wrote: > > Hello Stefan, > >=20 > > Here is the output from virtofsd.. > >=20 > > ------------------------------------ > > - Journal begins at Thu 2022-05-26 04:23:42 UTC, ends at Thu 2022-05-26= 04:41:57 UTC. -- > > May 26 04:24:13 localhost systemd[1]: Started virtio-fs vhost-user devi= ce daemon.<>. > > May 26 04:24:13 localhost virtiofsd[505]: [2022-05-26T04:24:13Z INFO v= irtiofsd] Waiting for vhost-user socket connection... > > May 26 04:27:11 device-3e4b479e2ee9c647 virtiofsd[505]: [2022-05-26T04:= 27:11Z INFO virtiofsd] Client connected, servicing requests > > May 26 04:27:33 device-3e4b479e2ee9c647 virtiofsd[505]: [2022-05-26T04:= 27:33Z ERROR virtiofsd] Waiting for daemon failed: HandleRequest(InvalidPar= am) >=20 > virtiofsd terminated. I have CCed Sergio and German, who may have > suggestions about what to check next. Hi, Could you please tell us which QEMU version are you running, and paste here the its command line? Thanks, Sergio. --feyfazohf67u3abn Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEvtX891EthoCRQuii9GknjS8MAjUFAmKPNRMACgkQ9GknjS8M AjWQqQ//VMhoUpU/3z5bX4+aYFYQ1y7bmOJgUXZSsKsng2jD6I92HTaP0mwZn/1n SyNBf104uJUMjb+NVNlj+Td7h284NZ2BHbaAOcEMwacjpvU7TP+AG37U1eCxGHSA f4bsjrRrNrLux4IrWFOtXD53pDzf63JUMj9utdtbMHgqxxgdXqW/5MQzG2rngkUu /hZhxQk/abOtyY6Gs8ES9rvs2FTArdOHbF92HdLXcMLj904sLkBqtkBO4lad2kqx p3t0E5ONP2mBfsRsKZNLRhKSqAPX4gGKOnDxFnslDUf8N9W+U04POMFXJuXCVclV YzYUrU+j8SPeCB5N1ynYiogx/vgIa4oUxl8vwjmx8uuc6BTqlppF1hM1WCWXxOaW KAQPZeF0hTaoPfza810mA5kleqZB9lI2RPFCB+51KSwoQ8V2ZVjS8Ka93kaT8Ukl xWU7zeMIxj47djB3Rx+Fdz2Sr9O8jIWoUUVjb4Udx2N4zJjrQ/d1Lm83R3Z7ygTK SfT4QKxFpf6lP5jjMO6LvaIG8Ec5l7OLeoJrs3j72BgkoZ0TdU/J9BhR+MVhVzgf 58KJNnm8JyS7qDKVid9E1jiwXs38faTXb2zga/ZsFDfE08pMQZrBtP53n48NlBw7 rsGsQ1DZ1t2mxgVl/6ctZnWMCGHxLFG3tV1wVkTLT9Jtt7LQs5E= =+EMf -----END PGP SIGNATURE----- --feyfazohf67u3abn-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DClJNJjM0VvuE109xlTu0Nz7firXTPJlP/7lDPOiIKU=; b=uMapYbebVOGZBcLktSNj6hwr0HtZJ5kllDuBxQHR+JmrLLNv4/dhYib0zLXAjceG6TMlwhGPifH8SjzT22KA8ckY0be6/IO0j0GVe0HjrBjE3b2c0QXl8B7KCoAUQGb/FXbcsISUKIBN+0yvY9oAPBw3G9udFoJOGNJDbzCFV6u8+nDog0lKWjBBtmkmf+ouJn0u6Zsbs4o+G8u42PnHDDmqsh44g/BKgDFZCKFRGkCffx5DY21sJ9oVoxYkYLZYLQ5E2SIdJ78lttiqPcc/wlUDp58C3urPe7uT0+eQn0qhbazGOvH3qco9VTLWsCEaqQ3SssSF3YPMzPCGT9s0Vg== From: "Pra.. Dew.." Date: Thu, 26 May 2022 16:07:36 +0000 Message-ID: MIME-Version: 1.0 Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_BY5PR14MB3893C17AAED9ECD7280AE7FD86D99BY5PR14MB3893namp_" Subject: Re: [Virtio-fs] vhost_set_vring_kick failed List-Id: Development discussions about virtio-fs List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Sergio Lopez , Stefan Hajnoczi Cc: "virtio-fs@redhat.com" , "gmaglione@redhat.com" --_000_BY5PR14MB3893C17AAED9ECD7280AE7FD86D99BY5PR14MB3893namp_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Thanks for the response. The version of the qemu is 5.1.0 and here is th= e comandline.. [/usr/bin/qemu-system-test-machine test --enable-kvm -display none -mon mon= -console,mode=3Dreadline -chardev socket,host=3D,port=3D5001,server,nowait,= id=3Dmon-console -device virtio-serial-device,max_ports=3D2 -device virtcon= sole,chardev=3Dconsole,name=3Dconsole -chardev socket,host=3D,port=3D5000,s= erver,nowait,id=3Dconsole -netdev tap,id=3Dtap0,ifname=3Dtap0,vhost=3Don,sc= ript=3Dno,downscript=3Dno -device virtio-net-device,netdev=3Dtap0 -device v= host-vsock-device,guest-cid=3D4 -object rng-random,filename=3D/dev/urandom,= id=3Drng0 -device virtio-rng-device,rng=3Drng0 -chardev socket,id=3Dchar0,p= ath=3D/run/vm001-vhost-fs.sock -device vhost-user-fs-device,queue-size=3D2,= chardev=3Dchar0,tag=3Dmyfs -object memory-backend-memfd,id=3Dmem,size=3D4G,= share=3Don -kernel /var/lib/are/images/active/kernel.bin -dtb /var/lib/are/= images/active/device-tree.dtb -device virtio-blk-device,drive=3Dwritable,se= rial=3Dwritable -drive file=3D/var/lib/are/volumes/writable.bin,if=3Dnone,i= d=3Dwritable,format=3Draw -device virtio-blk-device,drive=3Drootfs,serial= =3Drootfs -drive file=3D/var/lib/are/images/active/rootfs/rootfs.bin,if=3Dn= one,id=3Drootfs,format=3Draw,readonly] ________________________________ From: Sergio Lopez Sent: Thursday, May 26, 2022 8:06 AM To: Stefan Hajnoczi Cc: Prashant Dewan; virtio-fs@redhat.com; gmaglione@redhat.com Subject: Re: [Virtio-fs] vhost_set_vring_kick failed On Thu, May 26, 2022 at 08:49:34AM +0100, Stefan Hajnoczi wrote: > On Thu, May 26, 2022 at 05:34:42AM +0000, Prashant Dewan wrote: > > Hello Stefan, > > > > Here is the output from virtofsd.. > > > > ------------------------------------ > > - Journal begins at Thu 2022-05-26 04:23:42 UTC, ends at Thu 2022-05-26= 04:41:57 UTC. -- > > May 26 04:24:13 localhost systemd[1]: Started virtio-fs vhost-user devi= ce daemon.<>. > > May 26 04:24:13 localhost virtiofsd[505]: [2022-05-26T04:24:13Z INFO v= irtiofsd] Waiting for vhost-user socket connection... > > May 26 04:27:11 device-3e4b479e2ee9c647 virtiofsd[505]: [2022-05-26T04:= 27:11Z INFO virtiofsd] Client connected, servicing requests > > May 26 04:27:33 device-3e4b479e2ee9c647 virtiofsd[505]: [2022-05-26T04:= 27:33Z ERROR virtiofsd] Waiting for daemon failed: HandleRequest(InvalidPar= am) > > virtiofsd terminated. I have CCed Sergio and German, who may have > suggestions about what to check next. Hi, Could you please tell us which QEMU version are you running, and paste here the its command line? Thanks, Sergio. --_000_BY5PR14MB3893C17AAED9ECD7280AE7FD86D99BY5PR14MB3893namp_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Thanks for the response.  The  version of the qemu is 5.1.0 = and  here is the comandline..


[/usr/bin/qemu-system-test-machine test --enable-kvm -display none -mon mon= -console,mode=3Dreadline -chardev socket,host=3D,port=3D5001,server,nowait,= id=3Dmon-console -device virtio-serial-device,max_ports=3D2 -device virtcon= sole,chardev=3Dconsole,name=3Dconsole -chardev socket,host=3D,port=3D5000,server,nowait,id=3Dconsole -netdev tap,id=3Dtap= 0,ifname=3Dtap0,vhost=3Don,script=3Dno,downscript=3Dno -device virtio-net-d= evice,netdev=3Dtap0 -device vhost-vsock-device,guest-cid=3D4 -object rng-ra= ndom,filename=3D/dev/urandom,id=3Drng0 -device virtio-rng-device,rng=3Drng0 -chardev socket,id=3Dchar0,path=3D/run/vm001-vhost-fs.sock -device vhost-u= ser-fs-device,queue-size=3D2,chardev=3Dchar0,tag=3Dmyfs -object memory-back= end-memfd,id=3Dmem,size=3D4G,share=3Don -kernel /var/lib/are/images/active/= kernel.bin -dtb /var/lib/are/images/active/device-tree.dtb -device virtio-blk-device,drive=3Dwritable,serial=3Dwritable -drive file= =3D/var/lib/are/volumes/writable.bin,if=3Dnone,id=3Dwritable,format=3Draw -= device virtio-blk-device,drive=3Drootfs,serial=3Drootfs -drive file=3D/var/= lib/are/images/active/rootfs/rootfs.bin,if=3Dnone,id=3Drootfs,format=3Draw,= readonly]


From: Sergio Lopez
Sent: Thursday, May 26, 2022 8:06 AM
To: Stefan Hajnoczi
Cc: Prashant Dewan; virtio-fs@redhat.com; gmaglione@redhat.com
Subject: Re: [Virtio-fs] vhost_set_vring_kick failed

On Thu, May 26, 2022 at 08:49:34AM +0100, Stefan H= ajnoczi wrote:
> On Thu, May 26, 2022 at 05:34:42AM +0000, Prashant Dewan wrote:
> > Hello Stefan,
> >
> > Here is the output from virtofsd..
> >
> > ------------------------------------
> > - Journal begins at Thu 2022-05-26 04:23:42 UTC, ends at Thu 2022= -05-26 04:41:57 UTC. --
> > May 26 04:24:13 localhost systemd[1]: Started virtio-fs vhost-use= r device daemon.<<Placeholder: run this daemon as non-privileged user= >>.
> > May 26 04:24:13 localhost virtiofsd[505]: [2022-05-26T04:24:13Z I= NFO  virtiofsd] Waiting for vhost-user socket connection...
> > May 26 04:27:11 device-3e4b479e2ee9c647 virtiofsd[505]: [2022-05-= 26T04:27:11Z INFO  virtiofsd] Client connected, servicing requests
> > May 26 04:27:33 device-3e4b479e2ee9c647 virtiofsd[505]: [2022-05-= 26T04:27:33Z ERROR virtiofsd] Waiting for daemon failed: HandleRequest(Inva= lidParam)
>
> virtiofsd terminated. I have CCed Sergio and German, who may have
> suggestions about what to check next.

Hi,

Could you please tell us which QEMU version are you running, and paste
here the its command line?

Thanks,
Sergio.
--_000_BY5PR14MB3893C17AAED9ECD7280AE7FD86D99BY5PR14MB3893namp_-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Mon, 30 May 2022 18:11:54 -0400 From: Vivek Goyal Message-ID: References: <20220526080646.vsfmqgxyiglhe64r@mhamilton> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20220526080646.vsfmqgxyiglhe64r@mhamilton> Subject: Re: [Virtio-fs] vhost_set_vring_kick failed List-Id: Development discussions about virtio-fs List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Sergio Lopez , Prashant Dewan Cc: Stefan Hajnoczi , "virtio-fs@redhat.com" On Thu, May 26, 2022 at 10:06:46AM +0200, Sergio Lopez wrote: > On Thu, May 26, 2022 at 08:49:34AM +0100, Stefan Hajnoczi wrote: > > On Thu, May 26, 2022 at 05:34:42AM +0000, Prashant Dewan wrote: > > > Hello Stefan, > > > > > > Here is the output from virtofsd.. > > > > > > ------------------------------------ > > > - Journal begins at Thu 2022-05-26 04:23:42 UTC, ends at Thu 2022-05-26 04:41:57 UTC. -- > > > May 26 04:24:13 localhost systemd[1]: Started virtio-fs vhost-user device daemon.<>. > > > May 26 04:24:13 localhost virtiofsd[505]: [2022-05-26T04:24:13Z INFO virtiofsd] Waiting for vhost-user socket connection... > > > May 26 04:27:11 device-3e4b479e2ee9c647 virtiofsd[505]: [2022-05-26T04:27:11Z INFO virtiofsd] Client connected, servicing requests > > > May 26 04:27:33 device-3e4b479e2ee9c647 virtiofsd[505]: [2022-05-26T04:27:33Z ERROR virtiofsd] Waiting for daemon failed: HandleRequest(InvalidParam) So looks like following gave error message. There is no error messsage from the actual thread which failed. This is just the main thread waiting. if let Err(e) = daemon.wait() { match e { HandleRequest(PartialMessage) => info!("Client disconnected, shutting down"), _ => error!("Waiting for daemon failed: {:?}", e), } } I can't find "InvalidParam" in rust virtiofsd code. Does that mean, that this error is being returned by one of the crates we are using. Prashant, you could try running rust virtiofsd with option "-d" to enable debug messages and that might show something interesting. Also, if you are able to reproduce the problem all the time, I think you can start virtiofsd under gdb and see where exactly is it failing. Thanks Vivek > > > > virtiofsd terminated. I have CCed Sergio and German, who may have > > suggestions about what to check next. > > Hi, > > Could you please tell us which QEMU version are you running, and paste > here the its command line? > > Thanks, > Sergio. > _______________________________________________ > Virtio-fs mailing list > Virtio-fs@redhat.com > https://listman.redhat.com/mailman/listinfo/virtio-fs From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Tue, 31 May 2022 12:50:50 +0200 From: Sergio Lopez Message-ID: <20220531104953.3lpjjpuxepc7wwdg@mhamilton> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="w4e3uddoi5xt5n7k" Content-Disposition: inline In-Reply-To: Subject: Re: [Virtio-fs] vhost_set_vring_kick failed List-Id: Development discussions about virtio-fs List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Pra.. Dew.." Cc: Stefan Hajnoczi , "virtio-fs@redhat.com" , "gmaglione@redhat.com" , Vivek Goyal --w4e3uddoi5xt5n7k Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="pbfxsatzvymrrfo5" Content-Disposition: inline --pbfxsatzvymrrfo5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, May 26, 2022 at 04:07:36PM +0000, Pra.. Dew.. wrote: > Thanks for the response. The version of the qemu is 5.1.0 and here is = the comandline.. >=20 >=20 > [/usr/bin/qemu-system-test-machine test --enable-kvm -display none -mon m= on-console,mode=3Dreadline -chardev socket,host=3D,port=3D5001,server,nowai= t,id=3Dmon-console -device virtio-serial-device,max_ports=3D2 -device virtc= onsole,chardev=3Dconsole,name=3Dconsole -chardev socket,host=3D,port=3D5000= ,server,nowait,id=3Dconsole -netdev tap,id=3Dtap0,ifname=3Dtap0,vhost=3Don,= script=3Dno,downscript=3Dno -device virtio-net-device,netdev=3Dtap0 -device= vhost-vsock-device,guest-cid=3D4 -object rng-random,filename=3D/dev/urando= m,id=3Drng0 -device virtio-rng-device,rng=3Drng0 -chardev socket,id=3Dchar0= ,path=3D/run/vm001-vhost-fs.sock -device vhost-user-fs-device,queue-size=3D= 2,chardev=3Dchar0,tag=3Dmyfs -object memory-backend-memfd,id=3Dmem,size=3D4= G,share=3Don -kernel /var/lib/are/images/active/kernel.bin -dtb /var/lib/ar= e/images/active/device-tree.dtb -device virtio-blk-device,drive=3Dwritable,= serial=3Dwritable -drive file=3D/var/lib/are/volumes/writable.bin,if=3Dnone= ,id=3Dwritable,format=3Draw -device virtio-blk-device,drive=3Drootfs,serial= =3Drootfs -drive file=3D/var/lib/are/images/active/rootfs/rootfs.bin,if=3Dn= one,id=3Drootfs,format=3Draw,readonly] Thanks for the details. I knew that we didn't support older versions, but didn't want to reply that without explaining why. I've just took a look, and the origin of this incompatibility is this change in "hw/virtio/vhost-user.c": qemu-5.1.0: typedef struct VhostUserMemRegMsg { uint32_t padding; VhostUserMemoryRegion region; } VhostUserMemRegMsg; qemu-upstream (since 3009edff8192991293fe9e2b50b0d90db83c4a89): typedef struct VhostUserMemRegMsg { uint64_t padding; VhostUserMemoryRegion region; } VhostUserMemRegMsg; Note how "padding" has changed from uint32_t to uint64_t. This means we only support QEMU versions starting 5.2.0. Regards, Sergio. --pbfxsatzvymrrfo5-- --w4e3uddoi5xt5n7k Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEvtX891EthoCRQuii9GknjS8MAjUFAmKV8wkACgkQ9GknjS8M AjWYrhAApFLNQAsOQaGf4zITTKo0GWWFr3l/faxGhku5d4nsDZgRr7+sasHsadaE 8R1JaloTftFfbfAzcyoF8m/xFxuLextHFkGBG2UpsxP0w4pQnb6TMclXKGSHX5SI Nz50JtppzM/2cMDObLRlMAth+a4VC2QyP6nN2KBsWkoS70CaJ/Z7c3gFNAggVv4P WCi0PLg6wHQxr3NSS2tYi5BQxapm0eufjGpj9oKSRMfTwgE7vKK3sGPmuc4kjwyz 8Z4tR3Ekg1n0oHerH7gD9YCzU/B48BjW8GAZzMlvqRKy9yiP7kFueeENw3O65xsP FCYUOXOD7exweNKQ/2AgSANiVd1jDsFvqLmoqhr8uUxEG387yolgG22+eNDoXu1T syhg5fVh+reSPttfD65J3TPj5y9eg96IT2N6tNsDP6KAEGOg7x4Z6+cFwB14R5AJ NTVIxtUJp+7iAIzxfa0vFzzIzNfg60FkzeqIf4PLPzrVDT7gcedWKrnQWybu1OUo 5ghYk2KtC1YGmeSHpXPy9fxHjRdiPo3aDE0fCggNS6JMSgG11SnTXvkH4jfC47kB DZYvSIy0mcIBzJEeO2WFV6ReqBUSJW3JpAogPL8bjT6YR1//qmXqc/piM+b/fiON 0NTat0fmZZtcnE/8yJR31LgYowaicJwUPjkr7VvrHdESStDGCQI= =2/kq -----END PGP SIGNATURE----- --w4e3uddoi5xt5n7k-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DJHT3jRd3z8fNRT2TKVAl4rH1uxtrwmVenArrBEFEHs=; b=scsvInu29hZbddofVyHxqqyDnZRUq4bl3KjEq+uPcirpIO0JuW9/aCyQEukil2rhsontfQPxvtiZMw2aEouh2yoiy9SSqML/Dz2idaMsa/+kkObKVKBkbY93KQTS+B1O0iZ8nllERM6tDkdgwczJnwgn3Y7Z41zxxA9X6D6pmNwXSctXV/MB2L7fkyx0ntHHbRXR77cNASFcVzRIhNGxTPMBLqzG0a+gaufNyWVtY0AeiIKBlulnHCWIiB5s9NToObo+aT/yOVPGlkJ9kNE3FzDbc/hSHu5czQxDh7jlZK4j5sz/Hk93QUlvc1n3kLTK9QILZffeVZZa+fPz93J3HA== From: "Pra.. Dew.." Date: Tue, 31 May 2022 18:33:18 +0000 Message-ID: MIME-Version: 1.0 Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_BY5PR14MB38930133483FCB8F1C2EC3F486DC9BY5PR14MB3893namp_" Subject: Re: [Virtio-fs] vhost_set_vring_kick failed List-Id: Development discussions about virtio-fs List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Sergio Lopez Cc: Stefan Hajnoczi , "virtio-fs@redhat.com" , "gmaglione@redhat.com" , Vivek Goyal --_000_BY5PR14MB38930133483FCB8F1C2EC3F486DC9BY5PR14MB3893namp_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Thank you Sergio. Is there a patch I could use to make it work with an old= er version? Thanks Prashant ________________________________ From: Sergio Lopez Sent: Tuesday, May 31, 2022 10:50 AM To: Pra.. Dew.. Cc: Stefan Hajnoczi; virtio-fs@redhat.com; gmaglione@redhat.com; Vivek Goya= l Subject: Re: [Virtio-fs] vhost_set_vring_kick failed On Thu, May 26, 2022 at 04:07:36PM +0000, Pra.. Dew.. wrote: > Thanks for the response. The version of the qemu is 5.1.0 and here is = the comandline.. > > > [/usr/bin/qemu-system-test-machine test --enable-kvm -display none -mon m= on-console,mode=3Dreadline -chardev socket,host=3D,port=3D5001,server,nowai= t,id=3Dmon-console -device virtio-serial-device,max_ports=3D2 -device virtc= onsole,chardev=3Dconsole,name=3Dconsole -chardev socket,host=3D,port=3D5000= ,server,nowait,id=3Dconsole -netdev tap,id=3Dtap0,ifname=3Dtap0,vhost=3Don,= script=3Dno,downscript=3Dno -device virtio-net-device,netdev=3Dtap0 -device= vhost-vsock-device,guest-cid=3D4 -object rng-random,filename=3D/dev/urando= m,id=3Drng0 -device virtio-rng-device,rng=3Drng0 -chardev socket,id=3Dchar0= ,path=3D/run/vm001-vhost-fs.sock -device vhost-user-fs-device,queue-size=3D= 2,chardev=3Dchar0,tag=3Dmyfs -object memory-backend-memfd,id=3Dmem,size=3D4= G,share=3Don -kernel /var/lib/are/images/active/kernel.bin -dtb /var/lib/ar= e/images/active/device-tree.dtb -device virtio-blk-device,drive=3Dwritable,= serial=3Dwritable -drive file=3D/var/lib/are/volumes/writable.bin,if=3Dnone= ,id=3Dwritable,format=3Draw -device virtio-blk-device,drive=3Drootfs,serial= =3Drootfs -drive file=3D/var/lib/are/images/active/rootfs/rootfs.bin,if=3Dn= one,id=3Drootfs,format=3Draw,readonly] Thanks for the details. I knew that we didn't support older versions, but didn't want to reply that without explaining why. I've just took a look, and the origin of this incompatibility is this change in "hw/virtio/vhost-user.c": qemu-5.1.0: typedef struct VhostUserMemRegMsg { uint32_t padding; VhostUserMemoryRegion region; } VhostUserMemRegMsg; qemu-upstream (since 3009edff8192991293fe9e2b50b0d90db83c4a89): typedef struct VhostUserMemRegMsg { uint64_t padding; VhostUserMemoryRegion region; } VhostUserMemRegMsg; Note how "padding" has changed from uint32_t to uint64_t. This means we only support QEMU versions starting 5.2.0. Regards, Sergio. --_000_BY5PR14MB38930133483FCB8F1C2EC3F486DC9BY5PR14MB3893namp_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Thank you Sergio.  Is there a patch I could use to make it work w= ith an older version?

Thanks
Prashant



From: Sergio Lopez
Sent: Tuesday, May 31, 2022 10:50 AM
To: Pra.. Dew..
Cc: Stefan Hajnoczi; virtio-fs@redhat.com; gmaglione@redhat.com; Viv= ek Goyal
Subject: Re: [Virtio-fs] vhost_set_vring_kick failed

On Thu, May 26, 2022 at 04:07:36PM +0000, Pra.. De= w.. wrote:
> Thanks for the response.  The  version of the qemu is 5.1.0 = and  here is the comandline..
>
>
> [/usr/bin/qemu-system-test-machine test --enable-kvm -display none -mo= n mon-console,mode=3Dreadline -chardev socket,host=3D,port=3D5001,server,no= wait,id=3Dmon-console -device virtio-serial-device,max_ports=3D2 -device vi= rtconsole,chardev=3Dconsole,name=3Dconsole -chardev socket,host=3D,port=3D5000,server,nowait,id=3Dconsole -netdev tap,id=3Dtap= 0,ifname=3Dtap0,vhost=3Don,script=3Dno,downscript=3Dno -device virtio-net-d= evice,netdev=3Dtap0 -device vhost-vsock-device,guest-cid=3D4 -object rng-ra= ndom,filename=3D/dev/urandom,id=3Drng0 -device virtio-rng-device,rng=3Drng0 -chardev socket,id=3Dchar0,path=3D/run/vm001-vhost-fs.sock -device vhost-u= ser-fs-device,queue-size=3D2,chardev=3Dchar0,tag=3Dmyfs -object memory-back= end-memfd,id=3Dmem,size=3D4G,share=3Don -kernel /var/lib/are/images/active/= kernel.bin -dtb /var/lib/are/images/active/device-tree.dtb -device virtio-blk-device,drive=3Dwritable,serial=3Dwritable -drive file= =3D/var/lib/are/volumes/writable.bin,if=3Dnone,id=3Dwritable,format=3Draw -= device virtio-blk-device,drive=3Drootfs,serial=3Drootfs -drive file=3D/var/= lib/are/images/active/rootfs/rootfs.bin,if=3Dnone,id=3Drootfs,format=3Draw,= readonly]

Thanks for the details. I knew that we didn't support older versions,
but didn't want to reply that without explaining why.

I've just took a look, and the origin of this incompatibility is this
change in "hw/virtio/vhost-user.c":

qemu-5.1.0:

typedef struct VhostUserMemRegMsg {
    uint32_t padding;
    VhostUserMemoryRegion region;
} VhostUserMemRegMsg;

qemu-upstream (since 3009edff8192991293fe9e2b50b0d90db83c4a89):

typedef struct VhostUserMemRegMsg {
    uint64_t padding;
    VhostUserMemoryRegion region;
} VhostUserMemRegMsg;

Note how "padding" has changed from uint32_t to uint64_t.

This means we only support QEMU versions starting 5.2.0.

Regards,
Sergio.
--_000_BY5PR14MB38930133483FCB8F1C2EC3F486DC9BY5PR14MB3893namp_-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TIi2EGnDNZNox4NtnyaH4vBbkYzcm0lmHCcO8hNlSPU=; b=oYe8xRsA0odN1frwqzzJFffY85mMEjx519wRLSwFUtjt2/WQ4c2pxd/6WXiDdFYXj5csrpx6qR/bsEhjEoC7ZGzwSSYk4+pfncztA4F3oBayzqm4IFj9pE916+XUoE+oFhDmAL4b8iurZt8JGS+MhGB5wDKJOh4P+6HY77+/lakUNW4EBEgNg86XuXlQHF9ULaiP79W/JSn+CA6EJeoV63MfrdX9HpAO9rhm4/2abfH9PNgplOIFTc1VgYDnf1Bah8QJM2eoV/XO/Fam9UpPTgW+np38Hahp+cM7EcSDIZt1mzYLx9djBUrpikL4zuLgk+1/dN3ZRgEfxVsPWFY4PQ== From: "Pra.. Dew.." Date: Tue, 31 May 2022 22:25:12 +0000 Message-ID: References: <20220526080646.vsfmqgxyiglhe64r@mhamilton> In-Reply-To: MIME-Version: 1.0 Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_BY5PR14MB3893602BAEFA362A5C62485386DC9BY5PR14MB3893namp_" Subject: Re: [Virtio-fs] vhost_set_vring_kick failed List-Id: Development discussions about virtio-fs List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Vivek Goyal , Sergio Lopez Cc: Stefan Hajnoczi , "virtio-fs@redhat.com" --_000_BY5PR14MB3893602BAEFA362A5C62485386DC9BY5PR14MB3893namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Thank you so much Vivek. I enabled the trace in the code itself and ran it.= The only message I got was InvalidParam message below. I did not get any o= ther message. We are going to try changing the structure [VhostUserMemRegM= sg] padding to see if that fixes the problem. Is there anything else we sh= ould try? ________________________________ From: Vivek Goyal Sent: Monday, May 30, 2022 10:11 PM To: Sergio Lopez ; Prashant Dewan Cc: Stefan Hajnoczi ; virtio-fs@redhat.com Subject: Re: [Virtio-fs] vhost_set_vring_kick failed On Thu, May 26, 2022 at 10:06:46AM +0200, Sergio Lopez wrote: > On Thu, May 26, 2022 at 08:49:34AM +0100, Stefan Hajnoczi wrote: > > On Thu, May 26, 2022 at 05:34:42AM +0000, Prashant Dewan wrote: > > > Hello Stefan, > > > > > > Here is the output from virtofsd.. > > > > > > ------------------------------------ > > > - Journal begins at Thu 2022-05-26 04:23:42 UTC, ends at Thu 2022-05-= 26 04:41:57 UTC. -- > > > May 26 04:24:13 localhost systemd[1]: Started virtio-fs vhost-user de= vice daemon.<>. > > > May 26 04:24:13 localhost virtiofsd[505]: [2022-05-26T04:24:13Z INFO = virtiofsd] Waiting for vhost-user socket connection... > > > May 26 04:27:11 device-3e4b479e2ee9c647 virtiofsd[505]: [2022-05-26T0= 4:27:11Z INFO virtiofsd] Client connected, servicing requests > > > May 26 04:27:33 device-3e4b479e2ee9c647 virtiofsd[505]: [2022-05-26T0= 4:27:33Z ERROR virtiofsd] Waiting for daemon failed: HandleRequest(InvalidP= aram) So looks like following gave error message. There is no error messsage from the actual thread which failed. This is just the main thread waiting. if let Err(e) =3D daemon.wait() { match e { HandleRequest(PartialMessage) =3D> info!("Client disconnected, = shutting down"), _ =3D> error!("Waiting for daemon failed: {:?}", e), } } I can't find "InvalidParam" in rust virtiofsd code. Does that mean, that this error is being returned by one of the crates we are using. Prashant, you could try running rust virtiofsd with option "-d" to enable debug messages and that might show something interesting. Also, if you are able to reproduce the problem all the time, I think you can start virtiofsd under gdb and see where exactly is it failing. Thanks Vivek > > > > virtiofsd terminated. I have CCed Sergio and German, who may have > > suggestions about what to check next. > > Hi, > > Could you please tell us which QEMU version are you running, and paste > here the its command line? > > Thanks, > Sergio. > _______________________________________________ > Virtio-fs mailing list > Virtio-fs@redhat.com > https://nam12.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flistm= an.redhat.com%2Fmailman%2Flistinfo%2Fvirtio-fs&data=3D05%7C01%7C%7Cc4d5= 792347234799874008da42896937%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C6= 37895455186641964%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luM= zIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3Dox5y7KH0Djioo= IeJ367DZiexKQutIouWSCi5OLld%2BNY%3D&reserved=3D0 --_000_BY5PR14MB3893602BAEFA362A5C62485386DC9BY5PR14MB3893namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Thank you so much Vivek. I enabled the trace in the code itself and ra= n it. The only message I got was InvalidParam message below. I did not get = any other message.  We are going to try changing the structure [VhostU= serMemRegMsg]  padding to see if that fixes the problem. Is there anything else we should= try? 
From: Vivek Goyal <vgoya= l@redhat.com>
Sent: Monday, May 30, 2022 10:11 PM
To: Sergio Lopez <slp@redhat.com>; Prashant Dewan <linux_le= arner@outlook.com>
Cc: Stefan Hajnoczi <stefanha@redhat.com>; virtio-fs@redhat.co= m <virtio-fs@redhat.com>
Subject: Re: [Virtio-fs] vhost_set_vring_kick failed
 
On Thu, May 26, 2022 at 10:06:46AM +0200, Sergio L= opez wrote:
> On Thu, May 26, 2022 at 08:49:34AM +0100, Stefan Hajnoczi wrote:
> > On Thu, May 26, 2022 at 05:34:42AM +0000, Prashant Dewan wrote: > > > Hello Stefan,
> > >
> > > Here is the output from virtofsd..
> > >
> > > ------------------------------------
> > > - Journal begins at Thu 2022-05-26 04:23:42 UTC, ends at Thu= 2022-05-26 04:41:57 UTC. --
> > > May 26 04:24:13 localhost systemd[1]: Started virtio-fs vhos= t-user device daemon.<<Placeholder: run this daemon as non-privileged= user>>.
> > > May 26 04:24:13 localhost virtiofsd[505]: [2022-05-26T04:24:= 13Z INFO  virtiofsd] Waiting for vhost-user socket connection...
> > > May 26 04:27:11 device-3e4b479e2ee9c647 virtiofsd[505]: [202= 2-05-26T04:27:11Z INFO  virtiofsd] Client connected, servicing request= s
> > > May 26 04:27:33 device-3e4b479e2ee9c647 virtiofsd[505]: [202= 2-05-26T04:27:33Z ERROR virtiofsd] Waiting for daemon failed: HandleRequest= (InvalidParam)

So looks like following gave error message. There is no error messsage
from the actual thread which failed. This is just the main thread
waiting.

    if let Err(e) =3D daemon.wait() {
        match e {
            HandleRe= quest(PartialMessage) =3D> info!("Client disconnected, shutting dow= n"),
            _ =3D>= ; error!("Waiting for daemon failed: {:?}", e),
        }
    }

I can't find "InvalidParam" in rust virtiofsd code. Does that mea= n,
that this error is being returned by one of the crates we are using.

Prashant, you could try running rust virtiofsd with option "-d" t= o enable
debug messages and that might show something interesting.

Also, if you are able to reproduce the problem all the time, I think
you can start virtiofsd under gdb and see where exactly is it failing.

Thanks
Vivek

> >
> > virtiofsd terminated. I have CCed Sergio and German, who may have=
> > suggestions about what to check next.
>
> Hi,
>
> Could you please tell us which QEMU version are you running, and paste=
> here the its command line?
>
> Thanks,
> Sergio.



> _______________________________________________
> Virtio-fs mailing list
> Virtio-fs@redhat.com
> https://nam12.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Flistman= .redhat.com%2Fmailman%2Flistinfo%2Fvirtio-fs&amp;data=3D05%7C01%7C%7Cc4= d5792347234799874008da42896937%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7= C637895455186641964%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2l= uMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=3Dox5y7KH= 0DjiooIeJ367DZiexKQutIouWSCi5OLld%2BNY%3D&amp;reserved=3D0

--_000_BY5PR14MB3893602BAEFA362A5C62485386DC9BY5PR14MB3893namp_-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Tue, 31 May 2022 18:50:14 -0400 From: Vivek Goyal Message-ID: References: <20220526080646.vsfmqgxyiglhe64r@mhamilton> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Subject: Re: [Virtio-fs] vhost_set_vring_kick failed List-Id: Development discussions about virtio-fs List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Pra.. Dew.." Cc: Sergio Lopez , Stefan Hajnoczi , "virtio-fs@redhat.com" On Tue, May 31, 2022 at 10:25:12PM +0000, Pra.. Dew.. wrote: > Thank you so much Vivek. I enabled the trace in the code itself and ran it. The only message I got was InvalidParam message below. I did not get any other message. We are going to try changing the structure [VhostUserMemRegMsg] padding to see if that fixes the problem. Is there anything else we should try? Hi Prashant, I was trying to look at some of the code today and looks like error message might be coming from vhost crate (src/vhost_user/slave_req_handler.rs). I don't understand the whole logic very well. So I will defer it to Sergio if something can be done to support older qemu. Thanks Vivek > ________________________________ > From: Vivek Goyal > Sent: Monday, May 30, 2022 10:11 PM > To: Sergio Lopez ; Prashant Dewan > Cc: Stefan Hajnoczi ; virtio-fs@redhat.com > Subject: Re: [Virtio-fs] vhost_set_vring_kick failed > > On Thu, May 26, 2022 at 10:06:46AM +0200, Sergio Lopez wrote: > > On Thu, May 26, 2022 at 08:49:34AM +0100, Stefan Hajnoczi wrote: > > > On Thu, May 26, 2022 at 05:34:42AM +0000, Prashant Dewan wrote: > > > > Hello Stefan, > > > > > > > > Here is the output from virtofsd.. > > > > > > > > ------------------------------------ > > > > - Journal begins at Thu 2022-05-26 04:23:42 UTC, ends at Thu 2022-05-26 04:41:57 UTC. -- > > > > May 26 04:24:13 localhost systemd[1]: Started virtio-fs vhost-user device daemon.<>. > > > > May 26 04:24:13 localhost virtiofsd[505]: [2022-05-26T04:24:13Z INFO virtiofsd] Waiting for vhost-user socket connection... > > > > May 26 04:27:11 device-3e4b479e2ee9c647 virtiofsd[505]: [2022-05-26T04:27:11Z INFO virtiofsd] Client connected, servicing requests > > > > May 26 04:27:33 device-3e4b479e2ee9c647 virtiofsd[505]: [2022-05-26T04:27:33Z ERROR virtiofsd] Waiting for daemon failed: HandleRequest(InvalidParam) > > So looks like following gave error message. There is no error messsage > from the actual thread which failed. This is just the main thread > waiting. > > if let Err(e) = daemon.wait() { > match e { > HandleRequest(PartialMessage) => info!("Client disconnected, shutting down"), > _ => error!("Waiting for daemon failed: {:?}", e), > } > } > > I can't find "InvalidParam" in rust virtiofsd code. Does that mean, > that this error is being returned by one of the crates we are using. > > Prashant, you could try running rust virtiofsd with option "-d" to enable > debug messages and that might show something interesting. > > Also, if you are able to reproduce the problem all the time, I think > you can start virtiofsd under gdb and see where exactly is it failing. > > Thanks > Vivek > > > > > > > virtiofsd terminated. I have CCed Sergio and German, who may have > > > suggestions about what to check next. > > > > Hi, > > > > Could you please tell us which QEMU version are you running, and paste > > here the its command line? > > > > Thanks, > > Sergio. > > > > > _______________________________________________ > > Virtio-fs mailing list > > Virtio-fs@redhat.com > > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flistman.redhat.com%2Fmailman%2Flistinfo%2Fvirtio-fs&data=05%7C01%7C%7Cc4d5792347234799874008da42896937%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637895455186641964%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ox5y7KH0DjiooIeJ367DZiexKQutIouWSCi5OLld%2BNY%3D&reserved=0 > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Date: Wed, 1 Jun 2022 08:01:19 +0200 From: Sergio Lopez Message-ID: <20220601060045.7wjgfqkmcruraif7@mhamilton> References: <20220526080646.vsfmqgxyiglhe64r@mhamilton> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="ubi6gtl647wom7hw" Content-Disposition: inline In-Reply-To: Subject: Re: [Virtio-fs] vhost_set_vring_kick failed List-Id: Development discussions about virtio-fs List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Pra.. Dew.." Cc: Vivek Goyal , Stefan Hajnoczi , "virtio-fs@redhat.com" --ubi6gtl647wom7hw Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="w73vybx4nlbw3jse" Content-Disposition: inline --w73vybx4nlbw3jse Content-Type: multipart/mixed; boundary="olxi4fcfwfrd7o5c" Content-Disposition: inline --olxi4fcfwfrd7o5c Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Tue, May 31, 2022 at 10:25:12PM +0000, Pra.. Dew.. wrote: > Thank you so much Vivek. I enabled the trace in the code itself and > ran it. The only message I got was InvalidParam message below. I did > not get any other message. We are going to try changing the > structure [VhostUserMemRegMsg] padding to see if that fixes the > problem. Is there anything else we should try? Applying 3009edff81 to qemu-5.1.0 should do the trick. I'm attaching the patch to this email, so you can simply apply it with something like: patch -p1 < 0001-vhost-user-fix-VHOST_USER_ADD-REM_MEM_REG-truncation.patch Sergio. --olxi4fcfwfrd7o5c Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="0001-vhost-user-fix-VHOST_USER_ADD-REM_MEM_REG-truncation.patch" Content-Transfer-Encoding: quoted-printable =46rom 3009edff8192991293fe9e2b50b0d90db83c4a89 Mon Sep 17 00:00:00 2001 =46rom: Stefan Hajnoczi Date: Mon, 9 Nov 2020 17:43:55 +0000 Subject: [PATCH] vhost-user: fix VHOST_USER_ADD/REM_MEM_REG truncation QEMU currently truncates the mmap_offset field when sending VHOST_USER_ADD_MEM_REG and VHOST_USER_REM_MEM_REG messages. The struct layout looks like this: typedef struct VhostUserMemoryRegion { uint64_t guest_phys_addr; uint64_t memory_size; uint64_t userspace_addr; uint64_t mmap_offset; } VhostUserMemoryRegion; typedef struct VhostUserMemRegMsg { uint32_t padding; /* WARNING: there is a 32-bit hole here! */ VhostUserMemoryRegion region; } VhostUserMemRegMsg; The payload size is calculated as follows when sending the message in hw/virtio/vhost-user.c: msg->hdr.size =3D sizeof(msg->payload.mem_reg.padding) + sizeof(VhostUserMemoryRegion); This calculation produces an incorrect result of only 36 bytes. sizeof(VhostUserMemRegMsg) is actually 40 bytes. The consequence of this is that the final field, mmap_offset, is truncated. This breaks x86_64 TCG guests on s390 hosts. Other guest/host combinations may get lucky if either of the following holds: 1. The guest memory layout does not need mmap_offset !=3D 0. 2. The host is little-endian and mmap_offset <=3D 0xffffffff so the truncation has no effect. Fix this by extending the existing 32-bit padding field to 64-bit. Now the padding reflects the actual compiler padding. This can be verified using pahole(1). Also document the layout properly in the vhost-user specification. The vhost-user spec did not document the exact layout. It would be impossible to implement the spec without looking at the QEMU source code. Existing vhost-user frontends and device backends continue to work after this fix has been applied. The only change in the wire protocol is that QEMU now sets hdr.size to 40 instead of 36. If a vhost-user implementation has a hardcoded size check for 36 bytes, then it will fail with new QEMUs. Both QEMU and DPDK/SPDK don't check the exact payload size, so they continue to work. Fixes: f1aeb14b0809e313c74244d838645ed25e85ea63 ("Transmit vhost-user memor= y regions individually") Cc: Raphael Norwitz Cc: Cornelia Huck Cc: Michael S. Tsirkin Cc: Christian Borntraeger Signed-off-by: Stefan Hajnoczi Message-Id: <20201109174355.1069147-1-stefanha@redhat.com> Reviewed-by: Michael S. Tsirkin Signed-off-by: Michael S. Tsirkin Fixes: f1aeb14b0809 ("Transmit vhost-user memory regions individually") Reviewed-by: Cornelia Huck Reviewed-by: Raphael Norwitz --- contrib/libvhost-user/libvhost-user.h | 2 +- docs/interop/vhost-user.rst | 21 +++++++++++++++++++-- hw/virtio/vhost-user.c | 5 ++--- 3 files changed, 22 insertions(+), 6 deletions(-) diff --git a/contrib/libvhost-user/libvhost-user.h b/contrib/libvhost-user/= libvhost-user.h index a1539dbb69..7d47f1364a 100644 --- a/contrib/libvhost-user/libvhost-user.h +++ b/contrib/libvhost-user/libvhost-user.h @@ -136,7 +136,7 @@ typedef struct VhostUserMemory { } VhostUserMemory; =20 typedef struct VhostUserMemRegMsg { - uint32_t padding; + uint64_t padding; VhostUserMemoryRegion region; } VhostUserMemRegMsg; =20 diff --git a/docs/interop/vhost-user.rst b/docs/interop/vhost-user.rst index 988f154144..6d4025ba6a 100644 --- a/docs/interop/vhost-user.rst +++ b/docs/interop/vhost-user.rst @@ -131,6 +131,23 @@ A region is: =20 :mmap offset: 64-bit offset where region starts in the mapped memory =20 +Single memory region description +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + ++---------+---------------+------+--------------+-------------+ +| padding | guest address | size | user address | mmap offset | ++---------+---------------+------+--------------+-------------+ + +:padding: 64-bit + +:guest address: a 64-bit guest address of the region + +:size: a 64-bit size + +:user address: a 64-bit user address + +:mmap offset: 64-bit offset where region starts in the mapped memory + Log description ^^^^^^^^^^^^^^^ =20 @@ -1281,7 +1298,7 @@ Master message types ``VHOST_USER_ADD_MEM_REG`` :id: 37 :equivalent ioctl: N/A - :slave payload: memory region + :slave payload: single memory region description =20 When the ``VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS`` protocol feature has been successfully negotiated, this message is submitted @@ -1296,7 +1313,7 @@ Master message types ``VHOST_USER_REM_MEM_REG`` :id: 38 :equivalent ioctl: N/A - :slave payload: memory region + :slave payload: single memory region description =20 When the ``VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS`` protocol feature has been successfully negotiated, this message is submitted diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index 9c5b4f7fbc..2fdd5daf74 100644 --- a/hw/virtio/vhost-user.c +++ b/hw/virtio/vhost-user.c @@ -149,7 +149,7 @@ typedef struct VhostUserMemory { } VhostUserMemory; =20 typedef struct VhostUserMemRegMsg { - uint32_t padding; + uint64_t padding; VhostUserMemoryRegion region; } VhostUserMemRegMsg; =20 @@ -800,8 +800,7 @@ static int vhost_user_add_remove_regions(struct vhost_d= ev *dev, uint64_t shadow_pcb[VHOST_USER_MAX_RAM_SLOTS] =3D {}; int nr_add_reg, nr_rem_reg; =20 - msg->hdr.size =3D sizeof(msg->payload.mem_reg.padding) + - sizeof(VhostUserMemoryRegion); + msg->hdr.size =3D sizeof(msg->payload.mem_reg); =20 /* Find the regions which need to be removed or added. */ scrub_shadow_regions(dev, add_reg, &nr_add_reg, rem_reg, &nr_rem_reg, --=20 2.36.1 --olxi4fcfwfrd7o5c-- --w73vybx4nlbw3jse-- --ubi6gtl647wom7hw Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEvtX891EthoCRQuii9GknjS8MAjUFAmKXAK8ACgkQ9GknjS8M AjV9uw/+IvfPp5u8AOoIG72lz8IhE3ad5ZIgXV1u/xsfXRJac7yQR/64KUmDpLNz EoSI9M+9Q0CDpNKDyFo2PF6+vqmNcbunXqmE+5TwiTPvW59bYYg/jlzoPkdxGDmh jNt7mVKDYnywxh3riEiPqwAx8EJEROwHLjhY8CJMwPbodbYSwMczVHUJ+2fUIEEP CHo3sZZOyyCRw2nh4N+KMg2iynrU9nllb8d/NFScu8/CPXgmCfL/4xJAtU6jk64C zKXedcyU9/iIWiXfTxL1cxSOqQs0dgwxMYdwKCFFtmBzBCD9JW5XpEcZXkikjgmj 4+O9pjfEN+eJCFyyb+y/23yqcxCz5r4JsWwysNlhRp6bujESxMYN3OilZM8aDDh8 RF9JflmF6SCS2Ec1c3TzEze8zi0YTgr4rk302TSQE150o3ka88mfan+Uv50U7T5A zxC5ZC+38Dzq1pTxgWznLHt6D6y8LVit1RRc4AoIugTNdObIlgo/TT7tQyJ3EXSI aDzPOpVmkjqSVy0Cx7aZKKJ78OzWczVecv9VkdImAB3BfmbkeQ1S7KDh6YwKaDNy Xr79EhGeVv+QGiqgnez9dFtx3I6sXcdGLcn8lCR4RldrnOOEftxMjmQahk2bP4YZ EuNqPkTTIDG26HdeKTvCCB6KmP4NnqpzpKd/Q1stxtd7ZaXsuhQ= =ZqQX -----END PGP SIGNATURE----- --ubi6gtl647wom7hw-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outlook.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DzIlSB3oFCPG/c5Gwwy9WpJfHe1YEQ92r9c7kohvyaI=; b=Q26uXGqHQ5OXoK2AqJe1qdYuBAB2eQUsI+OLCfmIp1ulk2bgGq4qiJyivghg5T03o+X7+XD2hZ+wxz502cTjKie6NffITOt2lzrEUs38qj3Zlxx0tJhXnGbOY9DSsEEve3oTJVGAaO+Nf1NXWhOJ6+bFZGNxYfURNWkV+CRCAMo5rr3N2kLhLw+NsUHaaTvLrhaSHYZQCHjDoEMuiF1W+zIQNEnJmEiNtbVIaiDDG7W+TZ457haolJ8XJXBZutIzFVRM/a9rgssTS88okCq6Is90UDOKwfLOFgCDDcIUxUPel6t+ErQyJnEA7jsjFTCX/UKgxw19WDREmtYAXuv1Dg== From: "Pra.. Dew.." Date: Wed, 1 Jun 2022 19:12:20 +0000 Message-ID: MIME-Version: 1.0 Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_BY5PR14MB38931D7111BD36AFD0FE79AA86DF9BY5PR14MB3893namp_" Subject: Re: [Virtio-fs] vhost_set_vring_kick failed List-Id: Development discussions about virtio-fs List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Sergio Lopez Cc: Vivek Goyal , Stefan Hajnoczi , "virtio-fs@redhat.com" --_000_BY5PR14MB38931D7111BD36AFD0FE79AA86DF9BY5PR14MB3893namp_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Thank you Sergio. When I applied the patch, I got stuck here. The header s= ize is correct (40 bytes). vhost-user.c 390 ret =3D qemu_chr_fe_write_all(chr, (const uint8_t *) msg, size); 391 if (ret !=3D size) { 392 error_report("Failed to write msg." 393 " Wrote %d instead of %d.", ret, size); 394 return -1; 395 } ---- Jun 01 19:01:52 device-d1fe4f98f7091676 systemd[1]: Started Application Run= time Environment VMM. Jun 01 19:01:52 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: info: Passing through UIO device usb-phy@29910000: Jun 01 19:01:52 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: info: Region: base: 0000000029910000 size: 0000000000010000 Jun 01 19:01:52 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: info: IRQ: 104 Jun 01 19:01:52 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: info: Passing through UIO device gpio@2d010000: Jun 01 19:01:52 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: info: Region: base: 000000002d010000 size: 0000000000002000 Jun 01 19:01:52 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: info: IRQ: 131 Jun 01 19:01:52 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: info: Passing through UIO device ram: Jun 01 19:01:52 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: info: Region: base: 00000000b0000000 size: 0000000050000000 Jun 01 19:01:55 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: info: Passing through UIO device uio@298c0000: Jun 01 19:01:55 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: info: Region: base: 00000000298c0000 size: 0000000000010000 Jun 01 19:01:55 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: info: Passing through UIO device uio@29800000: Jun 01 19:01:55 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: info: Region: base: 0000000029800000 size: 0000000000010000 Jun 01 19:01:55 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: info: Passing through UIO device usb@29900000: Jun 01 19:01:55 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: info: Region: base: 0000000029900000 size: 0000000000001000 Jun 01 19:01:55 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: info: IRQ: 103 Jun 01 19:01:55 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: info: Successfully realized 6 UIO devices Jun 01 19:01:57 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: warning: Could not locate arm,scmi-shmem node Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: AS_DEBUG: hdr_si= ze=3D0x28 Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: Failed to write msg. Wrote -1 instead of 52. Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: vhost_set_vring_addr failed: Invalid argument (22) Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: Failed to set msg fds. Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: vhost VQ 0 ring restore failed: -1: Invalid argument (22) Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: Error starting vhost: 22 Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: Failed to set msg fds. Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: vhost_set_vring_call failed: Resource temporarily unavailable (11) Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: Failed to set msg fds. Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: vhost_set_vring_call failed: Resource temporarily unavailable (11) Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: qemu-system-devi= ce: Failed to read from slave. device-d1fe4f98f7091676:~$ ________________________________ From: Sergio Lopez Sent: Wednesday, June 1, 2022 6:01 AM To: Pra.. Dew.. Cc: Vivek Goyal; Stefan Hajnoczi; virtio-fs@redhat.com Subject: Re: [Virtio-fs] vhost_set_vring_kick failed On Tue, May 31, 2022 at 10:25:12PM +0000, Pra.. Dew.. wrote: > Thank you so much Vivek. I enabled the trace in the code itself and > ran it. The only message I got was InvalidParam message below. I did > not get any other message. We are going to try changing the > structure [VhostUserMemRegMsg] padding to see if that fixes the > problem. Is there anything else we should try? Applying 3009edff81 to qemu-5.1.0 should do the trick. I'm attaching the patch to this email, so you can simply apply it with something like: patch -p1 < 0001-vhost-user-fix-VHOST_USER_ADD-REM_MEM_REG-truncation.patch Sergio. --_000_BY5PR14MB38931D7111BD36AFD0FE79AA86DF9BY5PR14MB3893namp_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Thank you Sergio. When I applied the patch, I got stuck  here. Th= e header size is correct (40 bytes).

vhost-user.c
 390     ret =3D qemu_chr_fe_write_all(chr, (const uint= 8_t *) msg, size);
 391     if (ret !=3D size) {
 392         error_report("Failed to wri= te msg."
 393                 &nbs= p;    " Wrote %d instead of %d.", ret, size);
 394         return -1;
 395     }


----
Jun 01 19:01:52 device-d1fe4f98f7091676 systemd[1]: Started Application Run= time Environment VMM.
Jun 01 19:01:52 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: info: Passing through UIO device usb-phy@29910000:
Jun 01 19:01:52 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: info:         Region: base: 0000000029910000 s= ize: 0000000000010000
Jun 01 19:01:52 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: info:         IRQ: 104
Jun 01 19:01:52 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: info: Passing through UIO device gpio@2d010000:
Jun 01 19:01:52 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: info:         Region: base: 000000002d010000 s= ize: 0000000000002000
Jun 01 19:01:52 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: info:         IRQ: 131
Jun 01 19:01:52 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: info: Passing through UIO device ram:
Jun 01 19:01:52 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: info:         Region: base: 00000000b0000000 s= ize: 0000000050000000
Jun 01 19:01:55 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: info: Passing through UIO device uio@298c0000:
Jun 01 19:01:55 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: info:         Region: base: 00000000298c0000 s= ize: 0000000000010000
Jun 01 19:01:55 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: info: Passing through UIO device uio@29800000:
Jun 01 19:01:55 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: info:         Region: base: 0000000029800000 s= ize: 0000000000010000
Jun 01 19:01:55 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: info: Passing through UIO device usb@29900000:
Jun 01 19:01:55 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: info:         Region: base: 0000000029900000 s= ize: 0000000000001000
Jun 01 19:01:55 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: info:         IRQ: 103
Jun 01 19:01:55 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: info: Successfully realized 6 UIO devices
Jun 01 19:01:57 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: warning: Could not locate arm,scmi-shmem node
Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: AS_DEBUG: h= dr_size=3D0x28
Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: qemu-sys= tem-device: Failed to write msg. Wrote -1 instead of 52.
Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: vhost_set_vring_addr failed: Invalid argument (22)
Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: Failed to set msg fds.
Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: vhost VQ 0 ring restore failed: -1: Invalid argument (22)
Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: Error starting vhost: 22
Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: Failed to set msg fds.
Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: vhost_set_vring_call failed: Resource temporarily unavailable (11)=
Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: Failed to set msg fds.
Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: vhost_set_vring_call failed: Resource temporarily unavailable (11)=
Jun 01 19:02:23 device-d1fe4f98f7091676 are-launcher[651]: qemu-system= -device: Failed to read from slave.
device-d1fe4f98f7091676:~$

From: Sergio Lopez
Sent: Wednesday, June 1, 2022 6:01 AM
To: Pra.. Dew..
Cc: Vivek Goyal; Stefan Hajnoczi; virtio-fs@redhat.com
Subject: Re: [Virtio-fs] vhost_set_vring_kick failed

On Tue, May 31, 2022 at 10:25:12PM +0000, Pra.. De= w.. wrote:
> Thank you so much Vivek. I enabled the trace in the code itself and > ran it. The only message I got was InvalidParam message below. I did > not get any other message.  We are going to try changing the
> structure [VhostUserMemRegMsg]  padding to see if that fixes the<= br> > problem. Is there anything else we should try?

Applying 3009edff81 to qemu-5.1.0 should do the trick. I'm attaching
the patch to this email, so you can simply apply it with something
like:

patch -p1 < 0001-vhost-user-fix-VHOST_USER_ADD-REM_MEM_REG-truncation.pa= tch

Sergio.
--_000_BY5PR14MB38931D7111BD36AFD0FE79AA86DF9BY5PR14MB3893namp_--