RE: [PATCH v2 1/4] hw/virtio: incorporate backend features in features

From: Yajun Wu <yajunw@nvidia.com>
To: "Alex Bennée" <alex.bennee@linaro.org>
Cc: "qemu-devel@nongnu.org" <qemu-devel@nongnu.org>,
	"mst@redhat.com" <mst@redhat.com>,
	Parav Pandit <parav@nvidia.com>
Subject: RE: [PATCH  v2 1/4] hw/virtio: incorporate backend features in features
Date: Thu, 10 Nov 2022 10:35:49 +0000	[thread overview]
Message-ID: <DM4PR12MB5168D79B6312F25544556B13B6019@DM4PR12MB5168.namprd12.prod.outlook.com> (raw)
In-Reply-To: <877d0n58t6.fsf@linaro.org>

Hi Alex,

Sorry for the late response, I missed your mail.
You can test together with dpdk and have reproduce.

Steps:
1. DPDK 
git clone https://github.com/DPDK/dpdk.git
git checkout v22.07
meson build -Dexamples=vhost_blk
ninja -C build
cd /tmp/
sudo dpdk/build/examples/dpdk-vhost_blk # it's a daemon 

2. Add blk device to qemu, then bootup VM.
  <qemu:commandline>
    <qemu:arg value='-chardev'/>
    <qemu:arg value='socket,id=char0,path=/tmp/vhost.socket'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vhost-user-blk-pci,chardev=char0,num-queues=1'/>
  </qemu:commandline>

Without this commit, you can get device ready log from dpdk-vhost_blk:

VHOST_CONFIG: (/tmp/vhost.socket) negotiated Vhost-user protocol features: 0x11ebf
VHOST_CONFIG: (/tmp/vhost.socket) negotiated Virtio features: 0x100000000
VHOST_CONFIG: (/tmp/vhost.socket) virtio is now ready for processing.
New Device /tmp/vhost.socket, Device ID 0
Ctrlr Worker Thread start

With this commit, device won't be ready and VM will hang. 
You can see VHOST_USER_F_PROTOCOL_FEATURES bit is added.

VHOST_CONFIG: (/tmp/vhost.socket) negotiated Virtio features: 0x140000000

Dpdk code related:
./lib/vhost/vhost_user.c

2044     /*                                                                          
2045      * When VHOST_USER_F_PROTOCOL_FEATURES is not negotiated,                   
2046      * the ring starts already enabled. Otherwise, it is enabled via            
2047      * the SET_VRING_ENABLE message.                                            
2048      */                                                                         
2049     if (!(dev->features & (1ULL << VHOST_USER_F_PROTOCOL_FEATURES))) {          
2050         vq->enabled = true;                                                     
2051     }

Thanks,
Yajun 

-----Original Message-----
From: Alex Bennée <alex.bennee@linaro.org> 
Sent: Wednesday, October 26, 2022 3:42 PM
To: Yajun Wu <yajunw@nvidia.com>
Cc: qemu-devel@nongnu.org; mst@redhat.com; Parav Pandit <parav@nvidia.com>
Subject: Re: [PATCH v2 1/4] hw/virtio: incorporate backend features in features

External email: Use caution opening links or attachments

Yajun Wu <yajunw@nvidia.com> writes:

> Hi Alex,
>
> With this change, VHOST_USER_F_PROTOCOL_FEATURES bit will be set to 
> backend for virtio block device (previously not).
>
> From https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.qemu.org%2Fdocs%2Fmaster%2Finterop%2Fvhost-user.html&amp;data=05%7C01%7Cyajunw%40nvidia.com%7C2e8901540bf441248ec608dab725ca87%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C638023670196631779%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=y5g9wwTfh%2BxrkESRypoo4pg3eKYInyDerDmI844PBSE%3D&amp;reserved=0 spec:
> If VHOST_USER_F_PROTOCOL_FEATURES has not been negotiated, the ring starts directly in the enabled state.
> If VHOST_USER_F_PROTOCOL_FEATURES has been negotiated, the ring is 
> initialized in a disabled state and is enabled by 
> VHOST_USER_SET_VRING_ENABLE with parameter 1.
>
> Vhost-user-blk won't send out VHOST_USER_SET_VRING_ENABLE today.
> Backend gets VHOST_USER_F_PROTOCOL_FEATURES negotiated and can't get VHOST_USER_SET_VRING_ENABLE.
> VQs keep in disabled state.

If the backend advertises protocol features but the stub doesn't support it how does it get enabled?

The testing I did was mostly by hand with the gpio backend and using the qtests. I Think we need to add some acceptance testing into avocado with some real daemons because I don't think we have enough coverage with the current qtest approach.

>
> Can you check on this scenario?
>
> Thanks
>
> -----Original Message-----
> From: Qemu-devel <qemu-devel-bounces+yajunw=nvidia.com@nongnu.org> On 
> Behalf Of Alex Bennée
> Sent: Thursday, July 28, 2022 9:55 PM
> To: qemu-devel@nongnu.org
> Cc: mst@redhat.com; Alex Bennée <alex.bennee@linaro.org>
> Subject: [PATCH v2 1/4] hw/virtio: incorporate backend features in 
> features
>
> External email: Use caution opening links or attachments
>
>
> There are some extra bits used over a vhost-user connection which are hidden from the device itself. We need to set them here to ensure we enable things like the protocol extensions.
>
> Currently net/vhost-user.c has it's own inscrutable way of persisting this data but it really should live in the core vhost_user code.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Message-Id: <20220726192150.2435175-7-alex.bennee@linaro.org>
> ---
>  hw/virtio/vhost-user.c | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c index 
> 75b8df21a4..1936a44e82 100644
> --- a/hw/virtio/vhost-user.c
> +++ b/hw/virtio/vhost-user.c
> @@ -1460,7 +1460,14 @@ static int vhost_user_set_features(struct vhost_dev *dev,
>       */
>      bool log_enabled = features & (0x1ULL << VHOST_F_LOG_ALL);
>
> -    return vhost_user_set_u64(dev, VHOST_USER_SET_FEATURES, features,
> +    /*
> +     * We need to include any extra backend only feature bits that
> +     * might be needed by our device. Currently this includes the
> +     * VHOST_USER_F_PROTOCOL_FEATURES bit for enabling protocol
> +     * features.
> +     */
> +    return vhost_user_set_u64(dev, VHOST_USER_SET_FEATURES,
> +                              features | dev->backend_features,
>                                log_enabled);  }

--
Alex Bennée