Re: [RFC 1/8] virtio/virtio-pci: Handle extra notification data

From: Jonah Palmer <jonah.palmer@oracle.com>
To: Eugenio Perez Martin <eperezma@redhat.com>
Cc: qemu-devel@nongnu.org, mst@redhat.com, jasowang@redhat.com,
	si-wei.liu@oracle.com, boris.ostrovsky@oracle.com,
	raphael@enfabrica.net, kwolf@redhat.com, hreitz@redhat.com,
	pasic@linux.ibm.com, borntraeger@linux.ibm.com,
	farman@linux.ibm.com, thuth@redhat.com,
	richard.henderson@linaro.org, david@redhat.com,
	iii@linux.ibm.com, cohuck@redhat.com, pbonzini@redhat.com,
	fam@euphon.net, stefanha@redhat.com, qemu-block@nongnu.org,
	qemu-s390x@nongnu.org, virtio-fs@lists.linux.dev
Subject: Re: [RFC 1/8] virtio/virtio-pci: Handle extra notification data
Date: Mon, 4 Mar 2024 12:32:01 -0500	[thread overview]
Message-ID: <d9c1aaa3-5b46-4b6d-adf8-5a787d3ebfaa@oracle.com> (raw)
In-Reply-To: <CAJaqyWcnfhRA=xXHxdgF-u8gxJEAt7FQFryCrwj52N+4aLPbUQ@mail.gmail.com>

On 3/4/24 12:24 PM, Eugenio Perez Martin wrote:
> On Mon, Mar 4, 2024 at 6:09 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>>
>>
>>
>> On 3/1/24 2:31 PM, Eugenio Perez Martin wrote:
>>> On Fri, Mar 1, 2024 at 2:44 PM Jonah Palmer <jonah.palmer@oracle.com> wrote:
>>>>
>>>> Add support to virtio-pci devices for handling the extra data sent
>>>> from the driver to the device when the VIRTIO_F_NOTIFICATION_DATA
>>>> transport feature has been negotiated.
>>>>
>>>> The extra data that's passed to the virtio-pci device when this
>>>> feature is enabled varies depending on the device's virtqueue
>>>> layout.
>>>>
>>>> In a split virtqueue layout, this data includes:
>>>>    - upper 16 bits: last_avail_idx
>>>>    - lower 16 bits: virtqueue index
>>>>
>>>> In a packed virtqueue layout, this data includes:
>>>>    - upper 16 bits: 1-bit wrap counter & 15-bit last_avail_idx
>>>>    - lower 16 bits: virtqueue index
>>>>
>>>> Signed-off-by: Jonah Palmer <jonah.palmer@oracle.com>
>>>> ---
>>>>    hw/virtio/virtio-pci.c     | 13 ++++++++++---
>>>>    hw/virtio/virtio.c         | 13 +++++++++++++
>>>>    include/hw/virtio/virtio.h |  1 +
>>>>    3 files changed, 24 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
>>>> index 1a7039fb0c..c7c577b177 100644
>>>> --- a/hw/virtio/virtio-pci.c
>>>> +++ b/hw/virtio/virtio-pci.c
>>>> @@ -384,7 +384,7 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)
>>>>    {
>>>>        VirtIOPCIProxy *proxy = opaque;
>>>>        VirtIODevice *vdev = virtio_bus_get_device(&proxy->bus);
>>>> -    uint16_t vector;
>>>> +    uint16_t vector, vq_idx;
>>>>        hwaddr pa;
>>>>
>>>>        switch (addr) {
>>>> @@ -408,8 +408,15 @@ static void virtio_ioport_write(void *opaque, uint32_t addr, uint32_t val)
>>>>                vdev->queue_sel = val;
>>>>            break;
>>>>        case VIRTIO_PCI_QUEUE_NOTIFY:
>>>> -        if (val < VIRTIO_QUEUE_MAX) {
>>>> -            virtio_queue_notify(vdev, val);
>>>> +        if (virtio_vdev_has_feature(vdev, VIRTIO_F_NOTIFICATION_DATA)) {
>>>> +            vq_idx = val & 0xFFFF;
>>>
>>> Nitpick, but since vq_idx is already a uint16_t the & 0xFFFF is not
>>> needed.
>>
>> Ah okay. I wasn't sure if it was worthwhile to keep the '& 0xFFFF' in or
>> not for the sake of clarity and good practice. In that case I could just
>> do away with vq_idx here and use explicit casting on 'val'.
>>
>>> I think it's cleaner just to call virtio_set_notification data
>>> in the has_feature(...) condition, but I'm happy with this too.
>>
>> Do you mean something like:
>>
>> if (virtio_vdev_has_feature(vdev, VIRTIO_F_NOTIFICATION_DATA) &&
>>       virtio_set_notification_data(vdev, vq_idx, val)) {
>>       ...
>> }
>>
> 
> Sorry I was not clear, I meant just to take out the common code of the
> conditionals:
> vq_idx = val;
> if (virtio_vdev_has_feature(vdev, VIRTIO_F_NOTIFICATION_DATA) {
>      virtio_set_notification_data(vdev, val);
> }
> 

Ah, no problem! Thank you for the clarification!

>> Though I'm not sure what would then go in the body of this conditional,
>> especially if I did something like:
>>
>> case VIRTIO_PCI_QUEUE_NOTIFY:
>>       if ((uint16_t)val < VIRTIO_QUEUE_MAX) {
>>           if (virtio_vdev_has_feature(vdev, VIRTIO_F_NOTIFICATION_DATA) &&
>>               virtio_set_notification_data(vdev, val)) {
>>               // Not sure what to put here other than a no-op
>>           }
>>
>>           virtio_queue_notify(vdev, (uint16_t)val);
>>       }
>>       break;
>>
>> But I'm not sure if you'd prefer this explicit casting of 'val' over
>> implicit casting like:
>>
>> uint16_t vq_idx = val;
>>
>>>
>>>> +            virtio_set_notification_data(vdev, vq_idx, val);
>>>> +        } else {
>>>> +            vq_idx = val;
>>>> +        }
>>>> +
>>>> +        if (vq_idx < VIRTIO_QUEUE_MAX) {
>>>> +            virtio_queue_notify(vdev, vq_idx);
>>>>            }
>>>>            break;
>>>>        case VIRTIO_PCI_STATUS:
>>>> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
>>>> index d229755eae..a61f69b7fd 100644
>>>> --- a/hw/virtio/virtio.c
>>>> +++ b/hw/virtio/virtio.c
>>>> @@ -2052,6 +2052,19 @@ int virtio_set_status(VirtIODevice *vdev, uint8_t val)
>>>>        return 0;
>>>>    }
>>>>
>>>> +void virtio_set_notification_data(VirtIODevice *vdev, uint16_t i, uint32_t data)
>>>> +{
>>>> +    VirtQueue *vq = &vdev->vq[i];
>>>> +
>>>> +    if (virtio_vdev_has_feature(vdev, VIRTIO_F_RING_PACKED)) {
>>>> +        vq->last_avail_wrap_counter = (data >> 31) & 0x1;
>>>> +        vq->last_avail_idx = (data >> 16) & 0x7FFF;
>>>> +    } else {
>>>> +        vq->last_avail_idx = (data >> 16) & 0xFFFF;
>>>> +    }
>>>
>>> It should not set last_avail_idx, only shadow_avail_idx. Otherwise,
>>> QEMU can only see the descriptors placed after the notification.
>>>
>>> Or am I missing something?
>>>
>>> In that regard, I would call this function
>>> "virtqueue_set_shadow_avail_idx". But I'm very bad at naming :).
>>
>> Ah that's right. This would make Qemu skip processing descriptors that
>> might've been made available before the notification but after the
>> host's last check of last_avail_idx. In other words, ignoring available
>> descriptors that were placed before the notification but not yet
>> processed. Good catch, thank you!
>>
>> So, for the packed VQ layout, we'll still want to save the wrap counter
>> but for the shadow_avail_idx, right? E.g.
>>
>> if (virtio_vdev_has_feature(vdev, VIRTIO_F_RING_PACKED)) {
>>       vq->shadow_avail_wrap_counter = (data >> 31) & 0x1;
>>       vq->shadow_avail_idx = (data >> 16) & 0x7FFF;
>> } else {
>>       vq->shadow_avail_idx = (data >> 16);
>> }
>>
> 
> Right, I was not clear enough again :). You probably have guessed
> already but not modifying avail_wrap_counter would make QEMu to read
> the wrong index too.
> 
> Thanks!
> 

Got it, thanks!

>>>
>>> The rest looks good to me.
>>>
>>> Thanks!
>>>
>>>> +    vq->shadow_avail_idx = vq->last_avail_idx;
>>>> +}
>>>> +
>>>>    static enum virtio_device_endian virtio_default_endian(void)
>>>>    {
>>>>        if (target_words_bigendian()) {
>>>> diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
>>>> index c8f72850bc..c92d8afc42 100644
>>>> --- a/include/hw/virtio/virtio.h
>>>> +++ b/include/hw/virtio/virtio.h
>>>> @@ -345,6 +345,7 @@ void virtio_queue_reset(VirtIODevice *vdev, uint32_t queue_index);
>>>>    void virtio_queue_enable(VirtIODevice *vdev, uint32_t queue_index);
>>>>    void virtio_update_irq(VirtIODevice *vdev);
>>>>    int virtio_set_features(VirtIODevice *vdev, uint64_t val);
>>>> +void virtio_set_notification_data(VirtIODevice *vdev, uint16_t i, uint32_t data);
>>>>
>>>>    /* Base devices.  */
>>>>    typedef struct VirtIOBlkConf VirtIOBlkConf;
>>>> --
>>>> 2.39.3
>>>>
>>>
>>
> 