From: Jason Wang <jasowang@redhat.com>
To: Gautam Dawar <gdawar@amd.com>
Cc: Gautam Dawar <gautam.dawar@amd.com>,
linux-net-drivers@amd.com, netdev@vger.kernel.org,
eperezma@redhat.com, tanuj.kamde@amd.com, Koushik.Dutta@amd.com,
harpreet.anand@amd.com, Edward Cree <ecree.xilinx@gmail.com>,
Martin Habets <habetsm.xilinx@gmail.com>,
"David S. Miller" <davem@davemloft.net>,
Eric Dumazet <edumazet@google.com>,
Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
linux-kernel@vger.kernel.org
Subject: Re: [PATCH net-next 08/11] sfc: implement device status related vdpa config operations
Date: Mon, 16 Jan 2023 10:55:58 +0800 [thread overview]
Message-ID: <CACGkMEsEZMhS0zSRY0XviXSmDALd2zons0nAk43DTCBu_iA5tw@mail.gmail.com> (raw)
In-Reply-To: <71066e12-1c5c-c226-bfb7-67bea171a4e1@amd.com>
On Fri, Jan 13, 2023 at 2:33 PM Gautam Dawar <gdawar@amd.com> wrote:
>
>
> On 1/13/23 11:50, Jason Wang wrote:
> > Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
> >
> >
> > On Fri, Jan 13, 2023 at 2:11 PM Gautam Dawar <gdawar@amd.com> wrote:
> >>
> >> On 1/13/23 09:58, Jason Wang wrote:
> >>> Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
> >>>
> >>>
> >>> On Wed, Jan 11, 2023 at 2:36 PM Jason Wang <jasowang@redhat.com> wrote:
> >>>> On Mon, Jan 9, 2023 at 6:21 PM Gautam Dawar <gdawar@amd.com> wrote:
> >>>>> On 12/14/22 12:15, Jason Wang wrote:
> >>>>>> Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Dec 7, 2022 at 10:57 PM Gautam Dawar <gautam.dawar@amd.com> wrote:
> >>>>>>> vDPA config opertions to handle get/set device status and device
> >>>>>>> reset have been implemented.
> >>>>>>>
> >>>>>>> Signed-off-by: Gautam Dawar <gautam.dawar@amd.com>
> >>>>>>> ---
> >>>>>>> drivers/net/ethernet/sfc/ef100_vdpa.c | 7 +-
> >>>>>>> drivers/net/ethernet/sfc/ef100_vdpa.h | 1 +
> >>>>>>> drivers/net/ethernet/sfc/ef100_vdpa_ops.c | 133 ++++++++++++++++++++++
> >>>>>>> 3 files changed, 140 insertions(+), 1 deletion(-)
> >>>>>>>
> >>>>>>> diff --git a/drivers/net/ethernet/sfc/ef100_vdpa.c b/drivers/net/ethernet/sfc/ef100_vdpa.c
> >>>>>>> index 04d64bfe3c93..80bca281a748 100644
> >>>>>>> --- a/drivers/net/ethernet/sfc/ef100_vdpa.c
> >>>>>>> +++ b/drivers/net/ethernet/sfc/ef100_vdpa.c
> >>>>>>> @@ -225,9 +225,14 @@ static int vdpa_allocate_vis(struct efx_nic *efx, unsigned int *allocated_vis)
> >>>>>>>
> >>>>>>> static void ef100_vdpa_delete(struct efx_nic *efx)
> >>>>>>> {
> >>>>>>> + struct vdpa_device *vdpa_dev;
> >>>>>>> +
> >>>>>>> if (efx->vdpa_nic) {
> >>>>>>> + vdpa_dev = &efx->vdpa_nic->vdpa_dev;
> >>>>>>> + ef100_vdpa_reset(vdpa_dev);
> >>>>>> Any reason we need to reset during delete?
> >>>>> ef100_reset_vdpa_device() does the necessary clean-up including freeing
> >>>>> irqs, deleting filters and deleting the vrings which is required while
> >>>>> removing the vdpa device or unloading the driver.
> >>>> That's fine but the name might be a little bit confusing since vDPA
> >>>> reset is not necessary here.
> >>>>
> >>>>>>> +
> >>>>>>> /* replace with _vdpa_unregister_device later */
> >>>>>>> - put_device(&efx->vdpa_nic->vdpa_dev.dev);
> >>>>>>> + put_device(&vdpa_dev->dev);
> >>>>>>> efx->vdpa_nic = NULL;
> >>>>>>> }
> >>>>>>> efx_mcdi_free_vis(efx);
> >>>>>>> diff --git a/drivers/net/ethernet/sfc/ef100_vdpa.h b/drivers/net/ethernet/sfc/ef100_vdpa.h
> >>>>>>> index a33edd6dda12..1b0bbba88154 100644
> >>>>>>> --- a/drivers/net/ethernet/sfc/ef100_vdpa.h
> >>>>>>> +++ b/drivers/net/ethernet/sfc/ef100_vdpa.h
> >>>>>>> @@ -186,6 +186,7 @@ int ef100_vdpa_add_filter(struct ef100_vdpa_nic *vdpa_nic,
> >>>>>>> enum ef100_vdpa_mac_filter_type type);
> >>>>>>> int ef100_vdpa_irq_vectors_alloc(struct pci_dev *pci_dev, u16 nvqs);
> >>>>>>> void ef100_vdpa_irq_vectors_free(void *data);
> >>>>>>> +int ef100_vdpa_reset(struct vdpa_device *vdev);
> >>>>>>>
> >>>>>>> static inline bool efx_vdpa_is_little_endian(struct ef100_vdpa_nic *vdpa_nic)
> >>>>>>> {
> >>>>>>> diff --git a/drivers/net/ethernet/sfc/ef100_vdpa_ops.c b/drivers/net/ethernet/sfc/ef100_vdpa_ops.c
> >>>>>>> index 132ddb4a647b..718b67f6da90 100644
> >>>>>>> --- a/drivers/net/ethernet/sfc/ef100_vdpa_ops.c
> >>>>>>> +++ b/drivers/net/ethernet/sfc/ef100_vdpa_ops.c
> >>>>>>> @@ -251,6 +251,62 @@ static bool is_qid_invalid(struct ef100_vdpa_nic *vdpa_nic, u16 idx,
> >>>>>>> return false;
> >>>>>>> }
> >>>>>>>
> >>>>>>> +static void ef100_reset_vdpa_device(struct ef100_vdpa_nic *vdpa_nic)
> >>>>>>> +{
> >>>>>>> + int i;
> >>>>>>> +
> >>>>>>> + WARN_ON(!mutex_is_locked(&vdpa_nic->lock));
> >>>>>>> +
> >>>>>>> + if (!vdpa_nic->status)
> >>>>>>> + return;
> >>>>>>> +
> >>>>>>> + vdpa_nic->vdpa_state = EF100_VDPA_STATE_INITIALIZED;
> >>>>>>> + vdpa_nic->status = 0;
> >>>>>>> + vdpa_nic->features = 0;
> >>>>>>> + for (i = 0; i < (vdpa_nic->max_queue_pairs * 2); i++)
> >>>>>>> + reset_vring(vdpa_nic, i);
> >>>>>>> +}
> >>>>>>> +
> >>>>>>> +/* May be called under the rtnl lock */
> >>>>>>> +int ef100_vdpa_reset(struct vdpa_device *vdev)
> >>>>>>> +{
> >>>>>>> + struct ef100_vdpa_nic *vdpa_nic = get_vdpa_nic(vdev);
> >>>>>>> +
> >>>>>>> + /* vdpa device can be deleted anytime but the bar_config
> >>>>>>> + * could still be vdpa and hence efx->state would be STATE_VDPA.
> >>>>>>> + * Accordingly, ensure vdpa device exists before reset handling
> >>>>>>> + */
> >>>>>>> + if (!vdpa_nic)
> >>>>>>> + return -ENODEV;
> >>>>>>> +
> >>>>>>> + mutex_lock(&vdpa_nic->lock);
> >>>>>>> + ef100_reset_vdpa_device(vdpa_nic);
> >>>>>>> + mutex_unlock(&vdpa_nic->lock);
> >>>>>>> + return 0;
> >>>>>>> +}
> >>>>>>> +
> >>>>>>> +static int start_vdpa_device(struct ef100_vdpa_nic *vdpa_nic)
> >>>>>>> +{
> >>>>>>> + int rc = 0;
> >>>>>>> + int i, j;
> >>>>>>> +
> >>>>>>> + for (i = 0; i < (vdpa_nic->max_queue_pairs * 2); i++) {
> >>>>>>> + if (can_create_vring(vdpa_nic, i)) {
> >>>>>>> + rc = create_vring(vdpa_nic, i);
> >>>>>> So I think we can safely remove the create_vring() in set_vq_ready()
> >>>>>> since it's undefined behaviour if set_vq_ready() is called after
> >>>>>> DRIVER_OK.
> >>>>> Is this (undefined) behavior documented in the virtio spec?
> >>>> This part is kind of tricky:
> >>>>
> >>>> PCI transport has a queue_enable field. And recently,
> >>>> VIRTIO_F_RING_RESET was introduced. Let's start without that first:
> >>>>
> >>>> In
> >>>>
> >>>> 4.1.4.3.2 Driver Requirements: Common configuration structure layout
> >>>>
> >>>> It said:
> >>>>
> >>>> "The driver MUST configure the other virtqueue fields before enabling
> >>>> the virtqueue with queue_enable."
> >>>>
> >>>> and
> >>>>
> >>>> "The driver MUST NOT write a 0 to queue_enable."
> >>>>
> >>>> My understanding is that:
> >>>>
> >>>> 1) Write 0 is forbidden
> >>>> 2) Write 1 after DRIVER_OK is undefined behaviour (or need to clarify)
> >>>>
> >>>> With VIRTIO_F_RING_RESET is negotiated:
> >>>>
> >>>> "
> >>>> If VIRTIO_F_RING_RESET has been negotiated, after the driver writes 1
> >>>> to queue_reset to reset the queue, the driver MUST NOT consider queue
> >>>> reset to be complete until it reads back 0 in queue_reset. The driver
> >>>> MAY re-enable the queue by writing 1 to queue_enable after ensuring
> >>>> that other virtqueue fields have been set up correctly. The driver MAY
> >>>> set driver-writeable queue configuration values to different values
> >>>> than those that were used before the queue reset. (see 2.6.1).
> >>>> "
> >>>>
> >>>> Write 1 to queue_enable after DRIVER_OK and after the queue is reset is allowed.
> >>>>
> >>>> Thanks
> >>> Btw, I just realized that we need to stick to the current behaviour,
> >>> that is to say, to allow set_vq_ready() to be called after DRIVER_OK.
> >> So, both set_vq_ready() and DRIVER_OK are required for vring creation
> >> and their order doesn't matter. Is that correct?
> > Yes.
> >
> >> Also, will set_vq_ready(0) after DRIVER_OK result in queue deletion?
> > I think it should be treated as suspended or stopped. Since the device
> > should survive from kicking the vq even if the driver does
> > set_vq_ready(0).
> Ok. Is it expected that a queue restart (set_vq_ready(0) followed by
> set_vq_ready(1)) will start the queue from the last queue configuration
> when VIRTIO_F_RING_RESET isn't negotiated?
I think it's better to have this.
Thanks
> >
> > Thanks
> >
> >>> It is needed for the cvq trap and migration for control virtqueue:
> >>>
> >>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg931491.html
> >>>
> >>> Thanks
> >>>
> >>>
> >>>>> If so, can
> >>>>> you please point me to the section of virtio spec that calls this order
> >>>>> (set_vq_ready() after setting DRIVER_OK) undefined? Is it just that the
> >>>>> queue can't be enabled after DRIVER_OK or the reverse (disabling the
> >>>>> queue) after DRIVER_OK is not allowed?
> >>>>>>> + if (rc)
> >>>>>>> + goto clear_vring;
> >>>>>>> + }
> >>>>>>> + }
> >>>>>>> + vdpa_nic->vdpa_state = EF100_VDPA_STATE_STARTED;
> >>>>>>> + return rc;
> >>>>>>> +
> >>>>>>> +clear_vring:
> >>>>>>> + for (j = 0; j < i; j++)
> >>>>>>> + if (vdpa_nic->vring[j].vring_created)
> >>>>>>> + delete_vring(vdpa_nic, j);
> >>>>>>> + return rc;
> >>>>>>> +}
> >>>>>>> +
> >>>>>>> static int ef100_vdpa_set_vq_address(struct vdpa_device *vdev,
> >>>>>>> u16 idx, u64 desc_area, u64 driver_area,
> >>>>>>> u64 device_area)
> >>>>>>> @@ -568,6 +624,80 @@ static u32 ef100_vdpa_get_vendor_id(struct vdpa_device *vdev)
> >>>>>>> return EF100_VDPA_VENDOR_ID;
> >>>>>>> }
> >>>>>>>
> >>>>>>> +static u8 ef100_vdpa_get_status(struct vdpa_device *vdev)
> >>>>>>> +{
> >>>>>>> + struct ef100_vdpa_nic *vdpa_nic = get_vdpa_nic(vdev);
> >>>>>>> + u8 status;
> >>>>>>> +
> >>>>>>> + mutex_lock(&vdpa_nic->lock);
> >>>>>>> + status = vdpa_nic->status;
> >>>>>>> + mutex_unlock(&vdpa_nic->lock);
> >>>>>>> + return status;
> >>>>>>> +}
> >>>>>>> +
> >>>>>>> +static void ef100_vdpa_set_status(struct vdpa_device *vdev, u8 status)
> >>>>>>> +{
> >>>>>>> + struct ef100_vdpa_nic *vdpa_nic = get_vdpa_nic(vdev);
> >>>>>>> + u8 new_status;
> >>>>>>> + int rc;
> >>>>>>> +
> >>>>>>> + mutex_lock(&vdpa_nic->lock);
> >>>>>>> + if (!status) {
> >>>>>>> + dev_info(&vdev->dev,
> >>>>>>> + "%s: Status received is 0. Device reset being done\n",
> >>>>>>> + __func__);
> >>>>>>> + ef100_reset_vdpa_device(vdpa_nic);
> >>>>>>> + goto unlock_return;
> >>>>>>> + }
> >>>>>>> + new_status = status & ~vdpa_nic->status;
> >>>>>>> + if (new_status == 0) {
> >>>>>>> + dev_info(&vdev->dev,
> >>>>>>> + "%s: New status same as current status\n", __func__);
> >>>>>>> + goto unlock_return;
> >>>>>>> + }
> >>>>>>> + if (new_status & VIRTIO_CONFIG_S_FAILED) {
> >>>>>>> + ef100_reset_vdpa_device(vdpa_nic);
> >>>>>>> + goto unlock_return;
> >>>>>>> + }
> >>>>>>> +
> >>>>>>> + if (new_status & VIRTIO_CONFIG_S_ACKNOWLEDGE &&
> >>>>>>> + vdpa_nic->vdpa_state == EF100_VDPA_STATE_INITIALIZED) {
> >>>>>> As replied before, I think there's no need to check
> >>>>>> EF100_VDPA_STATE_INITIALIZED, otherwise it could be a bug somewhere.
> >>>>> Ok. Will remove the check against EF100_VDPA_STATE_INITIALIZED.
> >>>>>>> + vdpa_nic->status |= VIRTIO_CONFIG_S_ACKNOWLEDGE;
> >>>>>>> + new_status &= ~VIRTIO_CONFIG_S_ACKNOWLEDGE;
> >>>>>>> + }
> >>>>>>> + if (new_status & VIRTIO_CONFIG_S_DRIVER &&
> >>>>>>> + vdpa_nic->vdpa_state == EF100_VDPA_STATE_INITIALIZED) {
> >>>>>>> + vdpa_nic->status |= VIRTIO_CONFIG_S_DRIVER;
> >>>>>>> + new_status &= ~VIRTIO_CONFIG_S_DRIVER;
> >>>>>>> + }
> >>>>>>> + if (new_status & VIRTIO_CONFIG_S_FEATURES_OK &&
> >>>>>>> + vdpa_nic->vdpa_state == EF100_VDPA_STATE_INITIALIZED) {
> >>>>>>> + vdpa_nic->status |= VIRTIO_CONFIG_S_FEATURES_OK;
> >>>>>>> + vdpa_nic->vdpa_state = EF100_VDPA_STATE_NEGOTIATED;
> >>>>>> I think we can simply map EF100_VDPA_STATE_NEGOTIATED to
> >>>>>> VIRTIO_CONFIG_S_FEATURES_OK.
> >>>>>>
> >>>>>> E.g the code doesn't fail the feature negotiation by clearing the
> >>>>>> VIRTIO_CONFIG_S_FEATURES_OK when ef100_vdpa_set_driver_feature fails?
> >>>>> Ok.
> >>>>>> Thanks
> >>>>> Regards,
> >>>>>
> >>>>> Gautam
> >>>>>
>
next prev parent reply other threads:[~2023-01-16 2:57 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-12-07 14:54 [PATCH net-next 00/11] sfc: add vDPA support for EF100 devices Gautam Dawar
2022-12-07 14:54 ` [PATCH net-next 01/11] sfc: add function personality " Gautam Dawar
2022-12-07 14:54 ` [PATCH net-next 02/11] sfc: implement MCDI interface for vDPA operations Gautam Dawar
2022-12-14 6:43 ` Jason Wang
2022-12-16 12:50 ` Gautam Dawar
2022-12-07 14:54 ` [PATCH net-next 03/11] sfc: implement init and fini functions for vDPA personality Gautam Dawar
2022-12-07 14:54 ` [PATCH net-next 04/11] sfc: implement vDPA management device operations Gautam Dawar
2022-12-07 16:31 ` kernel test robot
2022-12-14 6:43 ` Jason Wang
2022-12-15 7:07 ` Gautam Dawar
2022-12-07 14:54 ` [PATCH net-next 05/11] sfc: implement vdpa device config operations Gautam Dawar
2022-12-14 6:44 ` Jason Wang
2022-12-15 9:53 ` Gautam Dawar
2022-12-07 14:54 ` [PATCH net-next 06/11] sfc: implement vdpa vring " Gautam Dawar
2022-12-14 6:45 ` Jason Wang
2022-12-07 14:54 ` [PATCH net-next 07/11] sfc: implement filters for receiving traffic Gautam Dawar
2022-12-14 6:45 ` Jason Wang
2023-01-05 12:46 ` Gautam Dawar
2022-12-07 14:54 ` [PATCH net-next 08/11] sfc: implement device status related vdpa config operations Gautam Dawar
2022-12-14 6:45 ` Jason Wang
2023-01-09 10:21 ` Gautam Dawar
2023-01-11 6:36 ` Jason Wang
2023-01-13 4:28 ` Jason Wang
2023-01-13 6:10 ` Gautam Dawar
2023-01-13 6:20 ` Jason Wang
2023-01-13 6:33 ` Gautam Dawar
2023-01-16 2:55 ` Jason Wang [this message]
2022-12-07 14:54 ` [PATCH net-next 09/11] sfc: implement iova rbtree to store dma mappings Gautam Dawar
2022-12-14 6:46 ` Jason Wang
2022-12-16 12:48 ` Gautam Dawar
2022-12-19 6:03 ` Jason Wang
2023-01-05 13:08 ` Gautam Dawar
2023-01-09 8:49 ` Jason Wang
2023-02-09 13:39 ` Gautam Dawar
2022-12-07 14:54 ` [PATCH net-next 10/11] sfc: implement vdpa config_ops for dma operations Gautam Dawar
2022-12-07 19:23 ` kernel test robot
2022-12-14 6:46 ` Jason Wang
2022-12-07 14:54 ` [PATCH net-next 11/11] sfc: register the vDPA device Gautam Dawar
2022-12-11 18:05 ` [PATCH net-next 00/11] sfc: add vDPA support for EF100 devices Martin Habets
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CACGkMEsEZMhS0zSRY0XviXSmDALd2zons0nAk43DTCBu_iA5tw@mail.gmail.com \
--to=jasowang@redhat.com \
--cc=Koushik.Dutta@amd.com \
--cc=davem@davemloft.net \
--cc=ecree.xilinx@gmail.com \
--cc=edumazet@google.com \
--cc=eperezma@redhat.com \
--cc=gautam.dawar@amd.com \
--cc=gdawar@amd.com \
--cc=habetsm.xilinx@gmail.com \
--cc=harpreet.anand@amd.com \
--cc=kuba@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-net-drivers@amd.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=tanuj.kamde@amd.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).